Patent ID: 12254890

DESCRIPTION OF EMBODIMENTS

Embodiment

An overview of a voice signal generation system100according to an embodiment will be described usingFIG.1andFIG.2.FIG.1is an explanatory diagram illustrating an overview of a voice signal generation system100of the embodiment. The voice signal generation system100converts a voice signal (hereinafter referred to as a “conversion target voice signal”) representing a vocal sound (hereinafter referred to as “first speaker voice) uttered by a first speaker (hereinafter referred to as a “first speaker”) into a converted voice signal. The converted voice signal is a voice signal representing a voice having acoustic features of a vocal sound uttered by a second speaker rather than acoustic features of the vocal sound uttered by the first speaker while having the same content as the conversion target voice signal. The second speaker is a speaker designated in advance by a user or the like to the voice signal generation system100as a speaker of the voice represented by the converted voice signal.

The voice signal generation system100includes a voice signal conversion model learning device (audio signal conversion model learning apparatus)1and a voice signal conversion device (audio signal conversion apparatus)2. the voice signal conversion model learning device1updates a model of machine learning for converting a conversion target voice signal into a converted voice signal (hereinafter referred to as a “voice signal conversion model”) by machine learning until predetermined termination conditions are satisfied.

Hereinafter, machine learning is called learning for simplicity of description. In addition, updating a model of machine learning (hereinafter referred to as “machine learning model”) by machine learning means suitably adjusting parameter values in the machine learning model. The team “for learning” means use for updating the machine learning model. In the following description, learning to be A means that parameter values in the machine learning model are adjusted to satisfy A. A represents a condition.

FIG.2is an explanatory diagram illustrating an overview of the voice signal conversion model learning device1according to the embodiment. The voice signal conversion model learning device1updates the voice signal conversion model by performing learning using first learning data and second learning data. When the first learning data and the second learning data are not distinguished from each other, they are referred to as learning data.

The first learning data is data having a voice signal, conversion source speaker information, and conversion destination speaker information. The conversion source speaker information represents a speaker of a voice (hereinafter referred to as a “first learning voice”) represented by a voice signal (hereinafter referred to as a “first learning voice signal”) represented by the first learning data. The conversion destination speaker information represents a speaker set in advance as a speaker of a voice (hereinafter referred to as “a first type generation voice”) represented by a voice signal (hereinafter referred to as a “first type generated signal”) of a conversion destination by the voice signal conversion model of the first learning voice signal. The setting is performed by a user, for example. The speaker indicated by the conversion source speaker information and the speaker indicated by the conversion destination speaker information may be the same or different. Hereinafter, for simplicity of description, the first learning data in which the first learning voice signal is S0, the speaker indicated by the conversion source speaker intonation is C1, and the speaker indicated by the conversion destination speaker information is C2is represented by (S0, C1, C2). Meanwhile, a symbol (A1, A2, A3) represents that information of a set of information A1, information A2, and information A3is input to a generation unit110which will be described later.

The second learning data includes a voice signal, random speaker information, and speaker identification information. The speaker identification information represents a speaker set in advance as a speaker of a voice (hereinafter referred to as a “second learning voice”) represented by a voice signal (hereinafter referred to as a “a second learning voice signal”) represented by the second learning data. The random speaker information is information indicating a speaker randomly determined by a determination unit130, which will be described later, among a plurality of speakers prepared in advance. Random determination is performed using a technique for generating random numbers such as a random number generator. Hereinafter, for simplicity of description, the second learning data in which the second learning voice signal is S′0, the speaker indicated by the random speaker information is C′2, and the speaker indicated by the speaker identification information is C′1is represented by (S′0, C′2, C′1). Meanwhile, a symbol (A1, A2, A3) represents that information of a set of information A1, information A2, and information A3is input to an identification unit120or a loss acquisition unit140which will be described later.

The voice signal conversion model learning device1includes the generation unit110, the identification unit120, the determination unit130, and the loss acquisition unit140. The generation unit110acquires the first learning data and executes first type data generation processing and second type data generation processing using the acquired first learning data (S0, C1, C2).

The first type data generation processing is processing for generating first type generation data by the voice signal conversion model on the basis of the acquired first learning data. The first type generation data is data having a first type generation signal, conversion source speaker information, and conversion destination speaker information. Therefore, if the first type generation data is represented by a symbol following the expression of the first learning data, the first type generation data is represented by [S1, C1, C2] if the first type generation signal is S1.

The second type data generation processing is processing for generating second type generation data on the basis of the first type generation data generated by first type data generation processing. The second type generation data includes a second type generation signal, conversion source speaker information, and conversion destination speaker information. The second type generation signal is a voice signal (hereinafter referred to as a “reverse voice signal”) represented by an execution result of first type data generation processing for data for reverse generation.

The data for reverse generation is first learning data in which the conversion source speaker information of the first type generation data is conversion destination speaker information, the conversion destination speaker information of the first type generation data is conversion source speaker information, and the first type generation signal is a first learning voice signal. Therefore, if the data for reverse generation is represented by a symbol following the expression of the first learning data, the data for reverse generation is represented by (S1, C2, C1).

Since the data for reverse generation is represented by (S1, C2, if the second type generation data is represented by a symbol following the expression of the first learning data, the second type generation data is represented by [S2, C2, C1] when the reverse voice signal is S2. In this manner, second type data generation processing is first type data generation processing for the data for reverse generation.

The generation unit110outputs the generated first type generation data to the identification unit120. The generation unit110outputs the generated second type generation data to the loss acquisition unit140.

Hereinafter, a pair of information including the conversion source speaker information and the conversion destination speaker information included in the first type generation data is referred to as first pair information. Hereinafter, a pair of information including the random speaker information and the speaker identification information included in the second learning data is referred to as second pair information. Both the first pair information and the second pair information are pairs of information indicating a speaker. Therefore, when the first pair intonation and the second pair intonation are not distinguished from each other, they are referred to as pair information hereinafter. Further, both the first pair information and the second pair information include information indicating a speaker set in advance by a user or the like as a speaker of a voice signal included in the first type generation data or the second learning data including pair information. Specifically, the conversion destination speaker intonation included in the first type generation data is information included in the first pair information and indicating a speaker set in advance, and the speaker identification information included in the second learning data is information included in the second pair information and indicating a speaker set in advance. Hereinafter, when the conversion destination speaker information included in the first type generation data and the speaker identification information included in the second learning data are not distinguished from each other, they are referred to as a speaker setting information.

The identification unit120executes voice estimation processing. Voice estimation processing is processing for estimating whether or not a voice signal that is a processing target is a voice signal representing a vocal sound actually uttered by a speaker indicated by speaker setting information among information indicated by pair information on the basis of pair information of the voice signal that is the processing target.

A voice signal that is a processing target of the identification unit120is data input to the identification unit120and is a voice (hereinafter referred to as “identification voice”) represented by a voice signal represented by data (hereinafter referred to as “identification input data”) including the voice signal. The identification input data is specifically the first type generation data and the second learning data. An estimation result of the identification unit120is output to the loss acquisition unit140.

The determination unit130determines which of the first type generation data and the second learning data is the identification input data according to a predetermined rule. The predetermined rule may be any rule as long as the identification input data can be determined, and for example, is a rule of determining the first learning data and the second learning data as the identification input data at equal probability using a random number generated by a random number generator.

When the determination unit130has determined the first type generation data as the identification input data, the determination unit130determines first learning data to be input to the generation unit110from among a plurality of pieces of data included in a first learning data group according to a predetermined rule. The first learning data group is a set of first learning data. The predetermined rule may be any rule as long as the first learning data to be input to the generation unit110can be determined from a plurality of pieces of data included in the first learning data group. The predetermined rule may be, for example, a rule according to an order previously given to each piece of data. The predetermined rule may follow random sampling.

When the determination unit130has determined the second learning data as the identification input data, the determination unit130determines the second learning data to be input to the identification unit120from among a plurality of pieces of data included in a second learning data group according to a predetermined rule. The predetermined rule may be, for example, a rule according to an order previously given to each piece of data. The predetermined rule may follow random sampling. The second learning data group is a set of second learning data. Each piece of data of the first learning data group and the second learning data group is data stored in a storage unit which is included in the voice signal conversion model learning device1and will be described later.

The determination unit130outputs information (hereinafter referred to as “route information”) representing whether the identification input data has been determined as the first type generation data or the second learning data to the loss acquisition unit140.

When the determination unit130has determined the first type generation data as the identification input data, the generation unit110acquires the first learning data determined as the first learning data to be input to the generation unit110by the determination unit130. When the determination unit130has determined the second learning data as the identification input data, the identification unit120acquires the second learning data determined by the determination unit130as the second learning data to be input to the identification unit120.

When the second learning data has been determined as the identification input data, the determination unit130also determines random speaker information.

The loss acquisition unit140acquires the identification input data, the second type generation data, and the route information, and acquires a value of an objective function L (hereinafter referred to as “objective loss”) represented by the following formulas (1) to (4). The objective function L includes an extended adversarial loss function represented by the following formula (2), a cyclic loss function represented by the following formula (3), and an identity loss function represented by the following formula (4).
[Math. 1]
L=Lst_adv+λcycLcyc+λidLid(1)
[Math. 2]
Lst_adv=(x,c1)˜P(x,c1),c2˜P(c2)[logD(x,c2,c1)]+(x,c1)˜P(x,c1),c2˜P(c2)[logD(G(x,c1,c2),c1,c2)]   (2)
[Math. 3]
Lcyc=(x,c1)·P(x,c1),c2˜P(c2)[(∥x−G(G(x,c1,c2),c2,c1)∥1]   (3)
[Math. 4]
Lid=(x,c1)˜P(x,c1)[∥G(x,c1,c1)−x∥1]  (4)

D indicates mapping from identification input data by natural voice estimation processing and speaker estimation processing executed by the identification unit120to an estimation result. G indicates mapping representing conversion of data by first type data generation processing executed by the generation unit110.

X indicates a voice signal represented by the identification input data. Among the subscripts of E in the formulas (2) to (4), (x,c1) P(x,c1) indicates that an acoustic feature amount x and speaker information c1corresponding to the acoustic feature amount x are sampled from a distribution P(x,c1) of learning data. The speaker intonation means conversion source speaker intonation, conversion destination speaker information, random speaker information or speaker identification information. The distribution of learning data specifically represents a probability distribution having a feature amount of the first learning data in the first learning data group as a probability variable. That is, P(x,c1) is a multidimensional distribution in which the horizontal axis represents each dimension of (x,c1). E represents an expectation value.

Among the subscripts of E in the formulas (2) to (4), c2˜P(c2) represents that speaker information is randomly sampled.

In addition, x, c1and c2of the first term of the right side of the formula (2) represent S′0, C′1, C′2of the second learning data in order. In addition, c1and c2of the second term of the right side of the formula (2) represent C1and C2of the first learning data and the first type generation data in order, x represents S0of the first learning data, and G(x,c1, c2) represents S1of the first type generation data. In addition, c1and c2of the right side of the formula (3) represent C1and C2of the first learning data, the data for reverse generation, and the second type generation data in order. In addition, x of the right side of the formula (3) represents S0of the first learning data. In addition, G(x,c1,c2) of the right side of the formula (3) represents S1of the data for reverse generation, and G(G(x,c1,c2)c2,c1) represents S2of the second type generation data. Further, x of the right side of the formula (4) represents S0of the first learning data, and c1represents C1and C2of the first learning data.

The value of the extended adversarial loss function (hereinafter referred to as “extended adversarial loss”) indicates differences between the sound quality and speaker estimated by the identification unit120and the sound quality and speaker of the identification voice. The speaker of the identification voice is the speaker indicated by the conversion destination speaker information when the route intonation indicates that the first type generation data is the identification input data, and is the speaker indicated by the speaker identification information when the route information indicates that the second learning data is the identification input data. When the identification voice is the second learning voice, the sound quality of the identification voice is a natural voice set, and when the identification voice is the first type generation voice, the sound quality of the identification voice is a synthetic voice set.

The value of the cyclic loss function (hereinafter referred to as “cyclic loss”) indicates a difference between the voice signal represented by the second type generation data (that is, the second type generation signal) and the voice signal represented by the first learning data (that is, the first learning voice signal).

The identity loss function is a loss function introduced to restrict the first learning voice and the first type generation voice such that the two voices become identical when the speaker indicated by the conversion source speaker information of the first learning data input to the generation unit110and the speaker indicated by the conversion destination speaker information of the first learning data are identical.

The objective loss acquired by the loss acquisition unit140is output to the generation unit110and the identification unit120. The generation unit110and the identification unit120perform learning on the basis of the objective loss. More specifically, for example, the generation unit110performs learning such that the objective loss decreases, and the identification unit120performs learning such that the extended adversarial loss function increases. The generation unit110and the identification unit120may be any one as long as they can perform learning on the basis of the objective loss, and the generation unit110and the identification unit120are, for example, a neural network.

FIG.3is an explanatory diagram illustrating an example of a flow of first type data generation processing in the embodiment.

The generation unit110acquires first learning data (step S101). Next, the generation unit110generates first type generation data on the basis of the first learning data (step S102).

FIG.4is an explanatory diagram illustrating an example of a flow of second type data generation processing in the embodiment.

The generation unit110acquires first type generation data (step S201). Processing in step S201may be processing in step S102or processing in which the generation unit110re-acquires the first type generation data generated in processing in step S102. Next, the generation unit110generates second type generation data by executing first type data generation processing on data for reverse generation on the basis of the first type generation data (step S202).

FIG.5is an explanatory diagram illustrating an example of a flow of processing executed by the identification unit120in the embodiment. The identification unit120acquires identification input data (step S301). The identification unit120executes voice estimation processing (step S302).

FIG.6is a first diagram showing an example of a flow of processing executed by the voice signal conversion model learning device1of the embodiment. The same processing as that shown inFIG.3toFIG.5is denoted by the same reference numerals as those inFIGS.3to5, and description thereof is omitted.

The determination unit130determines the identification input data as the first type generation data (step S401). Next, processing of step S101is executed. Next, processing of step S102is executed. Next, processing of step S202is executed. Next, processing of step S301is executed. Next, processing of step S302is executed. Next, the loss acquisition unit140acquires objective loss on the basis of the first learning data acquired in step S101, the second type generation data, and the estimation result of step S302(step S402). The generation unit110and the identification unit120perform learning on the basis of the objective loss (step S403).

FIG.7is a diagram showing an example of a hardware configuration of the voice signal conversion model learning device1of the embodiment.

The voice signal conversion model learning device1includes a control unit10including a processor91such as a central processing unit (CPU) and a memory92, which are connected through a bus, and executes a program. The voice signal conversion model learning device1functions as a device including the control unit10, an input unit11, an interface unit12, a storage unit13, and an output unit14by executing the program. More specifically, the processor91reads the program stored in the storage unit13and stores the read program in the memory92. According to the processor91executing the program stored in the memory92, the voice signal conversion model learning device1functions as a device including the control unit10, the input unit11, the interface unit12, the storage unit13, and the output unit14.

The control unit10controls operations of various functional units included in the voice signal conversion model learning device1. The control unit10executes, for example, first type data generation processing. The control unit10executes, for example, second type data generation processing. The control unit10executes, for example, natural voice estimation processing. The control unit10executes, for example, speaker estimation processing.

The input unit11includes an input device such as a mouse, a keyboard, and a touch panel. The input unit11may also be configured as an interface for connecting these input devices to the host device.

The input unit11receives inputs of various types of information to the host device. The input unit11receives an input for instructing the start of learning, for example. The input unit11receives an input of data to be added to, for example, the first learning data group. The input unit11receives an input of data to be added to, for example, the second learning data group.

The interface unit12includes a communication interface for connecting the host device to an external device. The interface unit12communicates with the external device in a wired or wireless manner. The external device may be, for example, a storage device such as a Universal Serial Bus (USB) memory. When the external device outputs, for example, the first learning data, the interface unit12acquires the first learning data output from the external device through communication with the external device. When the external device outputs, for example, the second learning data, the interface unit12acquires the second learning data output from the external device through communication with the external device.

The interface unit12includes a communication interface for connecting the host device to the voice signal conversion device2. The interface unit12communicates with the voice signal conversion device2in a wired or wireless manner. The interface unit12outputs a learned voice signal conversion model to the voice signal conversion device2through communication with the voice signal conversion device2. “Learned” means that a predetermined termination condition is satisfied.

The storage unit13is configured using a non-transitory computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit13stores various types of information regarding the voice signal conversion model learning device1. The storage unit13stores, for example, the voice signal conversion model. The storage unit13stores, for example, the first learning data group in advance. The storage unit13stores, for example, the second learning data group in advance. The storage unit13stores first learning data and second learning data input via, for example, the input unit11or an interface unit12. The storage unit13stores, for example, an estimation result of the identification unit120.

The output unit14outputs various types of intonation. The output unit14includes a display device such as a cathode ray tube (CRT) display, a liquid crystal display, or an organic electroluminescence (EL) display. The output unit14may also be configured as an interface for connecting such a display device to the host device. The output unit14outputs, for example, information input to the input unit11.

FIG.8is a diagram showing an example of a functional configuration of the control unit10in the embodiment.

The control unit10includes a managed unit101and a management unit102. The managed unit101includes a generation unit110, an identification unit120, a determination unit130, and a loss acquisition unit140. The managed unit101updates the voice signal conversion model using the first learning data and the second learning data until termination conditions are satisfied.

The management unit102controls the operation of the managed unit101. The management unit102controls timing of processing executed by, for example, the generation unit110, the identification unit120, the determination unit130, and the loss acquisition unit140included in the managed unit101.

The management unit102controls operations of, for example, the input unit11, the interface unit12, the storage unit13, and the output unit14. The management unit102reads various types of information from, for example, the storage unit13and outputs the intonation to the managed unit101. The management unit102acquires, for example, information input to the input unit11and outputs the information to the managed unit101. The management unit102acquires, for example, information input to the input unit11and records the information in the storage unit13. The management unit102acquires, for example, intonation input to the interface unit12and outputs the intonation to the managed unit101. The management unit102acquires, for example, information input to then interface unit12and records the information in the storage unit13. The management unit102outputs, for example, information input to the input unit11to the output unit14.

The management unit102records, for example, the first type generation data generated by the generation unit110in the storage unit13. The management unit102records, for example, a result of the identification unit120in the storage unit13. The management unit102records, for example, a determination result of the determination unit130in the storage unit13. The management unit102records, for example, loss acquired by the loss acquisition unit140in the storage unit13.

FIG.9is a diagram showing an example of a hardware configuration of the voice signal conversion device2in the embodiment.

The voice signal conversion device2includes a control unit20including a processor93such as a central processing unit (CPU) and a memory94, which are connected through a bus, and executes a program. The voice signal conversion device2functions as a device including the control unit20, an input unit21, an interface unit22, a storage unit23, and an output unit24by executing the program. More specifically, the processor93reads the program stored in the storage unit23and stores the read program in the memory94. According to the processor93executing the program stored in the memory94, the voice signal conversion device2functions as a device including the control unit20, the input unit21, the interface unit22, the storage unit23, and the output unit24.

The control unit20controls operations of various functional units included in the voice signal conversion device2. The control unit20converts a conversion target voice signal into a converted voice signal using, for example, the learned voice signal conversion model obtained by the voice signal conversion model learning device1.

The input unit21includes input devices such as a mouse, a keyboard, and a touch panel. The input unit21may also be configured as an interface for connecting these input devices to the host device. The input unit21receives inputs of various types of information to the host device. The input unit21receives an input for instructing start of processing for converting, for example, a conversion target voice signal into a converted voice signal. The input unit21receives, for example, an input of a conversion target voice signal that is a conversion target.

The interface unit22includes a communication interface for connecting the host device to an external device. The interface unit22communicates with the external device in a wired or wireless manner. The external device is, for example, an output destination of a conversion target voice signal. In such a case, the interface unit22outputs the conversion target voice signal to the external device through communication with the external device. The external device at the time of outputting the conversion target voice signal is, for example, a voice output device such as a speaker.

The external device may be, for example, a storage device such as a USB memory storing the learned voice signal conversion model. When the external device stores, for example, the learned voice signal conversion model and outputs the learned voice signal conversion model, the interface unit22acquires the learned voice signal conversion model through communication with the external device.

The external device is, for example, an output source of a conversion target voice signal. In such a case, the interface unit22acquires the conversion target voice signal from the external device through communication with the external device.

The interface unit22includes a communication interface for connecting the host device to the voice signal conversion model learning device1. The interface unit22communicates with the voice signal conversion model learning device1in a wired or wireless manner. The interface unit22acquires a learned voice signal conversion model from the voice signal conversion model learning device1through communication with the voice signal conversion model learning device1.

The storage unit23is configured using a non-transitory computer-readable storage medium device such as a magnetic hard disk device or a semiconductor storage device. The storage unit23stores various types of information regarding the voice signal conversion device2.

The storage unit23stores, for example, the learned voice signal conversion model acquired via the interface unit22.

The output unit24outputs various types of information. The output unit24includes a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit24may also be configured as an interface for connecting such a display device to the host device. The output unit24outputs, for example, information input to the input unit21.

FIG.10is a diagram showing an example of a functional configuration of the control unit20in the embodiment. The control unit20includes a conversion target acquisition unit201, a conversion unit202, and a voice signal output control unit203.

The conversion target acquisition unit201acquires a conversion target voice signal that is a conversion target. The conversion target acquisition unit201acquires, for example, a conversion target voice signal input to the input unit21. The conversion target acquisition unit201acquires, for example, a conversion target voice signal input to the interface unit22.

The conversion unit202converts the conversion target acquired by the conversion target acquisition unit201into a converted voice signal using the learned voice signal conversion model. The converted voice signal is output to the voice signal output control unit203.

The voice signal output control unit203controls the operation of the interface unit22. The voice signal output control unit203controls the operation of the interface unit22to cause the interface unit22to output the converted voice signal.

FIG.11is a flowchart showing an example of a flow of processing executed by the voice signal conversion device2in the embodiment. The control unit20acquires a conversion target voice signal input to the interface unit22(step S501). Next, the control unit20converts the conversion target voice signal into a converted voice signal using a learned voice signal conversion model stored in the storage unit23(step S502). Next, the control unit20controls the operation of the interface unit22to cause the converted voice signal to be output to an output destination (step S503). The output destination is, for example, an external device such as a speaker.

The voice signal generation system100of the embodiment configured in this manner performs learning using conversion source speaker information, conversion destination speaker information, and speaker identification information and obtains a learned voice signal conversion model. Therefore, the voice signal generation system100can convert a voice represented by a voice signal that is a conversion target into a voice signal representing a voice closer to the voice of the speaker represented by the conversion destination speaker information than a voice signal converted on the basis of only the conversion destination speaker information. Therefore, the voice signal generation system100can convert a voice having a more appropriate experience distribution even when there are many candidates for both an attribute of a conversion source and an attribute of a conversion destination.

First Modified Example

The objective function may include the extended adversarial loss function and does not necessarily include the cyclic loss function and the identity loss function. The objective function may be, for example, the extended adversarial loss function, may include the extended adversarial loss function and the cyclic loss function and may not include the identity loss function, or may include the extended adversarial loss function and the identity loss function and may not include the cyclic loss function.

Although cross entropy is used as a measure in the description of the extended adversarial loss function, it may be based on any measure such as L2distance and Wasserstine measurement. Although L1distance is used in the description of the cyclic loss function, it may be based on any measure such as L2distance. Although L1distance is used in the description of the identity loss function, it may be based on any measure such as L2distance.

Second Modified Example

The generation unit110does not necessarily use conversion source speaker information in first type data generation processing. This generation unit110has, for example, a configuration shown inFIG.12below.

FIG.12is a diagram showing an example of a functional configuration of the generation unit110in a second modified example. The generation unit110includes an encoder111and a decoder112.

The encoder111is a neural network having a convolution layer. The encoder111encodes first learning data. The encoder111includes a data acquisition unit113, a first characteristic extraction unit114, a second characteristic extraction unit115, an extraction result conversion unit116, and an encoding result output unit117. The data acquisition unit113acquires first learning data input to the encoder111. Specifically, the data acquisition unit113is an input layer of a neural network constituting the encoder111.

The first characteristic extraction unit114executes first characteristic extraction processing. The first characteristic extraction processing is processing for acquiring information representing characteristics of a first learning voice signal of the first learning data (hereinafter referred to as “characteristic information”). The first characteristic extraction processing is, for example, processing for sequentially executing short-time Fourier transform for respective predetermined periods in a time axis direction. The first characteristic extraction processing may be processing for extracting a Melcepstrum or conversion processing using a neural network. Specifically, the first characteristic extraction unit114is a circuit that executes the first characteristic extraction processing. Therefore, the first characteristic extraction unit114is one of intermediate layers of the neural network constituting the encoder111when the first characteristic extraction processing is conversion processing using the neural network.

The second characteristic extraction unit115executes second characteristic extraction processing. The second characteristic extraction processing is processing for executing convolution processing in machine learning on characteristic information. The convolution processing in machine learning is processing for extracting characteristics of a processing target from the processing target. Therefore, the second characteristic extraction processing is processing for extracting intonation indicating characteristics different from the characteristics indicated by characteristic information of the processing target of the first characteristic extraction processing among the characteristics of the first learning voice signal. That is, the second characteristic extraction processing is also processing for acquiring characteristic information similarly to the first characteristic extraction processing. Specifically, the second characteristic extraction unit115is a convolution layer of the neural network constituting the encoder111.

The extraction result conversion unit116executes extraction result conversion processing. The extraction result conversion processing converts an execution result of the second characteristic extraction processing according to extraction result conversion mapping on the basis of conversion destination speaker information. Extraction result conversion mapping is mapping updated according to an estimation result of the identification unit120, mapping according to the conversion destination speaker information, and mapping for converting only the execution result of the second characteristic extraction processing between the conversion destination speaker information and the execution result of the second characteristic extraction processing (that is, characteristic information). Specifically, the extraction result conversion unit116is one of intermediate layers of the neural network constituting the encoder111.

The extraction result conversion mapping executes affine transformation depending on at least the conversion destination speaker information on the execution result of the second characteristic extraction processing. The extraction result conversion mapping may be affine transformation depending on not only the conversion destination speaker information but also the conversion source speaker information. An example of affine transformation for the execution result of the second characteristic extraction processing is a function CIN represented by the following formula (5).

[Math.5]CIN⁡(f;c2)=γc2(f-μ⁡(f)σ⁡(f))+βc2(5)

In formula (5), the tensor f is characteristic information. More specifically, the tensor f is a feature amount tensor in which each element represents a feature amount related to the first learning data. The tensor f is a tensor of at least third floor. μ(f) represents an average value of values of elements in the tensor of each second floor with respect to the tensor of each second floor perpendicular in a predetermined one direction of the tensor f. Therefore, μ(f) is a C-dimensional vector if the number of elements in the predetermined one direction is C. The predetermined one direction is, for example, a direction indicating a channel of a feature amount tensor on the third floor of height×width×channel extracted by CNN. The σ(f) represents the standard deviation of the values of the elements in the tensor of each second floor from which the μ(f) has been acquired. For this reason, σ(f) is a vector having the same number of elements as that of μ(f). The coefficients γc2and βc2are parameters updated for each speaker indicated by conversion destination speaker information according to learning using the objective function L.

A series of processing of first characteristic extraction, the second characteristic extraction, and extraction result conversion processing is encoding of the first learning data by the encoder111.

The encoding result output unit117outputs the encoded first learning data to the decoder112. Specifically, the encoding result output unit117is an output layer of the neural network constituting the encoder111.

The decoder112generates first type generation data on the basis of an output result of the encoder111. Processing performed by the encoder111and the decoder112for generating the first type generation data on the basis of the first learning data is an example of first type data generation processing.

FIG.13is a flowchart showing an example of a flow of processing executed by the generation unit110in the second modified example. The data acquisition unit113acquires first learning data (step S601). Next, the first characteristic extraction unit114executes first characteristic extraction processing (S602). Next, the second characteristic extraction unit115executes second characteristic extraction processing on characteristic information obtained by first characteristic extraction processing in step S602(step S603). Next, the extraction result conversion unit116executes extraction result conversion processing on characteristic intonation obtained by second characteristic extraction processing of step S603(step S604). Next, the encoding result output unit117outputs information obtained by processing of step S604to the decoder (step S605). Next, the decoder112decodes the information output in step S605(step S606). Information obtained by decoding in step S606is first type generation data.

Processing of step S603and step S604may be repeatedly executed a plurality of times after first characteristic extraction processing is executed and before processing of step S605is executed. In this case, an execution target of second and subsequent second characteristic extraction processing is information obtained by extraction result conversion processing of the characteristic information extracted by immediately previous second characteristic extraction processing.

As shown inFIG.13, in encoding processing performed by the generation unit110in the second modified example, second characteristic extraction processing for characteristic information is executed. Accordingly, even when processing of step S603and step S604is executed a plurality of times, second characteristic extraction processing for the characteristic information is executed at least once in encoding processing performed by the generation unit110in the second modified example.

In the voice signal conversion model learning device1of the second modified example configured in this manner, when convolution processing is executed by the convolution layer, convolution processing is executed on information independent of conversion destination speaker information, and the execution result of convolution processing is converted depending on the conversion destination speaker information. Therefore, the voice signal conversion model learning device1of the second modified example can process intonation while maintaining the level of orthogonality between a space representing conversion destination speaker information and a space representing characteristic information, as compared to a technique for performing convolution including the conversion destination speaker information at the time of performing convolution processing. Here, orthogonality means a degree to which an expression space representing a voice signal and an expression space representing information indicating a conversion destination are orthogonal to each other.

The lower orthogonality is, the more the boundary between conversion destination speaker information and characteristic information included in one piece of information becomes unclear, and thus the amount of calculation increases in the case of encoding or decoding. Therefore, the voice signal conversion model learning device1of the second modified example capable of maintaining orthogonality can reduce the amount of calculation as compared to the technique for performing convolution including conversion destination speaker information at the time of performing convolution processing.

Further, in the voice signal conversion model learning device1of the second modified example configured as described above, it is possible to efficiently execute conversion of characteristic information different for respective speakers for the following reason. The reason for this is as follows. In order to realize many-to-many voice conversion by a single model, it is important that conversion of characteristic intonation different for respective speakers can be performed selectively depending on speaker information. However, in the conventional technique for performing convolution including conversion destination speaker information at the time of executing convolution processing, speaker information is used as a part of information to be convoluted, and thus selection of characteristic information depending on the speaker information is not directly executed.

On the other hand, in the voice signal conversion model learning device1of the second modified example, it is possible to directly express the strength of characteristic information for each speaker using parameters which can be learned, as in the affine transformation represented by formula (5). Therefore, the voice signal conversion model learning device1of the second modified example can efficiently execute conversion of characteristic information different for respective speakers as compared to the conventional technique. Meanwhile, the parameters which can be learned indicate coefficients γc2and βc2in the case of formula (5). That is, the voice signal conversion model learning device1of the second modified example configured in this manner can provide a technique for curbing an increase in the number of parameters used in a mathematical model representing voice conversion.

The generation unit110of the second modified example may be applied to any device as long as it is a device (hereinafter referred to as a “general generation network”) including a generator such as an adversarial generation network (GAN), and a discriminator, which are updated by learning, in which the generator outputs a value on the basis of conversion destination speaker information. In such a case, the generation unit110of the second modified example operates as a generation unit included in the general generation network. For example, the StarGAN of NPL 1 is an example of the general generation network, and StarGAN of NPL 1 may use the generation unit110of the second modified example instead of the generator included in StarGAN of NPL 1. In this case, attribute in NPL 1 is conversion destination speaker information in the voice signal generation system100.

Third Modified Example

Although the voice signal generation system100for converting a speaker with respect to conversion of a voice signal has been described, voice conversion of the voice signal generation system100may not necessarily be conversion of a speaker as long as the attribute of a voice signal can be converted. In such a case, in the voice signal generation system100, conversion source attribute intonation is used instead of conversion source speaker intonation, conversion destination attribute information is used instead of conversion destination speaker information, and attribute identification information is used instead of speaker identification information. The conversion source attribute information indicates an attribute to which the first learning voice belongs. The conversion destination speaker information indicates an attribute which is a preset attribute to which the first type generation voice belongs. The attribute identification information indicates an attribute which is a preset attribute to which the second learning voice belongs. Random speaker information is information indicating an attribute randomly determined by the determination unit130among a plurality of attributes prepared in advance. In such a case, voice estimation processing is processing for estimating whether or not the voice signal is a voice signal having an attribute to which a processing target belongs and representing a voice actually uttered.

Although a speaker is one of attributes, the attribute may be, for example, a gender as another attribute. In such a case, the voice signal generation system100converts a voice signal of a male voice into a voice signal of a female voice, for example. The attribute may be, for example, emotion. In such a case, the voice signal generation system100converts a voice representing a joy feeling into a voice representing a sad feeling, for example. The attribute may be, for example, a type of pronunciation. In such a case, the voice signal generation system100converts non-native English into native English, for example. The attribute may be an attribute relating to the quality of voice. The attribute relating to the quality of voice is, for example, an attribute indicating either a synthetic voice or a natural voice. The natural voice is a vocal sound actually uttered by a person, and the synthetic voice is a voice generated by a device such as a computer. In such a case, the voice signal generation system100converts, for example, a synthetic voice into a natural voice.

(Experimental Results of Experiments Using Voice Signal Generation System100Obtained by Combining Embodiment and First to Third Modified Examples)

FIG.14is a diagram of results of experiments (hereinafter referred to as “first experiments”) showing differences in Mel-cepstral distortion (MCD) and differences in a modulation spectra distance (MSD) due to differences in an objective function used for learning a voice signal conversion model.

In the first experiments, an experiment (hereinafter referred to as a “(1-1)th experiment”) of learning the voice signal conversion model using a speaker identification loss function Lcls, an adversarial loss function Ladv, a cyclic loss function L′cyc, and an identity loss function L′idas objective functions L was performed. In the first experiments, an experiment (hereinafter referred to as a (1-2)th experiment”) of learning the voice signal conversion model using an adversarial loss function Lt-adv, the cyclic loss function L′cyc, and the identity loss function L′idas objective functions L was performed. In the first experiments, an experiment (hereinafter referred to as a (1-3)th experiment”) of learning the voice signal conversion model using the speaker identification loss function Lcls, the adversarial loss function Lt-adv, the cyclic loss function L′cyc, and the identity loss function L′idas objective functions L was performed. In the first experiments, an experiment (hereinafter referred to as a (1-4)th experiment”) of learning the voice signal conversion model using the function represented by formula (1) as an objective function L was performed. In the first experiments, λcycwas 10 and λidwas 1.

The speaker identification loss function Lclsis represented by the sum of formula (6) and formula (7) below, the adversarial loss function Ladvis represented by formula (8) below, the adversarial loss function Lt-advis represented by formula (9) below, the cyclic loss function L′cycis represented by formula (10), and the identity loss function L′idis represented by formula (11).
[Math. 6]
Lclsr=(x,c1)˜P(x,c1)[−logC(c1|x)]  (6)
[Math. 7]
Lclsf=x˜P(x),c2˜P(c2)[−logC(c2|G(x,c2))]  (7)
[Math. 8]
Ladv=x˜P(x)[logD(x)]+(x,c2)˜P(x,c2)[log(1−D(G(x,c2)))]  (8)
[Math. 9]
Lt_adv=(x,c1)˜P(x,c1)[logD(x,c1)]+(x,c2)˜P(x,c2)[log(1−D(G(x,c2),c2)))]  (9)
[Math. 10]
L′cyc=(x,c1)˜P(x,c1),c2˜P(c2)[∥x−G(G(x,c2),c1)∥1]   (10)
[Math. 11]
L′id=(x,c1)˜P(x,c1)[∥G(x,c1)−x∥1]  (11)

In addition, x and c1of the right side of formula (6) represent S′0and C′1of the second learning data in order. In addition, x and c2of the right side of formula (7) represent S0and C2of the first learning data in order. Further, x of the first term of the right side of formula (8) represents S′0of the second learning data. Further, x and c2of the second team of the right side of formula (8) represent S0and C2of the first learning data in order. In addition, x and c1of the first term of the right side of formula (9) represent S′0and C′1of the second learning data in this order. Further, x and c2of the second term of the right side of formula (9) represent S0and C2of the first learning data in order. In addition, x, c1and c2of the right side of formula (10) represent S0, C1and C2of the first learning data in order. Further, x and c1of the right side of formula (11) represent S0and C1of the first learning data in order.

FIG.14shows that the experimental results of the (1-4)th experiment are the smallest MCD and the smallest MSD. This indicates that the learned voice signal conversion model obtained by learning using the objective function L represented by formula (1) is capable of converting into a voice signal representing a voice closer to a voice of a speaker indicated by conversion destination speaker information than the learned voice signal conversion model obtained by the other learning inFIG.14. InFIG.14, “Lcls” represents the (1-1)th experiment, “Lt-adv” represents the (1-2)th experiment, “Lt-adv+Lcls(StarGAN-VC)” represents the (1-3)th experiment, and “Lst-adv(StarGAN-VC2)” represents the (1-4)th experiment.

FIG.15is a diagram of results of experiments (hereinafter referred to as “second experiments”) showing differences in MCD and differences in MSD due to differences in the functional configuration of the generation unit110used for learning a voice signal conversion model.

InFIG.15, “Channel-wise (StarGAN-VC)” is a technique of performing convolution including conversion destination speaker information at the time of executing convolution processing by the encoder111. That is, the results of the row of “Channel-wise (StarGAN-VC)” represent MSD and MCD of a learned voice signal conversion model obtained by learning of performing convolution including conversion destination speaker information at the time of executing convolution processing by the encoder111.

InFIG.15, “Modulation-based (StarGAN-VC2)” is a technique of encoding the first learning data by processing shown inFIG.13. That is, the results of the row of “Modulation-based (StarGAN-VC2)” represent MSD and MCD of a learned voice signal conversion model obtained by learning using the generation unit110of the second modified example.

FIG.15shows that values of MCD are substantially the same in “Channel-wise (StarGAN-VC)” and “Modulation-based (StarGAN-VC2).”FIG.15shows that the value of MSD of “Modulation-based (StarGAN-VC2)” is smaller than the value of MSD of “Channel-wise (StarGAN-VC).” Therefore,FIG.15shows that the learned voice signal conversion model obtained by learning of “Modulation-based (StarGAN-VC2)” is capable of converting into a voice signal representing a voice closer to a voice of a speaker indicated by conversion destination speaker information than the learned voice signal conversion model obtained by learning of “Channel-wise (StarGAN-VC).”

FIG.16is a diagram of results of experiments (hereinafter referred to as “third experiments”) showing a mean opinion score (MOS) due to differences in a combination of an objective function used for learning a voice signal conversion model and the function configuration of the generation unit110. The MOS has a maximum evaluation of 5 and a minimum evaluation of 1.

InFIG.16, “StarGAN-VC2” represents a learned voice signal conversion model obtained by the voice signal conversion model learning device1in which the objective function is represented by formula (1) and the generation unit110serves as a functional unit that encodes the first learning data through processing shown inFIG.13.

InFIG.16, “StarGAN-VC” represents a learned voice signal conversion model obtained by a comparison target device. The comparison target device is different from the voice signal conversion model learning device1in that objective functions are represented by formula (6), formula (7), formula (8), and a linear sum of formula (10) and formula (11), and the generation unit110performs convolution including conversion destination speaker information at the time of executing convolution processing by the encoder111.

InFIG.16, “Inter gender” indicates a MOS for conversion of a voice signal between different genders by the learned voice signal conversion model. InFIG.16, “Intra gender” indicates a MOS for conversion of a voice signal between the same genders by the learned voice signal conversion model. InFIG.16, “Ail” is the sum of results of “Inter gender” and “Intra gender.”

FIG.16shows that a voice signal obtained by “StarGAN-VC2” has a higher MOS than a voice signal obtained by “StarGAN-VC” in all of “Inter gender,” “Intra gender” and “All.”

FIG.17is a diagram of results of experiments (hereinafter referred to as “fourth experiments”) showing average preference scores on speaker similarity due to differences in a combination of an objective function used for learning a voice signal conversion model and the function configuration of the generation unit110. The average preference scores on speaker similarity are an experiment in which a subject determines whether a voice generated by a learned voice signal conversion model is similar to an actual voice of a speaker that is a voice signal conversion destination, or not.

InFIG.17, “Fair” indicates that the generated voice is not similar to the actual voice.FIG.17shows that a voice of a voice signal obtained by “StarGAN-VC2” is closer to a voice of a speaker that is a conversion destination in all of “Inter gender,” “Intra gender” and “All.”

Fourth Modified Example

The identification unit120may further execute speaker identification processing. Speaker identification processing is executed when the second learning data is input to the identification unit120. Speaker identification processing estimates a speaker with respect to a second learning voice signal S′0of the input second learning data. Specifically, speaker identification processing is executed by a neural network for executing speaker identification processing. The neural network for executing speaker identification processing is updated on the basis of the value of formula (6) or formula (7) acquired by the loss acquisition unit140. More specifically, the neural network for executing speaker identification processing is updated such that the value of formula (6) decreases on the basis of the value of formula (6) acquired by the loss acquisition unit140when the second learning data is input to the identification unit120. In addition, when the first learning data is input to the generation unit110, the neural network for executing speaker identification processing is updated such that the value of formula (7) decreases on the basis of the value of formula (7) acquired by the loss acquisition unit140. Further, when the first learning data is input to the generation unit110, the generation unit110performs learning such that the value of formula (7) decreases on the basis of the value of formula (7) acquired by the loss acquisition unit140. The function represented by C in formula (6) indicates speaker identification processing. Further, when speaker identification processing is executed, the identification unit120may use or may not use either or both of conversion source speaker information and conversion destination speaker information. When either or both of the conversion source speaker information and the conversion destination speaker information are not used, the identification unit120estimates whether or not a voice signal represented by identification input data is a voice signal representing an actually uttered vocal sound without using either or both of the conversion source speaker information and the conversion destination speaker information.

Fifth Modified Example

The second modified example described that conversion source speaker information is not necessarily used for the generation unit110. When the generation unit110does not use conversion source speaker information, the identification unit120may use or may not use the conversion source speaker information. When the conversion source speaker information is not used, the identification unit120estimates whether or not a voice signal represented by identification input data is a voice signal representing an actually uttered vocal sound without using the conversion source speaker information.

Sixth Modified Example

The processing executed in the second characteristic extraction processing is not necessarily convolution processing. The processing executed in the second characteristic extraction processing may be any processing as long as it is processing performed by a neural network, for example, a recurrent neural network or a fully connected neural network. The second characteristic extraction processing is an example of characteristic processing.

Seventh Modified Example

The first type data generation processing is an example of generation processing. The first learning data is an example of an input voice signal. The first type generation data is an example of a conversion destination voice signal. The natural voice estimation processing is an example of voice estimation process. The speaker estimation processing is an example of attribute estimation processing. The first type generation voice is an example of a conversion destination voice. Further, the first learning voice is an example of an input voice.

The voice signal conversion model learning device1may be implemented using a plurality of information processing devices connected via a network so as to be capable of communication. In this case, respective functional units included in the voice signal conversion model learning device1may be implemented in a distributed manner across the plurality of information processing devices.

The voice signal conversion device2may be implemented using a plurality of information processing devices connected via a network so as to be capable of communication. In this case, respective functional units included in the voice signal conversion device2may be implemented in a distributed manner across the plurality of intonation processing devices.

Some or all of the functions of the voice signal generation system100may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). A program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. The program may be transmitted via an electric communication line.

Although the embodiments of the present invention have been described in detail with reference to the drawings, specific configurations are not limited to these embodiments, and designs and the like within a range that does not deviating from the gist of the present invention are also included.

REFERENCE SIGNS LIST

100Voice signal generation system,1Voice signal conversion model learning device,2Voice signal conversion device,10Control unit,11Input unit,12Interface unit,13Storage unit,14Output unit,101Managed unit,102Management unit,110Generation unit,120Identification unit,130Determination unit,140Loss acquisition unit20Control unit,21Input unit,22Interface unit23Storage unit,24Output unit,201Conversion target acquisition unit,202Conversion unit,203Voice signal output control unit,91Processor,92Memory,93Processor,94Memory