Patent ID: 12217755

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG.1is a diagram illustrating a system configuration of a cross-modal processing system100according to the present invention. The cross-modal processing system100is a system that uses correlations between human voices and human faces to perform a process of converting an input voice to a timbre conforming to an input face image, and also generating a face image conforming to the timbre of an input voice. The cross-modal processing system100is provided with a voice conversion device10, an image generation device20, a voice conversion learning device30, and an image generation learning device40.

The voice conversion device10is a device that converts a voice to a timbre conforming to a face image, on the basis of an input voice and a face image.

The image generation device20is a device that generates a face image conforming to the timbre of an input voice. The voice conversion learning device30is a device that trains a neural network (NN) used by the voice conversion device10.

The image generation learning device40is a device that trains a NN used by the image generation device20.

Next, a specific configuration of the voice conversion device10, the image generation device20, the voice conversion learning device30, and the image generation learning device40will be described.

The voice conversion device10is provided with components such as a central processing unit (CPU), a memory, and an auxiliary storage device connected by a bus, and executes a voice conversion program. By executing the voice conversion program, the voice conversion device10functions as a device provided with a voice input unit11, a linguistic information extraction unit12, an image input unit13, an appearance feature extraction unit14, and a converted voice generation unit15. Note that all or some of the functions of the voice conversion device10may also be achieved using hardware, such as an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field-programmable gate array (FPGA), or a graphics processing unit (GPU). In addition, the voice conversion program may also be recorded onto a computer-readable recording medium. The computer-readable recording medium refers to a portable medium such as a flexible disk, a magneto-optical disc, ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system, for example. The voice conversion program may also be transmitted and received over an electrical communication channel.

The voice input unit11accepts the input of a conversion source voice signal. The voice input unit11outputs the input conversion source voice signal to the linguistic information extraction unit12.

The linguistic information extraction unit12accepts the input of the conversion source voice signal output from the voice input unit11. The linguistic information extraction unit12extracts linguistic information corresponding to utterance content from the input conversion source voice signal. For example, the linguistic information extraction unit12is configured as a NN that accepts a conversion source voice signal as input, and extracts linguistic information corresponding to utterance content from the conversion source voice signal. The linguistic information extraction unit12outputs the extracted linguistic information to the converted voice generation unit15.

The image input unit13accepts the input of a face image obtained by capturing a human face. The image input unit13outputs the input face image to the appearance feature extraction unit14.

The appearance feature extraction unit14accepts the input of the face image output from the image input unit13. The appearance feature extraction unit14extracts appearance features from the input face image. For example, the appearance feature extraction unit14is configured as a NN that accepts a face image as input, and extracts appearance features from the face image. Here, appearance features designate features related to the look of a person's face specified from a face image of the person. The appearance feature extraction unit14outputs the extracted appearance features to the converted voice generation unit15.

The converted voice generation unit15accepts the input of linguistic information and appearance features. On the basis of the input linguistic information and appearance features, the converted voice generation unit15generates a converted voice. For example, the converted voice generation unit15is configured as a NN that accepts linguistic information and appearance features as input, and generates a converted voice. The converted voice generation unit15outputs the generated converted voice to the image generation device20.

The image generation device20is provided with components such as a CPU, a memory, and an auxiliary storage device connected by a bus, and executes an image generation program. By executing the image generation program, the image generation device20functions as a device provided with a voice input unit21, a timbre feature extraction unit22, and an image generation unit23. Note that all or some of the functions of the image generation device20may also be achieved using hardware, such as an ASIC, a PLD, an FPGA, or a GPU. In addition, the voice conversion program may also be recorded onto a computer-readable recording medium. The computer-readable recording medium refers to a portable medium such as a flexible disk, a magneto-optical disc, ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system, for example. The voice conversion program may also be transmitted and received over an electrical communication channel.

The voice input unit21accepts the input of a voice signal used in the generation of a face image. For example, the voice input unit21accepts the input of the converted voice generated by the voice conversion device10. Note that the voice input unit21may also accept the input of a voice other than the converted voice generated by the voice conversion device10. A voice other than the converted voice generated by the voice conversion device10may be a voice input using a voice input device such as a microphone for example, or a voice stored in advance. The voice input unit11outputs the input voice signal to the timbre feature extraction unit22.

The timbre feature extraction unit22accepts the input of the voice signal output from the voice input unit21. The timbre feature extraction unit22extracts timbre features expressing features related to vocal timbre from the input voice signal. For example, the timbre feature extraction unit22is configured as a NN that accepts a voice signal as input, and extracts timbre features expressing features related to vocal timbre from the input voice signal. The timbre feature extraction unit22outputs the extracted timbre features to the image generation unit23.

The image generation unit23accepts timbre features and appearance features as input. The image generation unit23generates a face image on the basis of the input timbre features and appearance features. For example, the image generation unit23is configured as a NN that accepts timbre features and appearance features as input, and generates a face image. The image generation unit23outputs the generated face image to an external device. Note that the image generation unit23may accept the input of the appearance features extracted by the appearance feature extraction unit14of the voice conversion device10, or accept appearance features input manually.

FIG.2is a schematic block diagram illustrating a functional configuration of the voice conversion learning device30according to the embodiment.

The voice conversion learning device30is provided with components such as a CPU, memory, and an auxiliary storage device connected by a bus, and executes a learning program. By executing the learning program, the voice conversion learning device30functions as a device provided with a voice input unit31, a linguistic information extraction unit32, an image input unit33, an appearance feature extraction unit34, a converted voice generation unit35, and a learning unit36. Note that all or some of the functions of the voice conversion learning device30may also be achieved using hardware, such as an ASIC, a PLD, an FPGA, or a GPU. In addition, the learning program may also be recorded onto a computer-readable recording medium. The computer-readable recording medium refers to a portable medium such as a flexible disk, a magneto-optical disc, ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system, for example. The learning program may also be transmitted and received over an electrical communication channel.

The voice input unit31accepts the input of a conversion source voice signal. The voice input unit31outputs the input conversion source voice signal to the linguistic information extraction unit32.

The linguistic information extraction unit32accepts the input of the conversion source voice signal output from the voice input unit31. The linguistic information extraction unit32extracts linguistic information corresponding to utterance content from the input conversion source voice signal. For example, the linguistic information extraction unit32is configured as a NN that accepts a conversion source voice signal as input, and extracts linguistic information corresponding to utterance content from the conversion source voice signal. The linguistic information extraction unit32outputs the extracted linguistic information to the converted voice generation unit35and the learning unit36.

The image input unit33accepts the input of a face image obtained by capturing a human face. The image input unit33outputs the input face image to the appearance feature extraction unit34.

The appearance feature extraction unit34accepts the input of the face image output from the image input unit33. The appearance feature extraction unit34extracts appearance features from the input face image. For example, the appearance feature extraction unit34is configured as a NN that accepts a face image as input, and extracts appearance features from the face image. The appearance feature extraction unit34outputs the extracted appearance features to the converted voice generation unit35and the learning unit36.

The converted voice generation unit35accepts the input of linguistic information and appearance features. The converted voice generation unit35generates a converted voice on the basis of the input linguistic information and the appearance features. For example, the converted voice generation unit35is configured as a NN that accepts linguistic information and appearance features as input, and generates a converted voice. The converted voice generation unit35outputs the generated converted voice to the learning unit36.

The learning unit36accepts the input of linguistic information, appearance features, and a converted voice. The learning unit36performs learning on the basis of the input linguistic information, appearance features, and converted voice such that when linguistic information z (=SpeechEnc(x)) as the output of the linguistic information extraction unit32and appearance features c (=FaceEnc(y)) as the output of the appearance feature extraction unit34are input into the converted voice generation unit35, the output converted voice {circumflex over ( )}x (where {circumflex over ( )} is written above x)=SpeechDec(SpeechEnc(x); FaceEnc(y)) is as close as possible to the voice signal x input into the linguistic information extraction unit32. This is because, in the case where the speaker of the voice signal x and the person of the face image y from the appearance features c originate are the same person, it is desirable for the linguistic information extraction unit32and the converted voice generation unit35to reconstruct the input voice as-is. On the basis of the learning result, the learning unit36updates parameters of the linguistic information extraction unit12, the appearance feature extraction unit14, and the converted voice generation unit15of the voice conversion device10.

FIG.3is a schematic block diagram illustrating a functional configuration of the image generation learning device40according to the embodiment.

The image generation learning device40is provided with components such as a CPU, memory, and an auxiliary storage device connected by a bus, and executes a learning program. By executing the learning program, the image generation learning device40functions as a device provided with an image input unit41, an appearance feature extraction unit42, a voice input unit43, a timbre feature extraction unit44, an image generation unit45, and a learning unit46. Note that all or some of the functions of the voice conversion learning device30may also be achieved using hardware, such as an ASIC, a PLD, an FPGA, or a GPU. In addition, the learning program may also be recorded onto a computer-readable recording medium. The computer-readable recording medium refers to a portable medium such as a flexible disk, a magneto-optical disc, ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system, for example. The learning program may also be transmitted and received over an electrical communication channel.

The image input unit41accepts the input of a face image obtained by capturing a human face. The image input unit41outputs the input face image to the appearance feature extraction unit42.

The appearance feature extraction unit42accepts the input of the face image output from the image input unit41. The appearance feature extraction unit42extracts appearance features from the input face image. For example, the appearance feature extraction unit42is configured as a NN that accepts a face image as input, and extracts appearance features from the face image. The appearance feature extraction unit42outputs the extracted appearance features to the image generation unit45and the learning unit46.

The voice input unit43accepts the input of a voice signal used in the generation of a face image. For example, the voice input unit43accepts the input of the converted voice generated by the voice conversion device10. The voice input unit43outputs the input voice signal to the timbre feature extraction unit44.

The timbre feature extraction unit44accepts the input of the voice signal output from the voice input unit43. The timbre feature extraction unit44extracts timbre features expressing features related to vocal timbre from the input voice signal. For example, the timbre feature extraction unit44is configured as a NN that accepts a voice signal as input, and extracts timbre features expressing features related to vocal timbre from the input voice signal. The timbre feature extraction unit44outputs the extracted timbre features to the image generation unit45and the learning unit46.

The image generation unit45accepts timbre features and appearance features as input. The image generation unit45generates a face image on the basis of the input timbre features and appearance features. For example, the image generation unit45is configured as a NN that accepts timbre features and appearance features as input, and generates a face image. The image generation unit45outputs the generated face image to the learning unit46.

The learning unit46accepts the input of appearance features, timbre features, and a face image. The learning unit46performs learning on the basis of the input appearance features and face image such that for any given face image y, when the appearance features c=FaceEnc(y) as the output of the appearance feature extraction unit42are input into the image generation unit45, the output face image {circumflex over ( )}y (where {circumflex over ( )} is written above y)=FaceDec(FaceEnc(y)) is as close as possible to the face image y input into the appearance feature extraction unit42. This is because it is desirable for the appearance features extracted by the appearance feature extraction unit42to retain as much information from the original face image as possible. On the basis of the learning result, the learning unit46updates parameters of the image generation unit23of the image generation device20.

Additionally, the learning unit46performs learning on the basis of the input timbre features and face image such that when the converted voice {circumflex over ( )}x (where {circumflex over ( )} is written above x)=SpeechDec(z, c) as the output of a converted voice generation unit is input into the timbre feature extraction unit44, the output timbre features {circumflex over ( )}c (where {circumflex over ( )} is written above c)=VoiceEnc(SpeechDec(z, c)) is as close as possible to the appearance features c used as the input into the converted voice generation unit. This is because it is desirable for the voice output by the converted voice generation unit to be influenced by the appearance features as strongly as possible. On the basis of the learning result, the learning unit46updates parameters of the timbre feature extraction unit22of the image generation device20.

Principle of Embodiment According to Present Invention

<Variational Autoencoder (VAE)>

A VAE is a probabilistic generative model containing encoder and decoder neural networks (NNs). In an ordinary autoencoder (AE), the encoder is a NN for the purpose of compressing input data x into a latent variable z of lower dimensionality, and the decoder is a NN for the purpose of restoring the data x from the latent variable z. In a VAE, the decoder is modeled as a NN that outputs probability distribution parameters (in the case of a Gaussian distribution, the mean and the variance) of a conditional distribution pθ(x|z) of the data x. Here, θ represents the parameters of the NN. The encoder corresponding to the above should produce a posterior distribution expressed as in the following Expression (1), but obtaining the posterior distribution analytically is difficult.

[Math.1]p⁡(z|x)=pθ(x|z)⁢p⁡(z)∫pθ(x|z)⁢p⁡(z)⁢dzExpression⁢(1)

Accordingly, an auxiliary distribution qφ(z|x) for the purpose of approximating the posterior distribution p(z|x) is newly introduced, and the goal of the VAE is to obtain an auxiliary distribution qφ(z|x) and a conditional distribution pθ(x|z) such that the auxiliary distribution qφ(z|x) and the true posterior distribution are as close as possible. The Kullback-Leibler (KL) divergence between the auxiliary distribution qφ(z|x) and the posterior distribution p(z|x) is expressed as in the following Expression (2).
[Math. 2]
KL[qφ(z|x)∥pθ(z|x)]=logp(x)−z˜qφ(z|x)[logpθ(x|z)]+KL[qφ(z|x)∥p(z)]  Expression (2)

If the unknown parameters θ, φ could be obtained so as to increase Ez˜qφ(z|x)[log pθ(x|z)]−KL[qφ(z|x)∥p(z)], an encoder and a decoder that are consistent with each other could be obtained. Here, as an example, suppose that μφ(x), log σ2φ(x) are the output of an encoder NN with the parameter φ, μθ(z), log σ2θ(z) are the output of a decoder NN with the parameter θ, and q(z|x) and p(x|z) are each Gaussian distributions having a mean and a variance. Also, let p(z) be the standard Gaussian distribution. At this point, assumptions are made as in the following Expressions (3) to (5).
[Math. 3]
qφ(z|x)=(z|μϕ(x),diag(σϕ2(x)))  Expression (3)
[Math. 4]
pθ(x|z)=(x|μθ(z),diag(σθ2(z)))  Expression (4)
[Math. 5]
p(z)=(z|0,I)  Expression (5)

If assumptions are made as in the above Expressions (3) to (5), the second term of Expression (2) represents an index expressing the reconstruction error of the input x by the encoder and the decoder, while the third term of Expression (2) is an index expressing the divergence between the distribution of the output z of the encoder and the standard Gaussian distribution p(z). Consequently, decreasing Expression (2) means obtaining an autoencoder in which the elements of the latent variable z are as uncorrelated as possible. Note that the KL divergence is a non-negative value. Consequently, the criterion Ez˜qφ(z|x)[log pθ(x|z)]−KL[qφ(z|x)∥p(z)] described above is the lower bound of the logarithmic marginal likelihood log p(x). From the above, the learning criterion that should be maximized when given a learning sample x˜p(x) is expressed as in the following Expression (6).
[Math. 6]
(ϕ,θ)=x˜p(x)[z˜qϕ(z|x)[logpθ(x|z)]−KL[qϕ(z|x)∥p(z)]]  Expression (6)

In Expression (6), Ex˜p(x)[●] means the sample mean of all data samples. A conditional VAE (CVAE) is an extension of the VAE in a form such that an auxiliary variable c is inputtable into the encoder and the decoder as in the following Expressions (7) and (8).
[Math. 7]
qϕ(z|x,c)=(z|μϕ(x,c),diag(σ2ϕ(x,c)))  Expression (7)
[Math. 8]
pθ(x|z,c)=(x|μθ(z,c),(diag(σθ2(z,c)))  Expression (8)

At this point, the learning criterion that should be maximized when given a learning sample (x, c)˜p(x, c) is expressed as in the following Expression (9).
[Math. 9]
(ϕ,θ)=c˜p(c),x˜pϕ(x|c)[z˜qϕ(z|x,c)[logpθ(x|z c)]−KL[qϕ(z|x,c)∥p(z)]]  Expression (9)

In Expression (9), Ec˜p(c), x˜pθ(x|c)[●] means the sample mean of all data samples for every attribute.

Timbre conversion by CVAE (conventional method)
[Math. 10]
xϵD×NExpression (10)

Provided that the above Expression (10) is a vector series of acoustic features (for example, mel-frequency cepstral coefficients) in units of utterances and c is an attribute code, the problem of timbre conversion to the attribute c can be formulated by the CVAE. Provided that attribute-labeled learning data is {xm, cm}Mm=1, the encoder is trained as a function that uses the attribute code cnas a cue to convert an acoustic feature xnto a latent variable zn, while the decoder is trained as a function that uses the attribute code cnas a cue to reconstruct an acoustic feature {circumflex over ( )}xn(where {circumflex over ( )} is written above x) from the latent variable zn. After learning is completed, by inputting an acoustic feature series of a voice to be converted into the encoder and then inputting the generated latent variable together with a target attribute code into the decoder to reconstruct the acoustic feature series, an acoustic feature series having the target attribute can be obtained. Thereafter, the above result is converting into a time-domain signal to obtain a converted voice.

Hereinafter, details of the present invention using the VAE and CVAE illustrated above will be described.

An acoustic feature series x of the voice of a certain speaker and a face image y of the speaker are expressed by the following Expressions (11) and (12), respectively.
[Math. 11]
x=[x1. . . xN]ϵD×NExpression (11)
[Math. 12]
yϵI×JExpression (12)

In Expression (11), D represents the dimensionality of the acoustic feature vector, and N represents the length of the acoustic feature series. Also, in Expression (12), I and J represent the image size. At this point, consider a CVAE that generates the acoustic feature series x and a VAE that generates the face image y. Consider using the VAE and the CVAE to model a joint distribution p(x, y) of the acoustic feature series x and the face image y. The encoder of the acoustic feature series x takes a role of extracting a latent variable series z from an acoustic feature series of an input voice. The decoder of the acoustic feature series x takes a role of reconstructing an acoustic feature series on the basis of the latent variable series z and the auxiliary variable c. If the latent variable series z may be considered to correspond to linguistic information, the encoder may be considered to be the linguistic information extraction unit12and the decoder may be considered to be (the acoustic model of) the converted voice generation unit15. In this case, it is desirable for the auxiliary variable c to include information corresponding to the target timbre.

On the other hand, the encoder of the face image y takes the role of the appearance feature extraction unit14that extracts appearance features from an input face image. The decoder of the face image y may be considered to be the image generation unit23that reconstructs a face image from appearance features. Accordingly, the latent variable extracted by the appearance feature extraction unit14such that information corresponding to the target timbre in the converted voice generation unit15is determined by the face image y may be considered to be the auxiliary input c of the converted voice generation unit15. In addition, to keep the influence of the auxiliary input c from being lost in the process of decoding the acoustic feature series x, it is conceivable to introduce an encoder that takes a role of reconstructing the auxiliary input c from the output of the decoder of the acoustic feature series x. The encoder is considered to be the timbre feature extraction unit22that retrieves information related to a timbre consistent with the appearance features of an input image from an input voice. The above is the basic approach of the method according to the present invention.

Hereinafter, the basic approach above will be formulated specifically. The converted voice generation unit15and the image generation unit23are NNs that output parameters of conditional distributions pθaud(x|z, c) and pθvis(y|c), respectively. Here, in the converted voice generation unit15and the image generation unit23, aud and vis are subscripts of θ. Also, the linguistic information extraction unit12and the appearance feature extraction unit14are NNs that output parameters of conditional distributions qφaud(z|x) and qφvis(c|y) respectively. Here, in the linguistic information extraction unit12and the appearance feature extraction unit14, aud and vis are subscripts of φ. The terms θaud, θvis, φaud, and φvisrepresent the parameters of each NN.

Here, the learning goal is to approximate the true posterior distribution of z, c derived from pθaud(x|z, c) and pθvis(y|c) using q(z, c|x, y)=qφaud(z|x) qφvis(c|y). Note that the true posterior distribution of z, c is expressed as in the following Expression (13).
[Math. 13]
p(z,c|x,y)∝pθaud(x|z,c)pθvis(y|c)  Expression (13)

The KL divergence between p(z, c|x, y) and q(z, cx, y) is expressed as in the following Expression (14).

[Math.14]KL[q⁡(z,c|x,y)||p⁡(z,c|x,y)]=log⁢p⁡(x,y)⁢-𝔼c~qϕvis(c|y),z~qϕaud(z|x)[log⁢pθaud(x|z,c)]⁢-𝔼c~qϕvis(c|y)[log⁢pθvis(y|c)]⁢+KL[qϕaud(z|x)||p⁡(z)]+KL[qϕvis(c|y)||p⁡(c)]Expression⁢(14)

Given a learning sample (x, y)˜pd(x, y), the learning criterion is expressed as in the following Expression (15).

[Math.15]𝒥⁡(θaud,ϕaud,θvis,ϕvis)=𝔼(x,y)~pd(x,y)⁢𝔼c~qϕvis(c|y),z~qϕaud(z|x)[log⁢pθaud(x|z,c)]+Ey~pd(y)⁢𝔼c~qϕvis(c|y)[log⁢pθvis(y|c)-Ex~pd(x)⁢KL[qϕaud(z|x)||p⁡(z)]-Ey~pd(y)⁢KL[qϕvis(c|y)||p⁡(c)]Expression⁢(15)

At this point, as an example, μφaud(x) and log σ2φaud(x) are the output of the NN of the linguistic information extraction unit12with the parameter φaud, μθaud(z, c) and log σ2θaud(z, c) are the output of the NN of the converted voice generation unit15with the parameter θaud, μφvis(y) and log σ2φvis(y) are the output of the NN of the appearance feature extraction unit14with the parameter φvis, μθvis(c) and log σ2θvis(c) are the output of the NN of the image generation unit23with the parameter θvis, and qφaud(z|x), pθaud(x|z, c), qφvis(c|y), and pθvis(y|c) are each Gaussian distributions having a mean and a variance. Also, let p(z) and p(c) be standard Gaussian distributions. At this point, assumptions are made as in the following Expressions (16) to (21).
[Math. 16]
qϕaud(z|x)=(z|μϕaud(x),diag(σϕaud2(x)))  Expression (16)
[Math. 17]
pθaud(x|z,c)=(x|μθaud(z,c),diag(σθaud2)))  Expression (17)
[Math. 18]
qϕvis(c|y)=(c|μϕvis(y),diag(σϕvis2(y)))  Expression (18)
[Math. 19]
pθvis(y|c)=(y|μθvis(c),diag(σθvis2(c)))  Expression (19)
[Math. 20]
p(z)=(z|0,I)  Expression (20)
[Math. 21]
p(c)=(c|0,I)  Expression (21)

If assumptions are made as in the above Expression (16) to (21), the first term of Expression (15) represents an index expressing the reconstruction error of the input x by the linguistic information extraction unit12and the converted voice generation unit15(a weighted square error between x and μθaud(z, c)), the second term of Expression (15) represents an index expressing the reconstruction error of the input y by the appearance feature extraction unit14and the image generation unit23(a weighted square error between y and μθvis(c)), and the third and fourth terms of Expression (15) represent indices expressing the divergence from the standard Gaussian distributions of qφaud(z|x) and qφvis(c|y).

Meanwhile, in the model described above, depending on the functional complexity and expressive ability of the linguistic information extraction unit12and the converted voice generation unit15, the possibility that pθaud(x|z, c) might be trained independently of the auxiliary input c cannot be ruled out. For example, the case where the conversion process with respect to the input x by the linguistic information extraction unit12and the converted voice generation unit15results in an identity mapping is easy to understand. In this case, the model is capable of expressing any given input x without relying on the auxiliary variable c. Consequently, the resulting situation is one in which pθaud(x|z, c) becomes pθaud(x|z, c)=pθaud(x|z), independent of the auxiliary variable c. With a model trained in this way, the input voice is generated from the converted voice generation unit15as-is without being influenced by the auxiliary variable c, and the conversion effect is undesirable. To avoid a situation like the above, it is conceivable to adopt a learning method that accounts for mutual information between the auxiliary variable c and the output of the converted voice generation unit15so that the influence of the auxiliary variable c on the converted voice generation unit15is not lost. Mutual information is expressed as in the following Expression (22).

[Math.22]ℓ⁡(θaud)=∫∫p⁡(c′,x)⁢log⁢p⁡(c′,x)p⁡(c′)⁢p⁡(x)⁢dxdc′=∫∫p⁡(x)⁢p⁡(c′|x)⁢log⁢p⁡(c′|x)⁢dxdc′+H=𝔼x~pθaud(x|z,c),c′~p⁡(c|x)[log⁢p⁡(c′|x)]+HExpression⁢(22)

In Expression (22), H represents the entropy of the auxiliary variable c, which is treated as a constant herein. Although the mutual information can be written in a form that includes the posterior distribution p(c|x) of the auxiliary variable c as in Expression (22), describing this distribution analytically is difficult. For this reason, learning each NN to increase Expression (22) directly is difficult. Accordingly, an auxiliary distribution r(c|x) is introduced for the purpose of approximating the true posterior distribution p(c|x), and a lower bound of the first term of Expression (22) is given using the auxiliary distribution r(c|x) as in the following Expression (23).

[Math.23]𝔼x~pθaud(x|z,c),c′~p⁡(c⁢x)[log⁢p⁡(c′|x)]=𝔼x~pθaud(x|z,c),c′~p⁡(c|x)[log⁢r⁡(c′|x)⁢p⁡(c′|x)r⁡(c′|x)]≥𝔼x~pθaud(x|z,c),c′~p⁡(c|x)[log⁢r⁡(c′|x)]=𝔼x~pθaud(x|z,c)[log⁢r⁡(c|x)]Expression⁢(23)

The sign of the inequality holds when r(c|x)=p(c|x). Consequently, treating r(c|x) as an argument function and increasing the right side with respect to r(c|x) corresponds to approximating p(c|x) with r(c|x) and also approximating the mutual information with the right side. Therefore, by increasing qφaud(z|x), pθaud(x|z, c), qφvis(c|y) and pθvis(y|c) as well as the right side of Expression (23) with respect to the auxiliary distribution r(c|x), the mutual information can be increased indirectly. The auxiliary distribution r(c|x) expresses the appearance features or timbre features of the speaker of x, and therefore may be considered to be the timbre feature extraction unit22. In the present embodiment, the distribution parameter of the auxiliary distribution r(c|x) is expressed by a NN, and the parameter ψ is trained together with θaud, φaud, θvis, and φvis. Hereinafter, the auxiliary distribution r(c|x) expressed by the NN of the parameters iv is denoted rψ(c|x). For example, as a concrete form of rψ(c|x), μψ(x) and σ2ψ(x) are treated as the output of the NN of the timbre feature extraction unit22, and a Gaussian distribution having the above as the mean and the variance is defined in the following Expression (24).
[Math. 24]
rψ(c|x)=(c|μψ(x),diag(σψ2(x)))  Expression (24)

The right side of Expression (24) is the negative weighted square error between c˜qφvis(c|y) and μψ(x), and thereby increasing the right side of Expression (24) corresponds to causing the outputs of the appearance feature extraction unit14and the timbre feature extraction unit22to approach each other.

From the above, the following Expression (25) is a learning criterion to be increased together with Expression (15).

[Math.25]R⁡(θaud,ϕaud,ϕvis,ψ)=𝔼x~~pd(x~),y~~pd(y~)⁢[𝔼z~qϕaud(z|x~),c~qϕvid(c|y~)⁢𝔼x~p⁢θaud(x|z,c)[log⁢rψ(c|x)]]Expression⁢(25)

Consequently, the following Expression (26) combining Expression (25) and Expression (15) is the learning criterion of the proposed method.
[Math. 26]
(θaud,ϕaud,θvis,ϕvis)+(θaud,ϕaud,ϕvis,ψ)  Expression (26)

At this point, one point that demands attention is the calculation of expected values with respect to z˜qφaud(z|x), c˜qφvis(c|y), and x˜pθaud(x|z, c) appearing in Expression (15) and Expression (25). The term log pθaud(x|z, c) is a nonlinear function of z and c, the term log pθvis(y|c) is a nonlinear function of c, and the term log rψ(c|x) is a nonlinear function of x, and in general, obtaining expected values for these terms analytically is difficult. Consequently, a calculation method using a Monte Carlo approximation by sampling z, c, and x according to the distribution of each is conceivable. However, in this case, the parameters φaud, φvis, and θaudare included in the source distribution to be sampled, and evaluating the gradient of each term with respect to φaud, φvis, and θaudcannot be evaluated in backpropagation.

Here, using the normal random number expressed in the following Expression (27) and the fact that Expressions (28) to (30) are equivalent expressions for z˜qφaud(z|x), c˜qφvis(c|y), and x˜pθaud(x|z, c), respectively, the expected value calculation described above can be replaced by a Monte Carlo approximation through a sampling of ε in actuality.
[Math. 27]
ϵ˜(ϵ|0,I)  Expression (27)
[Math. 28]
z=μϕaud(x)+σϕaud(x)⊚ϵ  Expression (28)
[Math. 29]
c=μϕvis(y)+σϕvis(y)⊚ϵ  Expression (29)
[Math. 30]
x=μθaud(z,c)+σθaud(z,c)⊚ϵ  Expression (30)

In Expressions (28) to (30), the sign illustrated as a circle with a central dot denotes the element product of a vector. With this arrangement, the parameters φaud, φvisand ωaudcan be shifted into log pθaud(x|z, c), log pθvis(y|c), and log rψ(c|x). As a result, it is possible to evaluate the gradient of each term with respect to the parameters φaud, φvisand θaud. This technique is called a variable transformation trick.

FIGS.4to8are diagrams illustrating examples of the architecture of each neural network (NN).

FIG.4is a diagram illustrating an example of the architecture of the linguistic information extraction unit12according to the embodiment.FIG.5is a diagram illustrating an example of the architecture of the converted voice generation unit15according to the embodiment.FIG.6is a diagram illustrating an example of the architecture of the appearance feature extraction unit14according to the embodiment.FIG.7is a diagram illustrating an example of the architecture of the image generation unit23according to the embodiment.FIG.8is a diagram illustrating an example of the architecture of the timbre feature extraction unit22according to the embodiment. In each diagram, a notation describing an image of height h, width w, and number of channels c is adopted as the input and output of each NN. The terms “Conv”, “Deconv”, and “Linear” illustrated in each diagram denote a convolutional layer, a convolutional layer, and a fully connected layer, respectively. Also, “Batch norm”, “GLU”, “LReLU”, and “SoftPlus” denote a batch normalization layer, a gated linear unit (GLU) layer, a leaky rectified linear unit (leaky ReLU) layer, and a softplus layer, respectively. Also, “Broadcast” and “Reshape” denote an array broadcasting process and an array reshaping process. The letters “k”, “c”, and “s” denote the kernel size, output channel, and stride of a convolutional layer.

<Conversion Method (Method According to Embodiment of Present Invention>

For the acoustic features, any of the following (A1) to (A5) may be used.(A1) Vector containing a logarithmic amplitude spectrum as elements(A2) Vector containing mel-frequency cepstral coefficients as elements(A3) Vector containing linear predictor coefficients as elements(A4) Vector containing partial correlation (PARCOR) coefficients as elements(A5) Vector containing line spectral pair (LSP) parameters as elements

The above (A1) can be obtained by using time-frequency analysis such as the short-time Fourier transform (STFT) or wavelet transforms. The above (A2) can be obtained by using mel-frequency cepstral analysis. The above (A3) can be obtained by using linear prediction analysis. The above (A4) can be obtained by using PARCOR analysis. The above (A5) can be obtained by using LSP analysis. In addition, (A1) may also be a spectral envelope obtained by a method such as STRAIGHT analysis or WORLD analysis, and (A2) to (A5) may also be obtained through respective analyses of the spectral envelope. For example, the following (B1) to (B5) may also be used as the acoustic features.(B1) Vector containing logarithmic spectral envelope as elements(B2) Vector containing mel-frequency cepstral coefficients obtained from B1 as elements(B3) Vector containing linear predictor coefficients obtained from B1 as elements(B4) Vector containing PARCOR coefficients obtained from B1 as elements(B5) Vector containing LSP parameters obtained from B1 as elements

After the learning of φ and θ is completed, an acoustic feature series x of an input voice and an input face image y can be used to obtain an acoustic feature series of a converted voice according to the following Expression (31).
[Math. 31]
{circumflex over (x)}=μϕaud(μϕaud(x),μϕvis(y))  Expression (31)

The time-domain signal can be obtained by using an inverse transform of the time-frequency analysis (such as the inverse SIFT or inverse wavelet transforms) in the case of using (A1) as the acoustic feature vector, or by using a vocoder in the case of using (A2) to (A5) or (B1) to (B5). Also, the acoustic feature series x of the input voice can be used to generate a face image according to the following Expression (32).
[Math. 32]
ŷ=μθvis(μψ(x))  Expression (32)

FIG.9is a flowchart illustrating the flow of a converted voice generation process by the voice conversion device10according to the embodiment.

The linguistic information extraction unit12extracts linguistic information corresponding to utterance content from the input conversion source voice signal (step S101). The linguistic information extraction unit12outputs the extracted linguistic information to the converted voice generation unit15. The appearance feature extraction unit14extracts appearance features from the input face image (step S102). The appearance feature extraction unit14outputs the extracted appearance features to the converted voice generation unit15. The converted voice generation unit15generates a converted voice on the basis of the linguistic information output from the linguistic information extraction unit12and the appearance features output from the appearance feature extraction unit14(step S103). The converted voice generation unit15outputs the generated converted voice to the image generation device20.

FIG.10is a flowchart illustrating the flow of a face image generation process by image generation device20according to the embodiment.

The timbre feature extraction unit22extracts timbre features expressing features related to vocal timbre from the input voice signal (step S201). The timbre feature extraction unit22outputs the extracted timbre features to the image generation unit23. The image generation unit23generates a face image on the basis of the input timbre features and appearance features (step S202). The image generation unit23outputs the generated face image to an external device.

FIG.11is a flowchart illustrating the flow of a learning process by the voice conversion learning device30according to the embodiment.

The linguistic information extraction unit32extracts linguistic information corresponding to utterance content from the input conversion source voice signal (step S301). The linguistic information extraction unit32outputs the extracted linguistic information to the learning unit36. The appearance feature extraction unit34extracts appearance features from the input face image (step S302). The appearance feature extraction unit34outputs the extracted appearance features to the learning unit36. The converted voice generation unit35generates a converted voice on the basis of the input linguistic information and the appearance features (step S303). The converted voice generation unit35outputs the generated converted voice to the learning unit36. On the basis of the input linguistic information, appearance features, and converted voice, the learning unit36trains and updates parameters of the linguistic information extraction unit12, the appearance feature extraction unit14, and the converted voice generation unit15of the voice conversion device10(step S304).

FIG.12is a flowchart illustrating the flow of a learning process by the image generation learning device40according to the embodiment.

The appearance feature extraction unit42extracts appearance features from the input face image (step S401). The appearance feature extraction unit42outputs the extracted appearance features to the image generation unit45and the learning unit46. The timbre feature extraction unit44extracts timbre features expressing features related to vocal timbre from the input voice signal (step S402). The timbre feature extraction unit44outputs the extracted timbre features to the image generation unit45and the learning unit46. The image generation unit45generates a face image on the basis of the input timbre features and appearance features (step S404). The image generation unit45outputs the generated face image to the learning unit46. On the basis of the input appearance features and face image, the learning unit46trains and updates parameters of the image generation unit23of the image generation device20(step S404). Also, on the basis of the input timbre features and face image, the learning unit46updates parameters of the timbre feature extraction unit22of the image generation device20.

According to the cross-modal processing system100configured as above, it is possible to achieve a novel cross-modal process using correlations between voices and faces. Specifically, the voice conversion device10accepts an input voice signal and face image as input, and converts the voice to a timbre conforming to the input face image. In this way, the voice conversion device10is capable of achieving a novel cross-modal process using correlations between voices and faces.

Also, the image generation device20accepts an input voice signal and appearance features obtained from a face image as input, and generates a face image conforming to the timbre of the input voice. In this way, the voice conversion device10is capable of achieving a novel cross-modal process using correlations between voices and faces.

Experimental Results

To confirm the conversion effect of the cross-modal timbre conversion according to the method of the present invention, virtual pair data of voices and face images were constructed using the voice data from the Voice Conversion Challenge 2018 (VCC 2018) and the face image data from the Large-scale CelebFaces Attributes (CelebA) dataset, and then subjectively evaluated. A learning dataset and a test dataset were created by a method of classifying each of the voice data and the face image data according to sex and age class (young, aged), and treating voice data and face image data randomly chosen from groups with the same attributes as virtual pair data. The sampling frequency of all voice signals was set to 22050 Hz, and each face image was downsampled to the size 32×32.

The spectral envelope, base frequency (F0), and aperiodicity were extracted from each utterance by WORLD analysis, and 35th-order mel-frequency cepstral analysis was performed on the extracted spectral envelope series. Both the learning and test datasets were normalized such that the mel-frequency cepstral series x=[x1, . . . , xN]=(xd, n)D×Nof each utterance treated as the input has a mean of 0 and a variance of 1 in each dimension, as expressed by the following Expression (33).

[Math.33]xd,n←xd,n-αdβdExpression⁢(33)

In the above, αdand βdrepresent the mean and the standard deviation of xd, nin each voiced segment. A network that predicts the logarithm F0lnin voiced segments as well as the mean and the variance in each dimension of the mel-frequency cepstrum xd, nfrom a face image was trained separately, and during testing, the logarithm F0and the mel-frequency cepstrum of the output voice was converted using the mean and the variance predicted by the network, as expressed in the following Expressions (34) and (35).

[Math.34]x^d,n←β^dβd⁢(xd,n-αd)+α^dExpression⁢(34)[Math.35]l^n←v^v⁢(ln-m)+m^Expression⁢(35)

The terms m and v indicated in Expression (35) represent the mean and the standard deviation in voiced segments of the logarithm F0ln. The terms {circumflex over ( )}αd, {circumflex over ( )}βd, {circumflex over ( )}m, and {circumflex over ( )}v indicated in Expressions (34) and (35) represent the mean and the standard deviation (in voiced segments) of the mel-frequency cepstrum and the logarithm F0predicted from the input image. As baseline methods for comparative experiment, (1) a method of simply converting the logarithm F0and the mel-frequency cepstrum of the input voice on the basis of the sex label and age class label such that the mean and the variance match the mean and the variance of the logarithm F0and the mel-frequency cepstrum of a voice having the same attributes in the learning data (Baseline 1), and (2) a method of performing voice conversion in stages with a CVAE treating a face attribute identifier that predicts the sex and age class from a face image, the sex, and the age class as attribute codes (Baseline 2) were implemented.

To compare the effect of the voice conversion according to the proposed method between these baseline methods, a subjective evaluation was performed using an ABX test. In the ABX text, converted voices obtained by the proposed method and the baseline methods were treated as A and B, and the face image corresponding to the input voice was treated as X. Each listener chose whether the voice A or the voice B better fits the face image X, and then selected one from among A, B, and Fair (equivalent) for each utterance. As illustrated inFIG.13, the experimental results confirmed the superiority of the proposed method with respect to Baseline 1, and confirmed that a conversion effect on a par with Baseline 2 was obtained. Unlike the proposed method, because both of the baseline methods use voice and face image attribute information, the proposed method is being compared under disadvantageous conditions in the present test, and therefore the above results are thought to indicate the effectiveness of the proposed method.

In addition, an experiment of combining the timbre feature extraction unit22and the image generation unit23to generate a face image from an input voice was performed. As illustrated inFIG.14, the experiment confirmed that face images conforming to the attributes of the input voices to some degree are generated successfully.

Modifications

The voice conversion device10and the voice conversion learning device30may also be configured as a single device. The image generation device20and the image generation learning device40may also be configured as a single device.

An embodiment of the present invention has been described in detail and with reference to the drawings, but the specific configuration is not limited to the above embodiment, and includes designs and the like within a scope that does not depart from the gist of the present invention.

REFERENCE SIGNS LIST

10Voice conversion device20Image generation device30Voice conversion learning device40Image generation learning device11Voice input unit12Linguistic information extraction unit13Image input unit14Appearance feature extraction unit15Converted voice generation unit21Voice input unit22Timbre feature extraction unit23Image generation unit31Voice input unit32Linguistic information extraction unit33Image input unit34Appearance feature extraction unit35Learning unit41Image input unit42Appearance feature extraction unit43Image generation unit44Voice input unit45Timbre feature extraction unit46Learning unit