Patent Publication Number: US-2023138232-A1

Title: Conversion learning apparatus, conversion learning method, conversion learning program and conversion apparatus

Description:
TECHNICAL FIELD 
     The present invention relates to a conversion learning device, a conversion learning method, a conversion learning program, and a conversion device. 
     BACKGROUND ART 
     Attention has been focused on a technology, “voice conversion”, that converts non-linguistic information or paralinguistic information into other values while preserving linguistic information included in a voice (a sequence of characters of a spoken sentence). Through voice conversion, for example, a voice with voice quality (one of speaker identities) changed from that of a speaker can be generated while linguistic information is kept unchanged. Paralanguage that can be converted through voice conversion widely ranges over speaker identity, mode of speaking, accent, voice quality, and the like. Accordingly, voice conversion can be applied to various fields such as speech aids and speech recognition. 
     A voice conversion can be formulated as a regression analysis problem to estimate a mapping function from feature amounts of a conversion-source voice to feature amounts of a conversion-target (to-be-converted-to) voice. For example, schemes using Gaussian mixture models (GMM) are highly effective and versatile, and are widely used. In recent years, studies have also been conducted on schemes using neural networks (NN), schemes using non-negative matrix factorization (NMF), and the like. 
     However, the above-mentioned schemes, all of which aim to convert voice quality, are not necessarily suitable to convert prosodic features. The prosodic features here are also called suprasegmental features in some cases, and are important elements that characterize speaker identities and a mode of speaking, such as a basic frequency pattern, speed of speaking, and rhythm of speaking. Accordingly, it is desired to develop a voice conversion that can effectively convert not only voice quality but also a prosodic feature. 
     Moreover, as disclosed in Non-Patent Literatures 1, 2, and 3, development of voice conversion using a sequence-to-sequence conversion (S2S) learning model is pursued, and such voice conversion demonstrates high effectiveness in tasks such as machine translation, speech recognition, and text-to-speech synthesis. A sequence-to-sequence conversion learning model is a type of deep learning that learns a conversion function from a sequence to a sequence. 
     CITATION LIST 
     Non-Patent Literature 
     
         
         Non-Patent Literature 1: H. Miyoshi, Y. Saito, S. Takamichi, and H. Saruwatari, “Voice conversion using sequence-to-sequence learning of context posterior probabilities,” in Proc. Interspeech, 2017, pp. 1268-1272. 
         Non-Patent Literature 2: J.-X. Zhang, Z.-H. Ling, L.-J. Liu, Y. Jiang, L.-R. Dai, “Sequence-to-sequence acoustic modeling for voice conversion,” arXiv:1810.06865 [cs.SD], October 2018. 
         Non-Patent Literature 3: F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanvesky, and Y. Jia, “Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation,” in Proc. Interspeech, 2019, pp. 4115-4119. 
       
    
     Moreover, a principle of voice conversion can also be applied to, for example, image conversion and video conversion. Accordingly, it is also desired to develop techniques that make it possible to perform effective image conversion and video conversion, by applying the schemes related to voice conversion as described above. 
     SUMMARY OF THE INVENTION 
     Technical Problem 
     The present invention has been made in light of the above-described circumstances, and an object of the present invention is to provide a conversion learning device, a conversion learning method, a conversion learning program, and a conversion device that make it possible to perform effective conversion processing. 
     Means for Solving the Problem 
     An aspect of the present invention is a conversion learning device including: a source encoding unit that converts, by using a first machine learning model, a feature amount sequence of a source domain that is a characteristic of conversion-source content data, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the source domain are arranged; a target encoding unit that converts, by using a second machine learning model, a feature amount sequence of a target domain that is a characteristic of conversion-target content data, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the target domain are arranged; an attention matrix calculation unit that calculates, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the source domain to the individual locations of the feature amount sequence of the target domain, and calculates a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; a target decoding unit that calculates, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the source domain into the conversion domain, by using a third machine learning model; and a learning execution unit that causes at least one of the target encoding unit and the target decoding unit to learn such that a distance between a submatrix of the feature amount sequence of the target domain and a submatrix of the feature amount sequence of the conversion domain becomes shorter. 
     An aspect of the present invention is a conversion learning device including: a source encoding unit that converts, by using a first machine learning model, a feature amount sequence of a first domain that is a characteristic of conversion-source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the first domain are arranged; a target encoding unit that converts, by using a second machine learning model, a feature amount sequence of a second domain that is a characteristic of conversion-target content data and is assigned a second index that is different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the second domain are arranged; an attention matrix calculation unit that calculates, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the first domain to the individual locations of the feature amount sequence of the second domain, and calculates a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; a target decoding unit that calculate, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the first domain into the conversion domain, by using a third machine learning model; and a learning execution unit that causes at least one of the target encoding unit and the target decoding unit to learn such that a distance between a submatrix of the feature amount sequence of the second domain and a submatrix of the feature amount sequence of the conversion domain becomes shorter. 
     An aspect of the present invention is a conversion learning method including: a source encoding step of converting, by using a first machine learning model, a feature amount sequence of a source domain that is a conversion source, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the source domain are arranged; a target encoding step of converting, by using a second machine learning model, a feature amount sequence of a target domain that is a conversion target, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the target domain are arranged; an attention matrix calculation step of calculating, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the source domain to the individual locations of the feature amount sequence of the target domain, and calculating a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; a target decoding step of calculating, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the source domain into the conversion domain, by using a third machine learning model; and a learning execution step of causing at least one of a target encoding unit including the first machine learning model and a target decoding unit including the second machine learning model to learn such that a distance between a submatrix of the feature amount sequence of the target domain and a submatrix of the feature amount sequence of the conversion domain becomes shorter. 
     An aspect of the present invention is a conversion learning method including: a source encoding step of converting, by using a first machine learning model, a feature amount sequence of a first domain that is a characteristic of conversion-source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the first domain are arranged; a target encoding step of converting, by using a second machine learning model, a feature amount sequence of a second domain that is a characteristic of conversion-target content data and is assigned a second index that is different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the second domain are arranged; an attention matrix calculation step of calculating, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the first domain to the individual locations of the feature amount sequence of the second domain, and calculating a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; a target decoding step of calculating, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the first domain into the conversion domain, by using a third machine learning model; and a learning execution step of causing at least one of a target encoding unit including the first machine learning model and a target decoding unit including the second machine learning model to learn such that a distance between a submatrix of the feature amount sequence of the second domain and a submatrix of the feature amount sequence of the conversion domain becomes shorter. 
     An aspect of the present invention is a conversion device including: a source encoding unit that converts, by using a first machine learning model, a feature amount sequence of a source domain that is a characteristic of conversion-source content data, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the source domain are arranged; a target encoding unit that converts, by using a second machine learning model, a feature amount sequence of a target domain that is a characteristic of conversion-target content data, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the target domain are arranged; an attention matrix calculation unit that calculates, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the source domain to the individual locations of the feature amount sequence of the target domain, and calculates a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; and a target decoding unit that calculates, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the source domain into the conversion domain, by using a third machine learning model. 
     An aspect of the present invention is a conversion device including: a source encoding unit that converts, by using a first machine learning model, a feature amount sequence of a first domain that is a characteristic of conversion-source content data and is assigned a first index, into a first internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the first domain are arranged; a target encoding unit that converts, by using a second machine learning model, a feature amount sequence of a second domain that is a characteristic of conversion-target content data and is assigned a second index that is different from the first index, into a second internal representation vector sequence that is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the second domain are arranged; an attention matrix calculation unit that calculates, by using the first internal representation vector sequence and the second internal representation vector sequence, an attention matrix that is a matrix mapping the individual locations of the feature amount sequence of the first domain to the individual locations of the feature amount sequence of the second domain, and calculates a third internal representation vector sequence that is a product of an internal representation vector sequence calculated by linear conversion of the first internal representation vector sequence and the attention matrix; and a target decoding unit that calculates, by using the third internal representation vector sequence, a feature amount sequence of a conversion domain that is used to convert the first domain into the conversion domain, by using a third machine learning model. 
     Effects of the Invention 
     According to the present invention, it possible to perform effective conversion processing. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG.  1    shows an example of a conversion learning device according to a first embodiment. 
         FIG.  2    is a diagram for describing iteration of processing performed by a target encoding unit, an attention matrix calculation unit, and a target decoding unit according to the first embodiment. 
         FIG.  3    is a flowchart showing an example of processing performed in a learning phase by the conversion learning device according to the first embodiment. 
         FIG.  4    is a flowchart showing an example of processing performed in an inference phase by a conversion device according to the first embodiment. 
         FIG.  5    shows an example of an attention matrix in a case where the conversion device according to the first embodiment does not perform forward attention correction in the inference phase. 
         FIG.  6    shows an example of an attention matrix in a case where the conversion device according to the first embodiment performs forward attention correction in the inference phase. 
         FIG.  7    shows examples of mel-cepstral coefficients in cases where the conversion device according to the first embodiment does not perform forward attention correction, and performs forward attention correction. 
         FIG.  8    shows an example of a conversion learning device according to a second embodiment. 
         FIG.  9    is a flowchart showing an example of processing performed in a learning phase by the conversion learning device according to the second embodiment. 
         FIG.  10    is a flowchart showing an example of processing performed in an inference phase by a conversion device according to the second embodiment. 
         FIG.  11    shows an example of an attention matrix in a case where a source encoding unit, a target encoding unit, a target decoding unit, and a target reconstruction unit according to the second embodiment have regular batch normalization layers in the inference phase. 
         FIG.  12    shows an example of an attention matrix in a case where the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have batch normalization layers depending on a first domain or a second domain in the inference phase. 
         FIG.  13    shows examples of mel-cepstral coefficients in cases where the conversion device according to the second embodiment performs one-to-one conversion, and performs many-to-one conversion in the inference phase. 
         FIG.  14    shows examples of mel-cepstral coefficients in cases where the conversion device according to the second embodiment performs batch conversion, and performs real-time conversion in the inference phase. 
         FIG.  15    shows examples of mel-cepstral coefficients in cases where each of the conversion device according to the first embodiment and the conversion device according to the second embodiment performs conversion without taking into consideration any of context preservation loss and orthogonal attention loss, performs conversion taking only the orthogonal attention loss into consideration, and performs conversion taking only the context preservation loss into consideration. 
         FIG.  16    shows examples of mel-cepstral coefficients in cases of a conventional conversion device, the conversion device according to the first embodiment, and the conversion device according to the second embodiment. 
     
    
    
     DESCRIPTION OF EMBODIMENTS 
     First Embodiment 
     Examples of a conversion learning device and a conversion device according to a first embodiment will be described with reference to  FIGS.  1  and  2   .  FIG.  1    shows an example of the conversion learning device according to the first embodiment. As shown in  FIG.  1   , the conversion learning device  1  includes the conversion device  10  and a learning execution unit  20 . The conversion device  10  includes a source encoding unit  11 , a target encoding unit  12 , an attention matrix calculation unit  13 , a target decoding unit  14 , a target reconstruction unit  15 , and a forward attention correction execution unit  16 . 
       FIG.  2    is a diagram for describing iteration of processing performed by the target encoding unit, the attention matrix calculation unit, and the target decoding unit according to the first embodiment. As shown in  FIGS.  1  and  2   , the processing performed in a processing block P, that is, a series of the processing performed by the target encoding unit  12 , the attention matrix calculation unit  13 , and the target decoding unit  14  is iterated L (L: a natural number) times. 
     The source encoding unit  11  converts a feature amount sequence X of a source domain, which is a characteristic of conversion-source content data, into a first internal representation vector sequence Z by using a first machine learning model. 
     The source domain is data indicating a characteristic of the conversion-source content data, for example, a prosodic feature of conversion-source speech data. The feature amount sequence X of the source domain is represented by a following expression (1) including feature amount vectors x 1 , . . . , x N  at individual locations of the feature amount sequence of the source domain, dimensions D of the feature amount vectors x 1 , . . . , x N , and the number N of elements in the feature amount sequence of the source domain. When the content data is speech data, the individual locations are locations on a time axis, that is, time points. 
       [Math. 1] 
         X =[ x   1   , . . . ,x   N ]∈   D×N   (1)
 
     The first machine learning model is, for example, a convolutional neural network (CNN). The first internal representation vector sequence Z is a matrix in which internal representation vectors at the individual locations of the feature amount sequence of the source domain are arranged, and is represented by following expressions (2) and (3). The expression (2) includes the number of channels, that is, dimensions D Z  of the internal representation vectors. 
       [Math. 2] 
         Z∈     D     Z     ×N   (2)
 
       [Math. 3] 
         Z =SrcEnc( X )  (3)
 
     The target encoding unit  12  converts a feature amount sequence Y of a target domain, which is a characteristic of conversion-target content data, into a second internal representation vector sequence Q by using a second machine learning model. A spoken sentence included in the conversion-target content data is identical to a spoken sentence included in the conversion-source content data. 
     The target domain is data indicating a prosodic feature of conversion-target speech data. The feature amount sequence Y of the target domain is represented by a following expression (4) including feature amount vectors y 1 , . . . , y M  at individual locations of the feature amount sequence of the target domain, dimensions D of the feature amount vectors y 1 , . . . , y M , and the number M of elements in the feature amount sequence of the target domain. 
       [Math. 4] 
         Y =[ y   1   , . . . ,y   M ]∈   D×M   (4)
 
     The second machine learning model is, for example, a convolutional neural network. The second internal representation vector sequence Q is a matrix in which internal representation vectors at the individual locations of the feature amount sequence of the target domain are arranged, and is represented by following expressions (5) and (6). The expression (5) includes the number of channels, that is, dimensions D Q  of the internal representation vectors. 
       [Math. 5] 
         Q∈     D     Q     ×M   (5)
 
       [Math. 6] 
         Q =TrgEnc( Y )  (8)
 
     The attention matrix calculation unit  13  calculates an attention matrix A by using the first internal representation vector sequence Z and the second internal representation vector sequence Q. The attention matrix calculation unit  13  then calculates a third internal representation vector sequence R, which is a product of an internal representation vector sequence V calculated by linear conversion of the first internal representation vector sequence Z and the attention matrix A. 
     First, the attention matrix calculation unit  13  calculates an internal representation vector sequence K and the internal representation vector sequence V by applying a linear conversion L to the internal representation vector sequence Z of the source domain, as represented by a following expression (7), 
       [Math. 7] 
       ( K,V )=( Z )  (7)
 
     The attention matrix calculation unit  13  then calculates the attention matrix A by using a following expression (8) including the internal representation vector sequence K and the second internal representation vector sequence Q. Note that “softmax” included in the expression (8) means applying a softmax function to each column of a matrix. Moreover, “D′” included in the expression (8) is dimensions of column vectors included in the internal representation vector sequence K, the internal representation vector sequence Q, and the second internal representation vector sequence Q. 
       [Math. 8] 
         A =softmax( K   T   Q /√{square root over ( D ′)})  (8)
 
     The attention matrix A corresponds to a similarity matrix between the second internal representation vector sequence Q and the internal representation vector sequence K, and is a matrix mapping the individual locations of the feature amount sequence X of the source domain to the individual locations of the feature amount sequence Y of the target domain. Accordingly, the third internal representation vector sequence R, which is a matrix obtained by warping the internal representation vector sequence V in line with the time axis of the feature amount sequence Y of the target domain, is represented by a following expression (9). 
       [Math. 9] 
         R=VA   (9)
 
     Alternatively, the attention matrix calculation unit  13  calculates H sets of an internal representation vector sequence K h , an internal representation vector sequence V h , and a second internal representation vector sequence Q h  by applying a linear conversion L Z  to the internal representation vector sequence Z of the source domain and applying a linear conversion L Q  to the second internal representation vector sequence Q, as represented by following expressions (10) to (13). Note that “split” included in the expressions (12) and (13) represents a channel-wise split. 
       [Math. 10] 
       ( K,V )= L   Z ( Z )  (10)
 
       [Math. 11] 
       ( K   1   , . . . ,K   H )=split( K )  (11)
 
       [Math. 12] 
       ( V   1   , . . . ,V   H )=split( V )  (12)
 
       [Math. 13] 
       ( Q   1   , . . . ,Q   H )= L   Q ( Q )  (13)
 
     The attention matrix calculation unit  13  then calculates an attention matrix A by using a following expression (14) including the internal representation vector sequence K h  and the second internal representation vector sequence Q h . 
       [Math. 14] 
         A   h =softmax( K   h   T   Q   h /√{square root over ( D ′)})( h= 1, . . . , H )  (14)
 
     In this case, the attention matrix calculation unit  13  calculates a third internal representation vector sequence R that is represented by following expressions (15) and (16). Note that “concat” included in the expression (16) represents a channel-wise concatenation. 
       [Math. 15] 
         R   h   =V   h   A   h ( h= 1, . . . , H )  (15)
 
       [Math. 16] 
         R =concat( R   1   , . . . ,R   H )  (16)
 
     In a description below, the processing in which the attention matrix calculation unit  13  calculates the third internal representation vector sequence R from the first internal representation vector sequence Z and the second internal representation vector sequence Q is expressed as in a following expression (17). 
       [Math. 17] 
         R =attention( Z,Q )  (17)
 
     The target decoding unit  14  calculates, by using the third internal representation vector sequence R, a feature amount sequence I of a conversion domain that is used to convert the source domain into the conversion domain, by using a third machine learning model, as represented by a following expression (18). The third machine learning model is, for example, a convolutional neural network. 
       [Math. 18] 
         I =TrgDec( R )  (18)
 
     Here, in a learning phase in which the above-mentioned first, second, and third machine learning models are learned, a pair of a source domain and a target domain are given. On the other hand, in an inference phase in which voice conversion is performed by using the three machine learning models, only a source domain is given. Accordingly, in the inference phase, the target encoding unit  12  receives as input, for example, the feature amount sequence I of the conversion domain that is most recently outputted by the own device. 
     Accordingly, in the learning phase in which the first, second, and third machine learning models are learned, it is necessary to make the feature amount sequence I of the conversion domain come to match, as closely as possible, a sequence obtained by shifting the time points of the feature amount sequence Y of the target domain. Moreover, when each of the target encoding unit  12  and the target decoding unit  14  calculates each output, it is necessary to perform the above-described processing under a condition that after an output of interest is calculated, subsequent outputs cannot be used. Such a condition corresponds to imposing causality on convolutional layers when the second machine learning model included in the target encoding unit  12  and the third machine learning model included in the target decoding unit  14  are implemented by using convolutional neural networks. 
     Accordingly, the distance between a submatrix of the feature amount sequence I of the conversion domain and a submatrix of the feature amount sequence Y of the target domain, represented by a following expression (19), can be thought of as a learning loss. The expression (19) includes a sign “∥·∥” representing a matrix norm. The expression (19) includes a submatrix I d:d′,m:m′ , which includes elements in d to d′ rows and m to m′ columns of the feature amount sequence I of the conversion domain, and a submatrix Y d:d′,m:m′ , which includes elements in d to d′ rows and m to m′ columns of the feature amount sequence Y of the target domain. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               19 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     dec 
                   
                   = 
                   
                     
                       1 
                       M 
                     
                     ⁢ 
                     
                        
                       
                         
                           I 
                           
                             
                               1 
                               : 
                               D 
                             
                             , 
                             
                               
                                 1 
                                 : 
                                 M 
                               
                               - 
                               1 
                             
                           
                         
                         - 
                         
                           Y 
                           
                             
                               1 
                               : 
                               D 
                             
                             , 
                             
                               2 
                               : 
                               M 
                             
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
     The learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the distance between the feature amount sequence Y of the target domain and the feature amount sequence I of the conversion domain becomes shorter. For example, the learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the right-side value of the expression (19) is minimized. 
     By taking into consideration the learning loss represented by the expression (19), information on feature amounts of the target domain corresponding to element vectors at locations after a location of interest can be included in the third internal representation vector sequence R. Moreover, by taking into consideration the learning loss represented by the expression (19), information on a feature amount of the target domain at the same location as the location of interest can be concurrently preserved, whereby a result of inference in the inference phase can be made stable. The target reconstruction unit  15  is therefore introduced. 
     The target reconstruction unit  15  calculates, by using the third internal representation vector sequence R, a feature amount sequence J of a reconstructed target domain, which is a domain obtained by reconstructing the target domain, by using a fourth machine learning model, as represented by a following expression (20). The fourth machine learning model is, for example, a convolutional neural network. 
       [Math. 20] 
         J =TrgRec( R )  (20)
 
     The learning execution unit  20  causes the target reconstruction unit  15  to learn such that the distance between the feature amount sequence Y of the target domain and the feature amount sequence J of the reconstructed target domain becomes shorter. For example, the learning execution unit  20  causes the target reconstruction unit  15  to learn such that the right-side value of a following expression (21) is minimized. A learning loss represented by the expression (21) is referred to as context preservation loss (CPL). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               21 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     rec 
                   
                   = 
                   
                     
                       1 
                       M 
                     
                     ⁢ 
                     
                        
                       
                         J 
                         - 
                         Y 
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   21 
                   ) 
                 
               
             
           
         
       
     
     When the number L of iterations of the processing performed in the processing block P shown in  FIGS.  1  and  2    is two or more, L third internal representation vector sequences R are calculated by the attention matrix calculation unit  13 . In such a case, the context preservation loss is an arithmetic mean of the expression (21) with respect to at least some of the L third internal representation vector sequences R. 
     Moreover, since a spoken sentence included in the conversion-source content data and a spoken sentence included in the conversion-target content data are identical as mentioned above, diagonal components and elements in vicinities thereof of the attention matrix A tend to be predominant in many cases. 
     Accordingly, the learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that non-diagonal components of the attention matrix A become smaller. For example, the learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the right-side value of a following expression (22) is minimized. A learning loss represented by the expression (22) is referred to as diagonal attention loss (DAL). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               22 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     diag 
                   
                   = 
                   
                     
                       1 
                       NM 
                     
                     ⁢ 
                     
                        
                       
                         
                           
                             W 
                             
                               N 
                               × 
                               M 
                             
                           
                           ( 
                           ν 
                           ) 
                         
                         ⊙ 
                         A 
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   22 
                   ) 
                 
               
             
           
         
       
     
     Note that a sign between “W N×M (ν)” and “A” in the expression (22) denotes element-wise products of a matrix W N×M (ν) and the attention matrix A. Moreover, the matrix W N×M (ν) is a matrix with N rows and M columns as represented by a following expression (23), and is a matrix in which an (n, m) element is represented by a following expression (24). 
       [Math. 23] 
         W   N×M (ν)∈   N×M   (23)
 
       [Math. 24] 
       θ n,m =1− e   −n/N−m/M)     2     /2ν     2     (24)
 
     Each location of the feature amount sequence Y of the target domain is thought to correspond to one, or a few at most, of the locations of the feature amount sequence X of the source domain. This implies that arbitrary column vectors included in the attention matrix A are orthogonal. 
     Accordingly, the learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the number of orthogonal column vectors included in the attention matrix A becomes larger. For example, the learning execution unit  20  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the right-side value of a following expression (25) is minimized. A learning loss represented by the expression (25) is referred to as orthogonal attention loss (OAL). Note that a sign between “W N×M (ρ)” and “(AA T )” in the expression (25) denotes element-wise products of a matrix W N×M (ρ) and (AA T ), 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               25 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     orth 
                   
                   = 
                   
                     
                       1 
                       
                         N 
                         2 
                       
                     
                     ⁢ 
                     
                        
                       
                         
                           
                             W 
                             
                               N 
                               × 
                               N 
                             
                           
                           ( 
                           ρ 
                           ) 
                         
                         ⊙ 
                         
                           ( 
                           
                             AA 
                             T 
                           
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   25 
                   ) 
                 
               
             
           
         
       
     
     When the expressions (19), (21), (22), and (25) are taken into consideration, a learning loss is represented by a following expression (26). 
       [Math. 26] 
         =   (X,Y) [   dec +λ r     rec +λ d     diag +λ 0     orth ]  (26)
 
     Note that when in the inference phase, the target encoding unit  12  receives, as input, not the feature amount sequence Y of the target domain but the feature amount sequence I of the conversion domain outputted from the target decoding unit  14 . Then, the target encoding unit  12  converts the feature amount sequence I of the conversion domain into a second internal representation vector sequence Q by using the second machine learning model. 
     Moreover, when in the inference phase, and when the number L of iterations of the processing performed in the processing block P shown in  FIGS.  1  and  2    is two or more, the forward attention correction execution unit  16  restricts the attention matrix calculation unit  13  such that a peak of a distribution of attention locations of the attention matrix A changes monotonically and continuously. When the content data is speech data, the attention locations here are attention time points. Note that the conversion device  10  does not need to include the forward attention correction execution unit  16 . 
     When the conversion device  10  performs conversion in real time in the inference phase, the learning execution unit  20  causes the source encoding unit  11  to learn under a constraint that the feature amount sequence X of the source domain that is inputted into the source encoding unit  11  before a location at which the feature amount sequence X of the source domain is to be converted into the first internal representation vector sequence Z, is converted into the first internal representation vector sequence Z. 
     Next, an example of processing performed in the learning phase by the conversion learning device  1  will be described with reference to  FIG.  3   .  FIG.  3    is a flowchart showing an example of the processing performed in the learning phase by the conversion learning device according to the first embodiment. The processing shown in  FIG.  3    is performed each time a pair of a source domain and a target domain are inputted into the conversion learning device  1 . 
     In step S 31 , the source encoding unit  11  converts a feature amount sequence X of a source domain into a first internal representation vector sequence Z. 
     In step S 32 , the target encoding unit  12  converts a feature amount sequence Y of a target domain into a second internal representation vector sequence Q. 
     In step S 33 , the attention matrix calculation unit  13  calculates an attention matrix A by using the first internal representation vector sequence Z and the second internal representation vector sequence Q. 
     In step S 34 , the attention matrix calculation unit  13  calculates a third internal representation vector sequence R, which is a product of an internal representation vector sequence V calculated by linear conversion of the first internal representation vector sequence Z and the attention matrix A. 
     In step S 35 , the target decoding unit  14  calculates a feature amount sequence I of a conversion domain that is used to convert the source domain into the conversion domain by using the third internal representation vector sequence R. 
     In step S 36 , the target reconstruction unit  15  calculates a feature amount sequence J of a reconstructed target domain, which is a domain obtained by reconstructing the target domain by using the third internal representation vector sequence R. 
     In step S 37 , the learning execution unit  20  causes at least one of the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit to learn. 
     Note that the conversion learning device  1  may interchange order of step S 31  and step S 32 . The conversion learning device  1  may interchange order of step S 35  and step S 36 . 
     Next, an example of processing performed in the inference phase by the conversion device  10  will be described with reference to  FIG.  4   .  FIG.  4    is a flowchart showing an example of the processing performed in the inference phase by the conversion device according to the first embodiment. 
     In step S 41 , the source encoding unit  11  converts a feature amount sequence X of a source domain into a first internal representation vector sequence Z. 
     In step S 42 , the target encoding unit  12  converts a feature amount sequence Y of a target domain into a second internal representation vector sequence Q. 
     In step S 43 , the attention matrix calculation unit  13  calculates an attention matrix A by using the first internal representation vector sequence Z and the second internal representation vector sequence Q. 
     In step S 44 , the attention matrix calculation unit  13  calculates a third internal representation vector sequence R, which is a product of an internal representation vector sequence V calculated by linear conversion of the first internal representation vector sequence Z and the attention matrix A. 
     In step S 45 , the target decoding unit  14  calculates a feature amount sequence I of a conversion domain that is used to convert the source domain into the conversion domain by using the third internal representation vector sequence R. 
     Note that the conversion device  10  may interchange order of step S 41  and step S 42 . 
     The conversion learning device  1  and the conversion device  10  according to the first embodiment have been described hereinabove. The conversion learning device  1  converts a feature amount sequence X of a source domain into a first internal representation vector sequence Z by using the first machine learning model, and converts a feature amount sequence Y of a target domain into a second internal representation vector sequence Q by using the second machine learning model. The conversion learning device  1  next calculates an attention matrix A by using the first internal representation vector sequence X and the second internal representation vector sequence Y, and calculates a third internal representation vector sequence R, which is a product of an internal representation vector sequence V calculated by linear conversion of the first internal representation vector sequence X and the attention matrix A. The conversion learning device  1  then calculates, by using the third internal representation vector sequence R, a feature amount sequence I of a conversion domain that is used to convert the source domain into the conversion domain, by using the third machine learning model. Moreover, the conversion learning device  1  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the distance between a submatrix of the feature amount sequence Y of the target domain and a submatrix of the feature amount sequence I of the conversion domain becomes shorter. 
     Thus, the conversion learning device  1  can perform learning with a higher probability of accuracy, can generate effective learning models, and can make it possible to perform effective conversion processing using the learning models. When the content data is speech data, the conversion learning device  1  can make it possible to perform effective conversion processing including a prosodic feature. 
     The conversion learning device  1  calculates, by using the third internal representation vector sequence R, a feature amount sequence I of a reconstructed target domain, which is a domain obtained by reconstructing the target domain, by using the fourth machine learning model. Then, the conversion learning device  1  causes the target reconstruction unit to learn such that the distance between the feature amount sequence Y of the target domain and the feature amount sequence J of the reconstructed target domain becomes shorter. Thus, the conversion learning device  1  can make it possible for the conversion device  10  to output a more stable result of inference. 
     The conversion learning device  1  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that non-diagonal components of the attention matrix A become smaller. Thus, the conversion learning device  1  can perform learning that is based on a premise that a spoken sentence in the content data including the source domain matches a spoken sentence in the content data including the target domain, and can make it possible for the conversion device  10  to output a more stable result of inference. 
     The conversion learning device  1  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the number of orthogonal column vectors included in the attention matrix A becomes larger. Thus, the conversion learning device  1  can perform learning that is based on a premise that each location of the feature amount sequence X of the source domain and each location of the feature amount sequence Y of the target domain largely correspond to each other, and can make it possible for the conversion device  10  to output a more stable result of inference. 
     The conversion learning device  1  causes the source encoding unit  11  to learn under the constraint that the feature amount sequence X of the source domain that is inputted into the source encoding unit  11  before a location at which the feature amount sequence X of the source domain is to be converted into the first internal representation vector sequence Z, is converted into the first internal representation vector sequence Z. Thus, the conversion learning device  1  makes it possible for the conversion device  10  to perform conversion processing online in real time. 
     Since the conversion device  10  performs conversion processing by using the feature amount sequence of the conversion domain through the conversion learning device  1 , the conversion device  10  can perform effective conversion processing. 
     Moreover, when the processing in which the target encoding unit  12  converts the feature amount sequence I of the conversion domain into a second internal representation vector sequence Q, the attention matrix calculation unit  13  calculates a third internal representation vector sequence R, and the target decoding unit  14  calculates a feature amount sequence I of the conversion domain, is performed at least two times, the conversion device  10  restricts the attention matrix calculation unit such that the peak of the distribution of the attention locations of the attention matrix A changes monotonically and continuously. Thus, the conversion device  10  can perform conversion processing while the peak of the distribution of the attention locations continuously progresses. 
     Next, specific examples of effects brought about by the conversion learning device  1  and the conversion device  10  according to the first embodiment will be described with reference to  FIGS.  5  to  7   . In the description of the effects, all of the source encoding unit  11 , the target encoding unit  12 , the target decoding unit  14 , and the target reconstruction unit  15  include convolutional layers. 
     Moreover, for an acoustic feature amount vector, a combined form of a mel-cepstral coefficient, a logarithmic basic frequency pattern, an aperiodicity index, and a voiced/unvoiced indicator is used. The voiced/unvoiced indicator is “1” in a voiced segment, and is “0” in an unvoiced segment. 
     For an evaluation index, mel-cepstral distortion (MCD) is used. Assuming that a mel-cepstral coefficient for the target domain and a mel-cepstral coefficient for the conversion domain are represented by expressions (27) and (28), respectively, the mel-cepstral distortion is expressed as in an expression (29). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               27 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     c 
                     ^ 
                   
                   = 
                   
                     
                       [ 
                       
                         
                           
                             c 
                             ^ 
                           
                           1 
                         
                         , 
                         … 
                             
                         , 
                         
                           
                             c 
                             ^ 
                           
                           28 
                         
                       
                       ] 
                     
                     T 
                   
                 
               
               
                 
                   ( 
                   27 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               28 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   c 
                   = 
                   
                     
                       [ 
                       
                         
                           c 
                           1 
                         
                         , 
                         … 
                             
                         , 
                         
                           c 
                           28 
                         
                       
                       ] 
                     
                     T 
                   
                 
               
               
                 
                   ( 
                   28 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               29 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     
                       MCD 
                       [ 
                       dB 
                       ] 
                     
                     = 
                     
                       
                         10 
                         
                           ln 
                           ⁢ 
                           10 
                         
                       
                       ⁢ 
                       
                         
                           2 
                           ⁢ 
                           
                             
                               ∑ 
                               
                                 i 
                                 = 
                                 2 
                               
                               28 
                             
                             
                               
                                 ( 
                                 
                                   
                                     
                                       c 
                                       ^ 
                                     
                                     i 
                                   
                                   - 
                                   
                                     c 
                                     i 
                                   
                                 
                                 ) 
                               
                               2 
                             
                           
                         
                       
                     
                   
                   , 
                 
               
               
                 
                   ( 
                   29 
                   ) 
                 
               
             
           
         
       
     
     Note, however, that since time points of the target domain and time points of the conversion domain are not necessarily aligned with each other, quality of the conversion domain is evaluated by determining corresponding locations of the two sequences through dynamic time warping (DTW) using the mel-cepstral coefficients for a local distance, and then by calculating an average of mel-cepstral coefficients based on the corresponding locations. 
       FIG.  5    shows an example of an attention matrix in a case where the conversion device according to the first embodiment does not perform forward attention correction in the inference phase. In  FIG.  5   , a horizontal axis represents time points corresponding to feature amount vectors included in the target domain, and a vertical axis represents time points corresponding to feature amount vectors included in the source domain. Referring to  FIG.  5   , it can be understood that the peak of the distribution of the attention locations of the attention matrix A is discontinuous despite a fact that spoken sentences in the source domain and the target domain are reflected. This is because no forward attention correction is performed. 
       FIG.  6    shows an example of an attention matrix in a case where the conversion device according to the first embodiment performs forward attention correction in the inference phase. In  FIG.  6   , a horizontal axis represents time points corresponding to feature amount vectors included in the target domain, and a vertical axis represents time points corresponding to feature amount vectors included in the source domain. Referring to  FIG.  6   , it can be understood that the peak of the distribution of the attention locations of the attention matrix A changes monotonically and continuously. This is because forward attention correction is performed. 
       FIG.  7    shows examples of mel-cepstral coefficients in cases where the conversion device according to the first embodiment does not perform forward attention correction, and performs forward attention correction. Referring to a table shown in  FIG.  7   , it can be understood that the mel-cepstral distortion is smaller and conversion is more effectively performed when forward attention correction is performed, than when no forward attention correction is performed. 
     Second Embodiment 
     Examples of a conversion learning device and a conversion device according to a second embodiment will be described with reference to  FIG.  8   .  FIG.  8    shows an example of the conversion learning device according to the second embodiment. As shown in  FIG.  8   , the conversion learning device  3  includes the conversion device  30  and a learning execution unit  40 . The conversion device  30  includes a source encoding unit  31 , a target encoding unit  32 , an attention matrix calculation unit  33 , a target decoding unit  34 , and a target reconstruction unit  35 . Unlike the conversion learning device  1  and the conversion device  10  according to the first embodiment, the conversion learning device  3  and the conversion device  30  enable conversion between feature amount sequences X (1) , . . . , X (K)  of the same spoken voices belonging to a plurality of domains, for example, K types of domains, with a single network set. 
     The source encoding unit  31  converts a feature amount sequence X (k)  of a first domain, which is a characteristic of conversion-source content data and is assigned a first index k, into a first internal representation vector sequence Z (k)  by using a first machine learning model. The first internal representation vector sequence Z (k)  is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the first domain are arranged. When the content data is speech data, the individual locations are locations on a time axis, that is, time points. A relation between the feature amount sequence X (k)  of the first domain and the first internal representation vector sequence Z (k)  is represented by a following expression (30). The first machine learning model is, for example, a convolutional neural network. 
       [Math. 30] 
         Z   (k) =SrcEnc( X   (k)   ,k )  (30)
 
     The target encoding unit  32  converts a feature amount sequence X (k′)  of a second domain, which is a characteristic of conversion-target content data and is assigned a second index k′ that is different from the first index k, into a second internal representation vector sequence Q (k′)  by using a second machine learning model. The second internal representation vector sequence Q (k′)  is a matrix in which internal representation vectors at individual locations of the feature amount sequence of the second domain are arranged. A relation between the feature amount sequence X (k′)  of the second domain and the second internal representation vector sequence Q (k′)  is represented by a following expression (31). The second machine learning model is, for example, a convolutional neural network. 
       [Math. 31] 
         Q   (k′) =TrgEnc( X   (k′)   ,k ′)  (31)
 
     First, the attention matrix calculation unit  33 , similarly to the attention matrix calculation unit  13  according to the first embodiment, calculates an attention matrix A (k,k′)  by using the first internal representation vector sequence Z (k) , the second internal representation vector sequence Q (k′) , and following expressions (32) and (33). 
       [Math. 32] 
       ( K   (k)   ,V   (k) )= L ( Z   (k) )  (32)
 
       [Math. 33] 
         A   (k,k′) =softmax n ( K   (k)T   Q   (k′) /√{square root over ( D ′)})  (33)
 
     Then, similarly to the attention matrix calculation unit  13  according to the first embodiment, the attention matrix calculation unit  33  calculates a third internal representation vector sequence R (k,k′)  by using a following expression (34). 
       [Math. 34] 
         R   (k,k′)   =V   (k)   A   (k,k′)   (34)
 
     Alternatively, similarly to the attention matrix calculation unit  13  according to the first embodiment, the attention matrix calculation unit  33  calculates H sets of an internal representation vector sequence K h   (k) , an internal representation vector sequence V h   (k) , and a second internal representation vector sequence Q (k′)  by applying a linear conversion L Z  to the internal representation vector sequence Z (k)  of the first domain, and applying a linear conversion L Q  to the second internal representation vector sequence Q (k′) , as represented by following expressions (35) to (38). 
       [Math. 35] 
       ( K   (k)   ,V   (k) )= L   Z ( Z   (k) )  (35)
 
       [Math. 36] 
       ( K   1   (k)   , . . . ,K   H   (k) )=split( K   (k) )  (36)
 
       [Math. 37] 
       ( V   1   (k)   , . . . ,V   H   (k) )=spit( V   (k) )  (37)
 
       [Math. 38] 
       ( Q   1   (k′) ), . . . , Q   H   (k′) )= L   Q ( Q   (k′) )  (38)
 
     Then, similarly to the attention matrix calculation unit  13  according to the first embodiment, the attention matrix calculation unit  33  calculates an attention matrix A h   (k,k′)  by using a following expression (39) including the internal representation vector sequence K h   (k)  and the second internal representation vector sequence Q h   (k′) . 
       [Math. 39] 
         A   h   (k,k′) =softmax( K   h   (k)T   Q   h   (k′) )/√{square root over ( D ′)})( h= 1, . . . , H )  (39)
 
     In this case, the attention matrix calculation unit  33  calculates a third internal representation vector sequence R (k,k′)  that is represented by following expressions (40) and (41). 
       [Math. 40] 
         R   h   (k,k′)   =V   h   (k)   A   h   (k,k′) ( h= 1 . . . , H )  (40)
 
       [Math. 41] 
         R   (k,k′) =concat( R   1   (k,k′)   , . . . ,R   H   (k,k′) )  (41)
 
     In a description below, the processing in which the attention matrix calculation unit  33  calculates the third internal representation vector sequence R (k,k′)  from the first internal representation vector sequence Z (k)  and the second internal representation vector sequence Q (k′)  is expressed as in a following expression (42). 
       [Math. 42] 
         R   (k,k′) =attention( Z   (k)   ,Q   (k′) )  (42)
 
     The target decoding unit  34  calculates, by using the third internal representation vector sequence R (k,k′) , a feature amount sequence F (k,k′)  of a conversion domain that is used to convert the first domain into the conversion domain, by using a third machine learning model, as represented by a following expression (43). The third machine learning model is, for example, a convolutional neural network. 
       [Math. 43] 
         F   (k,k′) =TrgDec( R   (k,k′)   ,k ′)  (43)
 
     Here, in a learning phase in which the above-mentioned first, second, and third machine learning models are learned, a pair of a first domain and a target domain are given. On the other hand, in an inference phase in which voice conversion is performed by using the three machine learning models, only a first domain is given. Accordingly, in the inference phase, the target encoding unit  32  receives as input, for example, the feature amount sequence F (k,k′)  of the conversion domain that is most recently outputted by the own device. 
     A learning loss represented as a distance between the feature amount sequence F (k,k′)  of the conversion domain and the feature amount sequence X (k′)  of the second domain is represented by a following expression (44). The expression (44) includes a submatrix F (k,k′)   d:d′,m:m′ , which includes elements in d to d′ rows and m to m′ columns of the feature amount sequence F (k,k′)  of the conversion domain, and a submatrix X (k′)   d:d′,m:m′ , which includes elements in d to d′ rows and m to m′ columns of the feature amount sequence X (k′)  of the second domain. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               44 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     dec 
                     
                       ( 
                       
                         k 
                         , 
                         
                           k 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         N 
                         
                           ( 
                           
                             k 
                             ′ 
                           
                           ) 
                         
                       
                     
                     ⁢ 
                     
                        
                       
                         
                           F 
                           
                             
                               1 
                               : 
                               D 
                             
                             , 
                             
                               
                                 1 
                                 : 
                                 
                                   N 
                                   
                                     ( 
                                     
                                       k 
                                       ′ 
                                     
                                     ) 
                                   
                                 
                               
                               - 
                               1 
                             
                           
                           
                             ( 
                             
                               k 
                               , 
                               
                                 k 
                                 ′ 
                               
                             
                             ) 
                           
                         
                         - 
                         
                           X 
                           
                             
                               1 
                               : 
                               D 
                             
                             , 
                             
                               2 
                               : 
                               
                                 N 
                                 
                                   ( 
                                   
                                     k 
                                     ′ 
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             ( 
                             
                               k 
                               ′ 
                             
                             ) 
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   44 
                   ) 
                 
               
             
           
         
       
     
     The learning execution unit  40  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the distance between the submatrix of the feature amount sequence X (k′)  of the second domain and the submatrix of the feature amount sequence F (k,k′)  of the conversion domain becomes shorter. For example, the learning execution unit  40  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the right-side value of the expression (44) is minimized. 
     By taking into consideration the learning loss represented by the expression (44), information on feature amounts of the second domain corresponding to element vectors at locations after a location of interest can be included in the third internal representation vector sequence R (k,k′) . Moreover, by taking into consideration the learning loss represented by the expression (44), information on a feature amount of the second domain at the same location as the location of interest can be concurrently preserved, whereby a result of inference in the inference phase can be made stable. The target reconstruction unit  35  is therefore introduced. 
     The target reconstruction unit  35  calculates, by using the fourth internal representation vector sequence R (k,k′) , a feature amount sequence J (k,k′)  of a reconstructed domain, which is a domain obtained by reconstructing the second domain, by using a fourth machine learning model, as represented by a following expression (45). The fourth machine learning model is, for example, a convolutional neural network. 
       [Math. 45] 
         G   (k,k′) =TrgRec( R   (k,k′)   ,k ′)  (45)
 
     The learning execution unit  40  causes the target reconstruction unit  35  to learn such that the distance between the feature amount sequence X (k′)  of the second domain and the feature amount sequence J (k,k′)  of the reconstructed domain becomes shorter. For example, the learning execution unit  40  causes the target reconstruction unit  15  to learn such that the right-side value of a following expression (46) is minimized. A learning loss represented by the expression (46) is referred to as context preservation loss. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               46 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     rec 
                     
                       ( 
                       
                         k 
                         , 
                         
                           k 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         N 
                         
                           ( 
                           
                             k 
                             ′ 
                           
                           ) 
                         
                       
                     
                     ⁢ 
                     
                        
                       
                         
                           G 
                           
                             ( 
                             
                               k 
                               ′ 
                             
                             ) 
                           
                         
                         - 
                         
                           X 
                           
                             ( 
                             
                               k 
                               ′ 
                             
                             ) 
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   46 
                   ) 
                 
               
             
           
         
       
     
     When the number L of iterations of the processing performed in a processing block Q shown in  FIG.  8    is two or more, L third internal representation vector sequences R are calculated by the attention matrix calculation unit  33 . In such a case, the context preservation loss is an arithmetic mean of the expression (46) with respect to at least some of the L third internal representation vector sequences R. 
     Moreover, as in the first embodiment, since a spoken sentence included in the conversion-source content data and a spoken sentence included in the conversion-target content data are identical, diagonal components and elements in vicinities thereof of the attention matrix A (k,k′)  tend to be predominant in many cases. 
     Accordingly, the learning execution unit  40  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that non-diagonal components of the attention matrix A (k,k′)  become smaller. For example, the learning execution unit  40  causes at least one of the target encoding unit  12  and the target decoding unit  14  to learn such that the right-side value of a following expression (47) is minimized. A learning loss represented by the expression (47) is referred to as diagonal attention loss. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               47 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     diag 
                     
                       ( 
                       
                         k 
                         , 
                         
                           k 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         
                           N 
                           
                             ( 
                             k 
                             ) 
                           
                         
                         ⁢ 
                         
                           N 
                           
                             ( 
                             
                               k 
                               ′ 
                             
                             ) 
                           
                         
                       
                     
                     ⁢ 
                     
                        
                       
                         
                           
                             W 
                             
                               
                                 N 
                                 k 
                               
                               × 
                               
                                 N 
                                 
                                   k 
                                   ′ 
                                 
                               
                             
                           
                           ( 
                           ν 
                           ) 
                         
                         ⊙ 
                         
                           A 
                           
                             ( 
                             
                               k 
                               , 
                               
                                 k 
                                 ′ 
                               
                             
                             ) 
                           
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   47 
                   ) 
                 
               
             
           
         
       
     
     As in the first embodiment, each location of the feature amount sequence X (k′)  of the second domain is thought to correspond to one, or a few at most, of the locations of the feature amount sequence X (k)  of the first domain. This implies that arbitrary column vectors included in the attention matrix A (k,k′)  are orthogonal. 
     Accordingly, the learning execution unit  40  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the number of orthogonal column vectors included in the attention matrix A (k,k′)  becomes larger. For example, the learning execution unit  40  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the right-side value of a following expression (48) is minimized. A learning loss represented by the expression (48) is referred to as orthogonal attention loss. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               48 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     ℒ 
                     orth 
                     
                       ( 
                       
                         k 
                         , 
                         
                           k 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         N 
                         
                           
                             ( 
                             k 
                             ) 
                           
                           ⁢ 
                           2 
                         
                       
                     
                     ⁢ 
                     
                        
                       
                         
                           
                             W 
                             
                               
                                 N 
                                 k 
                               
                               × 
                               
                                 N 
                                 k 
                               
                             
                           
                           ( 
                           ρ 
                           ) 
                         
                         ⊙ 
                         
                           ( 
                           
                             
                               A 
                               
                                 ( 
                                 
                                   k 
                                   , 
                                   
                                     k 
                                     ′ 
                                   
                                 
                                 ) 
                               
                             
                             ⁢ 
                             
                               A 
                               
                                 
                                   ( 
                                   
                                     k 
                                     , 
                                     
                                       k 
                                       ′ 
                                     
                                   
                                   ) 
                                 
                                 ⁢ 
                                 T 
                               
                             
                           
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                   ( 
                   48 
                   ) 
                 
               
             
           
         
       
     
     When the expressions (44), (46), (47), and (48) are taken into consideration, a learning loss is represented by a following expression (49). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               49 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   ℒ 
                   = 
                   
                     
                       ∑ 
                       k 
                     
                     
                       
                         ∑ 
                         
                           
                             k 
                             ′ 
                           
                           ≠ 
                           k 
                         
                       
                       
                         
                           𝔼 
                           
                             ( 
                             
                               
                                 X 
                                 
                                   ( 
                                   k 
                                   ) 
                                 
                               
                               , 
                               
                                 X 
                                 
                                   ( 
                                   
                                     k 
                                     ′ 
                                   
                                   ) 
                                 
                               
                             
                             ) 
                           
                         
                         [ 
                         
                           
                             ℒ 
                             dec 
                             
                               ( 
                               
                                 k 
                                 , 
                                 
                                   k 
                                   ′ 
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             
                               λ 
                               r 
                             
                             ⁢ 
                             
                               ℒ 
                               rec 
                               
                                 ( 
                                 
                                   k 
                                   , 
                                   
                                     k 
                                     ′ 
                                   
                                 
                                 ) 
                               
                             
                           
                           + 
                           
                             
                               λ 
                               d 
                             
                             ⁢ 
                             
                               ℒ 
                               diag 
                               
                                 ( 
                                 
                                   k 
                                   , 
                                   
                                     k 
                                     ′ 
                                   
                                 
                                 ) 
                               
                             
                           
                           + 
                           
                             
                               λ 
                               o 
                             
                             ⁢ 
                             
                               ℒ 
                               orth 
                               
                                 ( 
                                 
                                   k 
                                   , 
                                   
                                     k 
                                     ′ 
                                   
                                 
                                 ) 
                               
                             
                           
                         
                         ] 
                       
                     
                   
                 
               
               
                 
                   ( 
                   49 
                   ) 
                 
               
             
           
         
       
     
     Note that when in the inference phase, the target encoding unit  32  receives, as input, not the feature amount sequence X (k′)  of the second domain but the feature amount sequence F (k,k′)  of the conversion domain outputted from the target decoding unit  34 . Then, the target encoding unit  32  converts the feature amount sequence F (k,k′)  of the conversion domain into a second internal representation vector sequence Q (k′)  by using the second machine learning model. 
     Next, layers included in the networks included in the conversion learning device  3  and the conversion device  30  will be described by using expressions (50) to (57) when necessary. 
     It is preferable that the networks included in the conversion learning device  3  and the conversion device  30  include a convolutional layer and a self-attention layer, for following reasons. In the conversion learning device  3  and the conversion device  30 , it is conceivable that a distribution of inputs into each hidden layer of each network can greatly vary, depending on at least one of the first domain and the second domain. Moreover, since a range of values to be covered by each layer is inevitably wide, compared to the conversion learning device  1  and the conversion device  10  according to the first embodiment, even if a batch normalization layer, an instance normalization layer, or the like is inserted as in regular cases, it is conceivable that accuracy of a result of inference in the inference phase can decrease. 
     To avoid such situations, it is preferable that the networks included in the conversion learning device  3  and the conversion device  30  include a layer that performs normalization, weight normalization, or layer normalization, depending on the first domain or the second domain. In other words, it is preferable that the source encoding unit  31  include a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the first domain. Moreover, it is preferable that the target encoding unit  32  include a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the second domain. It is preferable that the target decoding unit  34  include a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the first domain. It is preferable that the target reconstruction unit  35  include a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the second domain. 
     For example, regular batch normalization processing is represented by a following expression (47). 
       [Math. 50] 
         Y=B ( X )  (50)
 
     The batch normalization represented by the expression (47) corresponds to executing an arithmetic operation represented by a following expression (51) for each element. The expression (51) includes a batch mean μ d (X) of a d-th channel, and a standard deviation σ d (X) and a standard deviation σ d (X) of the d-th channel. Moreover, the expression (51) includes a shift amount γ represented by an expression (52) and corresponding to the standard deviation σ d (X), and a scaling factor β represented by an expression (53) and corresponding to the batch mean μ d (X). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               51 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     y 
                     
                       b 
                       , 
                       d 
                       , 
                       n 
                     
                   
                   = 
                   
                     
                       
                         γ 
                         d 
                       
                       ⁢ 
                       
                         
                           
                             x 
                             
                               b 
                               , 
                               d 
                               , 
                               n 
                             
                           
                           - 
                           
                             
                               μ 
                               d 
                             
                             ( 
                             X 
                             ) 
                           
                         
                         
                           
                             σ 
                             d 
                           
                           ( 
                           X 
                           ) 
                         
                       
                     
                     + 
                     
                       β 
                       d 
                     
                   
                 
               
               
                 
                   ( 
                   51 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               52 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   γ 
                   = 
                   
                     [ 
                     
                       
                         γ 
                         1 
                       
                       , 
                       … 
                           
                       , 
                       
                         γ 
                         D 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   52 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               53 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   β 
                   = 
                   
                     [ 
                     
                       
                         β 
                         1 
                       
                       , 
                       … 
                           
                       , 
                       
                         β 
                         D 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   53 
                   ) 
                 
               
             
           
         
       
     
     Here, when a concept of normalization depending on the first domain or the second domain is applied to batch normalization, the batch normalization depends on an index and is therefore represented by a following expression (54). 
       [Math. 54] 
         Y   (k)   =B   k ( X   (k) )  (54)
 
     The batch normalization represented by the expression (54) corresponds to executing an arithmetic operation represented by a following expression (55) for each element. The expression (55) includes a batch mean μ d (X (k) ) of a d-th channel, and a standard deviation σ d (X (k) ) and a standard deviation σ d (X (k) ) of the d-th channel. Moreover, the expression (56) includes a shift amount γ k  represented by an expression (57) and corresponding to the standard deviation σ d (X (k) ), and a scaling factor β k  represented by an expression (53) and corresponding to the batch mean μ d (X (k) ). 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               55 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     y 
                     
                       b 
                       , 
                       d 
                       , 
                       n 
                     
                     
                       ( 
                       k 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         γ 
                         d 
                         k 
                       
                       ⁢ 
                       
                         
                           
                             x 
                             
                               b 
                               , 
                               d 
                               , 
                               n 
                             
                             
                               ( 
                               k 
                               ) 
                             
                           
                           - 
                           
                             
                               μ 
                               d 
                             
                             ( 
                             
                               X 
                               
                                 ( 
                                 k 
                                 ) 
                               
                             
                             ) 
                           
                         
                         
                           
                             σ 
                             d 
                           
                           ( 
                           
                             X 
                             
                               ( 
                               k 
                               ) 
                             
                           
                           ) 
                         
                       
                     
                     + 
                     
                       β 
                       d 
                       k 
                     
                   
                 
               
               
                 
                   ( 
                   55 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               56 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     γ 
                     k 
                   
                   = 
                   
                     [ 
                     
                       
                         γ 
                         1 
                         k 
                       
                       , 
                       … 
                           
                       , 
                       
                         γ 
                         D 
                         k 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   56 
                   ) 
                 
               
             
           
         
       
       
         
           
             [ 
             
               Math 
               . 
                   
               57 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     β 
                     k 
                   
                   = 
                   
                     [ 
                     
                       
                         β 
                         1 
                         k 
                       
                       , 
                       … 
                           
                       , 
                       
                         β 
                         D 
                         k 
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   57 
                   ) 
                 
               
             
           
         
       
     
     Note that the networks included in the conversion learning device  3  and the conversion device  30  may include a layer that performs weight normalization or layer normalization, as mentioned above. 
     In the above description, a case is assumed in which the first domain inputted into the conversion learning device  3  and the conversion device  30  is already known. However, the first domain may be unknown. In such a case, the conversion learning device  3  and the conversion device  30  are devices that perform many-to-one conversion, and the source encoding unit  31  receives, as input, only the feature amount sequence X (k)  of the first domain, as represented by a following expression (58). 
       [Math. 58] 
       ( K   (k)   ,V   (k) )=SrcEnc( X   (k) )  (58)
 
     When the conversion device  30  performs conversion in real time in the inference phase, the learning execution unit  40  causes the source encoding unit  31  to learn under a constraint that the feature amount sequence X (k)  of the first domain that is inputted into the source encoding unit  31  before a location at which the feature amount sequence X (k)  of the first domain is to be converted into the first internal representation vector sequence Z (k) , is converted into the first internal representation vector sequence Z (k) . 
     Next, an example of processing performed in the learning phase by the conversion learning device  3  will be described with reference to  FIG.  9   .  FIG.  9    is a flowchart showing an example of the processing performed in the learning phase by the conversion learning device according to the second embodiment. Note that the processing shown in  FIG.  9    is performed each time a pair of a first domain and a second domain are inputted into the conversion learning device  3 . 
     In step S 91 , the source encoding unit  31  converts a feature amount sequence X (k)  of a first domain into a first internal representation vector sequence Z (k) . 
     In step S 92 , the target encoding unit  32  converts a feature amount sequence X (k′)  of a second domain into a second internal representation vector sequence Q (k′) ). 
     In step  593 , the attention matrix calculation unit  33  calculates an attention matrix A (k,k′)  by using the first internal representation vector sequence Z (k)  and the second internal representation vector sequence Q (k′) . 
     In step  594 , the attention matrix calculation unit  33  calculates a third internal representation vector sequence R (k,k′) , which is a product of an internal representation vector sequence V h   (k)  calculated by linear conversion of the first internal representation vector sequence Z (k)  and the attention matrix A (k,k′) . 
     In step S 95 , the target decoding unit  34  calculates a feature amount sequence F (k,k′)  of a conversion domain that is used to convert the first domain into the conversion domain by using the third internal representation vector sequence R (k,k′) . 
     In step S 96 , the target reconstruction unit  35  calculates a feature amount sequence J (k,k′)  of a reconstructed domain, which is a domain obtained by reconstructing the second domain by using the third internal representation vector sequence R (k,k′) . 
     In step  597 , the learning execution unit  40  causes at least one of the source encoding unit  31 , the target encoding unit  32 , the target decoding unit  34 , and the target reconstruction unit  35  to learn. 
     Note that the conversion learning device  3  may interchange order of step S 91  and step S 92 . The conversion learning device  3  may interchange order of step S 95  and step S 96 . 
     Next, an example of processing performed in the inference phase by the conversion device  30  will be described with reference to  FIG.  10   .  FIG.  10    is a flowchart showing an example of the processing performed in the inference phase by the conversion device according to the second embodiment. 
     In step S 101 , the source encoding unit  31  converts a feature amount sequence X (k)  of a first domain into a first internal representation vector sequence Z (k) . 
     In step S 102 , the target encoding unit  32  converts a feature amount sequence X (k′)  of a second domain into a second internal representation vector sequence Q (k′) . 
     In step S 103 , the attention matrix calculation unit  33  calculates an attention matrix A (k,k′)  by using the first internal representation vector sequence Z (k)  and the second internal representation vector sequence Q (k′) . 
     In step S 104 , the attention matrix calculation unit  33  calculates a third internal representation vector sequence R (k,k′) , which is a product of an internal representation vector sequence V h   (k)  calculated by linear conversion of the first internal representation vector sequence Z (k)  and the attention matrix A (k,k′) . 
     In step S 105 , the target decoding unit  34  calculates a feature amount sequence F (k,k′)  of a conversion domain that is used to convert the first domain into the conversion domain by using the third internal representation vector sequence R (k,k′) . 
     Note that the conversion device  30  may interchange order of step S 101  and step S 102 . 
     Next, the convolutional layers will be described with reference to expressions (59) to (65). When inputs are represented by an expression (59) and outputs are represented by an expression (60), the convolutional layers are represented by an expression (61) including a linear conversion L and an expression (62) including a non-linear conversion f. The non-linear conversion f is, for example, a sigmoid function, a rectified linear function, or a gated linear function. 
       [Math. 59] 
         X∈     D×N   (59)
 
       [Math. 60] 
         Y∈     D′×N   (60)
 
       [Math. 61] 
         X′=L ( X )  (61)
 
       [Math. 62] 
         Y=f ( X ′)  (62)
 
     Moreover, the linear conversion L with a causality-related constraint is represented, when expressed element by element, by a following expression (63) including a convolutional kernel size N′. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               63 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     x 
                     
                       
                         d 
                         ′ 
                       
                       , 
                       
                         n 
                         ′ 
                       
                     
                     ′ 
                   
                   = 
                   
                     
                       
                         ∑ 
                         
                           d 
                           = 
                           1 
                         
                         D 
                       
                       
                         
                           ∑ 
                           
                             n 
                             = 
                             
                               
                                 - 
                                 
                                   N 
                                   ′ 
                                 
                               
                               + 
                               1 
                             
                           
                           0 
                         
                         
                           
                             w 
                             
                               
                                 d 
                                 ′ 
                               
                               , 
                               d 
                               , 
                               n 
                             
                           
                           ⁢ 
                           
                             x 
                             
                               d 
                               , 
                               
                                 n 
                                 + 
                                 
                                   n 
                                   ′ 
                                 
                               
                             
                           
                         
                       
                     
                     + 
                     
                       
                         b 
                         
                           d 
                           ′ 
                         
                       
                       ( 
                       
                         
                           
                             n 
                             ′ 
                           
                           = 
                           1 
                         
                         , 
                         … 
                             
                         , 
                         N 
                         , 
                         
                           
                             d 
                             ′ 
                           
                           = 
                           1 
                         
                         , 
                         … 
                             
                         , 
                         
                           D 
                           ′ 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                   ( 
                   63 
                   ) 
                 
               
             
           
         
       
     
     Parameters to be learned by the layer are coefficients “W” and “b” of the linear conversion L, which are represented by following expressions (64) and (65), respectively. 
       [Math. 64] 
         w={ω   d′,d,n } D′×D×N′   (64)
 
       [Math. 65] 
         b={b   d′ } D   (65)
 
     Next, the self-attention layers will be described with reference to expressions (66) to (70). When inputs are represented by the expression (59) and outputs are represented by the expression (60), the self-attention layers are represented by following expressions (66) to (68). 
       [Math. 66] 
       ( Q,K,V )= L ( X )  (66)
 
       [Math. 67] 
         A =softmax n ( M+K   T   Q /√{square root over ( D ′)})  (67)
 
       [Math. 68] 
         Y=VA   (68)
 
     The expressions (63) to (68) represent the following processing. First, an internal representation vector sequence Q, an internal representation vector sequence K, and an internal representation vector sequence V represented by a following expression (69) are generated by linear conversion of an input X from the number D of channels into the number 3D′ of channels, for example, convolution with a kernel size of one. Next, a self-similarity matrix A between the internal representation vector sequence Q and the internal representation vector sequence K is calculated. Then, the outputs Y are calculated by using the expression (68). 
       [Math. 69] 
         Q,K,V∈     D′×N   (69)
 
     Note that a matrix M included in the expression (67) is a matrix in which all elements are zero when a causality-related constraint is not taken into consideration, and is a matrix in which an (m, n) element is represented by a following expression (70) when a causality-related constraint is taken into consideration. 
     
       
         
           
             [ 
             
               Math 
               . 
                   
               70 
             
             ] 
           
         
       
       
         
           
             
               
                 
                   
                     m 
                     
                       n 
                       , 
                       
                         n 
                         ′ 
                       
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           0 
                         
                         
                           
                             ( 
                             
                               n 
                               ≤ 
                               
                                 n 
                                 ′ 
                               
                             
                             ) 
                           
                         
                       
                       
                         
                           
                             - 
                             ∞ 
                           
                         
                         
                           
                             ( 
                             
                               n 
                               &gt; 
                               
                                 n 
                                 ′ 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   70 
                   ) 
                 
               
             
           
         
       
     
     Next, the multi-self-attention layers will be described with reference to expressions (71) to (78). When inputs are represented by the expression (59) and outputs are represented by the expression (60), the multi-self-attention layers execute arithmetic operations represented by following expressions (71) to (78). The expression (71) includes a linear conversion L 1 . The expression (78) includes a linear conversion L 2 . 
       [Math. 71] 
       ( Q,K,V )= L   1 ( X )  (71)
 
       [Math. 72] 
       ( Q   1   , . . . ,Q   H )=split( Q )  (72)
 
       [Math. 73] 
       ( K   1   , . . . ,K   H )=split( K )  (73)
 
       [Math. 74] 
       ( V   1   , . . . ,V   H )=split( V )  (74)
 
       [Math. 75] 
         A   h =softmax n ( M+K   h   T   Q   h /√{square root over ( D ′)})( h= 1, . . . , H )  (75)
 
       [Math. 76] 
         Y   h   =V   h   A   h ( h= 1, . . . , H )  (76)
 
       [Math. 77] 
         C =concat( Y   1   , . . . ,Y   H )  (77)
 
       [Math. 78] 
         Y=t   2 ( C )  (78)
 
     The conversion learning device  3  and the conversion device  30  according to the second embodiment have been described hereinabove. The conversion learning device  3  converts a feature amount sequence X (k)  of a first domain assigned a first index k into a first internal representation vector sequence Z (k)  by using the first machine learning model. The conversion learning device  3  converts a feature amount sequence X (k′)  of a second domain assigned a second index k′ into a second internal representation vector sequence Q (k′)  by using the second machine learning model. The conversion learning device  3  next calculates an attention matrix A (k,k′)  by using the first internal representation vector sequence Z (k)  and the second internal representation vector sequence Q (k′) , and the conversion learning device  3  calculates a third internal representation vector sequence R (k,k′) , which is a product of an internal representation vector sequence V (k)  calculated by linear conversion of the first internal representation vector sequence Z (k)  and the attention matrix A (k,k′) . The conversion learning device  3  then calculates, by using the third internal representation vector sequence R (k,k′) , a feature amount sequence I (k,k′)  of a conversion domain that is used to convert the first domain into the conversion domain, by using the third machine learning model. The conversion learning device  3  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the distance between a submatrix of the feature amount sequence X (k)  of the first domain and a submatrix of the feature amount sequence I (k,k′)  of the conversion domain becomes shorter. 
     Thus, the conversion learning device  3  can generate learning models that enable conversion between a plurality of domains with a single network set. Moreover, thus, the conversion learning device  3  can make it possible to perform effective conversion processing using the learning models. Furthermore, when the content data is speech data, the conversion learning device  3  can make it possible to perform effective conversion processing including a prosodic feature. 
     The conversion learning device  3  calculates, by using the third internal representation vector sequence R (k,k′) , a feature amount sequence J (k,k′)  of a reconstructed domain, which is a domain obtained by reconstructing the second domain, by using the fourth machine learning model. Then, the conversion learning device  3  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the distance between the feature amount sequence X (k′)  of the second domain and the feature amount sequence J (k,k′)  of the reconstructed domain becomes shorter. Thus, the conversion learning device  3  can make it possible for the conversion device  30  to output a more stable result of inference. 
     The conversion learning device  3  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that non-diagonal components of the attention matrix A (k,k′)  become smaller. Thus, the conversion learning device  3  can perform learning that is based on a premise that a spoken sentence in the content data including the first domain matches a spoken sentence of the content data including the second domain, and can make it possible for the conversion device  30  to output a more stable result of inference. 
     The conversion learning device  3  causes at least one of the target encoding unit  32  and the target decoding unit  34  to learn such that the number of orthogonal column vectors included in the attention matrix A (k,k′)  becomes larger. Thus, the conversion learning device  3  can perform learning that is based on a premise that each location of the feature amount sequence X (k)  of the first domain and each location of the feature amount sequence X (k′)  of the target domain largely correspond to each other, and can make it possible for the conversion device  30  to output a more stable result of inference. 
     In the conversion learning device  3 , the source encoding unit  31  includes a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the first domain. The target encoding unit  32  includes a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the second domain. Similarly, the target decoding unit  34  includes a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the first domain. The target reconstruction unit  35  includes a neural network including a layer that performs normalization, weight normalization, or layer normalization, depending on the second domain. Thus, the conversion learning device  3  makes it possible to perform normalization, weight normalization, or layer normalization, and can reduce a possibility that accuracy of a result of inference decreases in the inference phase. 
     The conversion learning device  3  causes the source encoding unit  31  to learn under a constraint that the feature amount sequence X (k)  of the first domain inputted into the source encoding unit  31  before a location at which the feature amount sequence X (k)  of the first domain is to be converted into the first internal representation vector sequence Z (k) , is converted into the first internal representation vector sequence Z (k) . Thus, the conversion learning device  3  makes it possible for the conversion device  30  to perform conversion processing online in real time. 
     Next, specific examples of effects brought about by the conversion learning device  3  and the conversion device  30  according to the second embodiment will be described with reference to  FIGS.  11  to  16   . In the description of the effects, all of the source encoding unit  31 , the target encoding unit  32 , the target decoding unit  34 , and the target reconstruction unit  35  include convolutional layers. 
     Moreover, for an acoustic feature amount vector, a combined form of a mel-cepstral coefficient, a logarithmic basic frequency pattern, an aperiodicity index, and a voiced/unvoiced indicator is used. The voiced/unvoiced indicator is “1” in a voiced segment, and is “0” in an unvoiced segment. 
     For an evaluation index, the mel-cepstral distortion (MCD) is used. Assuming that a mel-cepstral coefficient for the target domain and a mel-cepstral coefficient for the conversion domain are represented by the above-mentioned expressions (27) and (28), respectively, the mel-cepstral distortion is expressed as in the above-mentioned expression (29). 
       FIG.  11    shows an example of an attention matrix in a case where the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have regular batch normalization layers in the inference phase. In  FIG.  11   , a horizontal axis represents time points corresponding to feature amount vectors included in the target domain, and a vertical axis represents time points corresponding to feature amount vectors included in the source domain. Referring to  FIG.  11   , it can be understood that the peak of the distribution of attention locations gradually become unclear. This is due to use of the regular batch normalization layers. Note that when the content data is speech data, the attention locations are attention time points. 
       FIG.  12    shows an example of an attention matrix in a case where the source encoding unit, the target encoding unit, the target decoding unit, and the target reconstruction unit according to the second embodiment have batch normalization layers depending on a first domain or a second domain in the inference phase. In  FIG.  12   , a horizontal axis represents time points corresponding to feature amount vectors included in the target domain, and a vertical axis represents time points corresponding to feature amount vectors included in the source domain. Referring to  FIG.  12   , it can be understood that the peak of the distribution of attention locations changes monotonically and continuously. This is due to use of the batch normalization layers depending on the first domain or the second domain. 
       FIG.  13    shows examples of mel-cepstral coefficients in cases where the conversion device according to the second embodiment performs one-to-one conversion, and performs many-to-one conversion in the inference phase. In  FIG.  13   , a third column from the left shows mel-cepstral distortions in the case where one-to-one conversion is performed, and a fourth column from the left shows mel-cepstral distortions in the case where many-to-one conversion is performed. The many-to-one conversion does not require information on inputs and is therefore advantageous in practical scenarios, but accuracy in conversion can decrease. In light of such an aspect, it can be understood that when performing the many-to-one conversion, the conversion learning device  3  and the conversion device  30  can also demonstrate approximately as high performance as in the case of performing the one-to-one conversion. 
       FIG.  14    shows examples of mel-cepstral coefficients in cases where the conversion device according to the second embodiment performs batch conversion, and performs real-time conversion in the inference phase. In  FIG.  14   , a third column from the left shows mel-cepstral distortions in the case where batch processing is performed, and a fourth column from the left shows mel-cepstral distortions in the case where real-time processing is performed. When the two columns are compared, it can be understood that when performing real-time processing, the conversion learning device  3  and the conversion device  30  can also demonstrate approximately as high performance as in the case of performing batch processing. 
       FIG.  15    shows examples of mel-cepstral coefficients in cases where each of the conversion device according to the first embodiment and the conversion device according to the second embodiment performs conversion without taking into consideration any of the context preservation loss and the orthogonal attention loss, performs conversion taking only the orthogonal attention loss into consideration, and performs conversion taking only the context preservation loss into consideration. 
     In  FIG.  15   , third, fourth, and fifth columns from the left show mel-cepstral coefficients in the cases of conversion processing by the conversion learning device  1  and the conversion device  10  according to the first embodiment. Sixth, seventh, and eighth columns from the left of  FIG.  15    show mel-cepstral coefficients in the cases of conversion processing by the conversion learning device  3  and the conversion device  30  according to the second embodiment. Moreover, the third and sixth columns from the left of  FIG.  15    show the mel-cepstral coefficients in the case where neither the context preservation loss nor the orthogonal attention loss are taken into consideration. The fourth and seventh columns from the left of  FIG.  15    show the mel-cepstral coefficients in the case where only the context preservation loss is taken into consideration. The fifth and eighth columns from the left of  FIG.  15    show the mel-cepstral coefficients in the case where only the context preservation loss is taken into consideration. 
     Referring to  FIG.  15   , it can be understood that as more learning losses are taken into consideration, both pairs of the conversion learning device  1  and the conversion device  10  and of the conversion learning device  3  and the conversion device  30  achieve higher accuracy in conversion. 
       FIG.  16    shows examples of mel-cepstral coefficients in cases of conventional conversion devices, the conversion device according to the first embodiment, and the conversion device according to the second embodiment. In  FIG.  16   , a third column from the left shows mel-cepstral coefficients in a case of conversion processing based on “sproket”, which is a conventional technique. A fourth column from the left of  FIG.  16    shows mel-cepstral coefficients in a case of conversion processing based on sequence conversion using a recurrent neural network (RNN). A fifth column from the left of  FIG.  16    shows mel-cepstral coefficients in a case of conversion processing by the conversion device  10  according to the first embodiment. A sixth column from the left of  FIG.  16    shows mel-cepstral coefficients in a case of conversion processing by the conversion device  30  according to the second embodiment. 
     Referring to  FIG.  16   , it can be understood that both the conversion device  10  and the conversion device  30  can achieve conversion with higher accuracy than “sproket”. Moreover, referring to  FIG.  16   , it can be understood that the conversion device  10  can perform conversion processing with an equivalent degree of accuracy to the conversion processing based on the sequence conversion using the recurrent neural network. Further, considering a fact that the conversion device  10  and the conversion device  30  took approximately one day for learning while the conversion learning device using the recurrent neural network required approximately one week for learning, it can be understood that the conversion device  10  and the conversion device  30  bring about a higher effect than the conventional technique. 
     Although the description is given in the first and second embodiments by taking a case, as an example, where the content data is speech data, such a case does not constitute a limit. The content data may be, for example, still image data or moving image data. However, when the content data is still image data, the above-mentioned locations are locations on spatial coordinates on a still image, for example, locations of pixels included in the still image. When the content data is moving image data, and when attention is drawn to a still image corresponding to a specific time point in the moving image, the above-mentioned locations are locations on spatial coordinates on the still image. On the other hand, when the content data is moving image data, and when attention is drawn to a specific location on spatial coordinates, for example, a specific area including at least one pixel, the above-mentioned locations are locations on a time axis, that is, time points. 
     At least one or some of the functions included in the conversion learning device  1 , the conversion device  10 , the conversion learning device  3 , and the conversion device  30  may be implemented by hardware including circuitry such as large scale integration (LSI), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), or graphics processing unit (GPU). 
     At least one or some of the functions included in the conversion learning device  1 , the conversion device  10 , the conversion learning device  3 , and the conversion device  30  may be implemented through cooperation between such types of hardware and software. 
     For example, the software may be stored in a storage device including a non-transitory storage medium, and may be read and executed by the conversion learning device  1 , the conversion device  10 , the conversion learning device  3 , and the conversion device  30 . Such a storage device is, for example, a hard disk drive (HDD) or a solid state drive (SSD). 
     Alternatively, the software may be stored in a storage device including a removable non-transitory storage medium, and may be read and executed by the conversion learning device  1 , the conversion device  10 , conversion learning device  3 , or the conversion device  30 . Such a storage device is, for example, a DVD or a CD-ROM. 
     Embodiments of the present invention have been described in detail hereinabove, with reference to the drawings. However, embodiments of the present invention are not limited to the above-described embodiments, and at least one of various modifications, substitutions, and changes in design may be made within a scope that does not depart from the gist of the present invention. 
     REFERENCE SIGNS LIST 
     
         
         
           
               1 ,  3  Conversion learning device 
               10 ,  30  Conversion device 
               11 ,  31  Source encoding unit 
               12 ,  32  Target encoding unit 
               13 ,  33  Attention matrix calculation unit 
               14 ,  34  Target decoding unit 
               15 ,  35  Target reconstruction unit 
               16  Forward attention correction execution unit 
             P, Q Processing block