SPEECH SYNTHESIS SYSTEM

Abstract: The present invention relates to a speech synthesis system of the type which comprises synthesis parameter generating means (5, 6, 7, 8, 10, 11) for generating reference synthesis parameters (p, q) representing items of speech, and storage means (4) for storing the reference synthesis parameters. The system also comprises input means (1) for receiving an item of text to be synthesised, analysis means (15) for analysing the item of text, calculating means (13, 16) utilising the stored reference synthesis parameters and the results of the analysis of the item of text to create a set of operational synthesis parameters representing the item of text, and synthetic speech generating means (6, 7, 9, 17) utilising the created set of operational synthesis parameters to generate synthesised speech representing the item of text. According to the invention the system is characterised in that the synthesis parameter generating means comprises means for generating a first set of reference synthesis parameters in response to the receipt of a first item of natural speech, and means for generating a second set of reference synthesis parameters in response to the receipt of a second item of natural speech. The calculating means utilises the first and second set of reference synthesis parameters in order to create the set of operational synthesis parameters representing the item of text.

Description: The present invention relates to a speech synthesis system which can produce items of speech at different speeds of delivery while maintaining at a high quality the phonetic characteristics of the item of speech produced. In the speaking of items of natural speech, their speaking speeds, hence their durations, may be varied due to various factors. For example, the duration of a spoken sentence as a whole may be extended or reduced according to the speaking tempo. Also, the durations of certain phrases and words may be locally extended or reduced according to linguistic constraints such as structures, meanings and contents, etc., of sentences. Further, the durations of syllables may be extended or reduced according to the number of syllables spoken in one breathing interval. Therefore, it is necessary to control the duration of items of synthesised speech in order to obtain synthesised speech of high quality, similar to natural speech. In the prior art, there have been proposed two techniques for controlling the duration of items of synthetic speech. In one of the techniques, synthesis parameters forming certain portions of each item are removed or repeated, while, in the other of the techniques, periods of synthesis frames are varied (Periods of analysis frames are fixed). These techniques are described in Japanese Published Unexamined Patent Application No. 50- 62,709, for example. However, the above-mentioned technique of removing and repeating certain synthesis parameters requires the finding of constant vowel portions by inspection and setting them as variable portions beforehand, thus requiring complicated operations. Further, as the duration of an item of speech varies, the phonetic characteristics also change since the dynamic features of articulatory organs transform. For example, the formants of vowels are generally neutralised as the duration of an item of speech is reduced. In this prior technique, it is impossible to reflect such changes in synthesised items of speech. In the other technique of varying the periods of synthesis frames, although the duration of an item of speech can be varied conveniently, all the portions thereof will be extended or reduced uniformly. Since ordinary items of speech comprise portions extended or reduced remarkably or slightly, such a prior technique would generate quite unnaturally synthesized items of speech. Of course, this prior technique cannot reflect the above-stated changes of the phonetic characteristics in synthesised items of speech. The object of the present invention is to provide an improved speech synthesis system. The present invention relates to a speech synthesis system of the type comprising synthesis parameter generating means for generating reference synthesis parameters corresponding to synthesis units, storage means for storing the reference synthesis parameters, input means for receiving text to be synthesized, analysis means for analysing the text, calculating means utilising the stored reference synthesis parameters and the results of the analysis of the text to create a set of operational synthesis parameters corresponding to synthesis units representing the text, and synthetic speech generating means utilising the created set of operational synthesis parameters to generate synthesized speech representing the text. According to the invention the system is characterised in that the synthesis parameter generating means comprises means for generating a first set of reference synthesis parameters in response to the receipt of natural speech spoken at a relatively high speed and corresponding to one synthesis unit, means for generating a second set of reference synthesis parameters in response to the receipt of natural speech spoken at a relatively low speed and corresponding to another synthesis unit, and in that the calculating means comprises means for interpolating between the first and second sets of reference synthesis parameters in order to create the set of operational synthesis parameters for the synthesis units representing the text, means for calculating an interpolation variable based on the required duration of the synthesised speech, and means for utilising the interpolation variable to control the creation of said set of operational synthesis parameters so that said synthesised speech is generated at the required speed between the relatively high speed and the relatively low speed. The invention also provides a method of generating synthesised speech according to claim 6. In order that the invention may be more readily understood an embodiment will now be described with reference to the accompanying drawings, in which: Fig. 1 is a block diagram of a speech synthesis system according to the present invention, Fig. 2 is a flow chart illustrating the operation of the system illustrated in Fig. 1, Figs. 3 to 8 are diagrams for explaining in greater detail the operation illustrated in Fig. 2, Fig. 9 is a block diagram of another speech synthesis system according to the invention, Fig. 10 is a diagram for explaining a modification in the operation of the system illustrated in Fig. 1, Fig. 11 is a flow chart for explaining the modification illustrated in Fig. 10, and Fig. 12 is a diagram explaining another modification in the operation of the system illustrated in Fig. 1. Referring now to the drawings, a speech synthesis system according to the present invention will be explained in more detail with reference to an embodiment thereof applied to a Japanese text-to-speech synthesis system by rules. Such a text-to-speech synthesis system performs an automatic speech synthesis from any input text and generally includes four stages of (1) inputting an item of text, (2) analysing each sentence in the item of text, (3) generating speech synthesis parameters representing the items of text, and (4) outputting an item of synthesised speech. In the stage (2), phonetic data and prosodic data relating to the item of speech are determined with reference to a Kanji-Kana conversion dictionary and a prosodic rule dictionary. In the stage (3), the speech synthesis parameters are sequentially read out with reference to a parameter file. In the speech synthesis system to be described the output item of synthesised speech is generated using the previous input of two items of speech, as will be described below. A composite speech synthesis parameter file is employed. This will also be described later more in detail. In a speech synthesis system for speech synthesis of items of Japanese text, 101 Japanese syllables are used. Fig. 1 illustrates a one form of speech synthesis system according to the present invention. As illustrated in Fig. 1, the speech synthesis system includes a workstation 1 for inputting an item of Japanese text and for performing Japanese language processing such as Kanji-Kana conversions. The workstation 1 is connected through a line 2 to a host computer 3 to which an auxiliary storage 4 is connected. Most of the components of the system can be implemented by programs executed by the host computer 3. The components are illustrated by blocks indicating their functions for ease of understanding of the system. The functions in these blocks are detailed in Fig. 2. In the blocks of Figs. 1 and 2, like portions are illustrated with like numbers. A personal computer 6 is connected to the host computer 3, through a line 5, and an A/D - D/A converter 7 is connected to the personal computer 6. A microphone 8 and a loud speaker 9 are connected to the converter 7. The personal computer 6 executes routines for performing the A/D conversions and D/A conversions. In the above system, when an item of speech is input into the microphone 8, the input speech item is A/D converted, under the control of the personal computer 6, and then supplied to the host computer 3. A speech analysis function 10, 11 in the host computer 3 analyses the digital speech data for each of a series of analysis frame periods of time length T₀, generates speech synthesis parameters, and stores these parameters in the storage 4. This is illustrated by lines l₁ and l₂ in Fig. 3. For the lines l₁ and l₂, the analysis frame periods are shown each of length T₀ and the speech synthesis parameters are represented by p i and q i . In this embodiment, line spectrum pair parameters are employed as synthesis parameters, although α parameters, formant parameters, PARCOR coefficients, and so on may alternatively be employed. A parameter train for an item of text to be synthesised into speech is illustrated by line l₃ in Fig. 3. This parameter train is divided into M synthesis frame periods of lengths T₁ - T M respectively which are variables. The synthesis parameters are represented by r i . The parameter train will be explained later in more detail. The synthesis parameters of the parameter train are sequentially supplied to a speech synthesis function 17 in the host computer 3 and digital speech data representing the text to be synthesised is supplied to the converter 7 through the personal computer 6. The converter 7 converts the digital speech data to an analogue speech data under the control of the personal computer 6 to generate an item of synthesised speech through the loud speaker 9. Fig. 2 illustrates the operation of this embodiment as a whole. As illustrated in Fig. 2, a synthesis parameter file is first established by speaking into the microphone 8 one of the synthesis units used for speech synthesis, i.e., one of the 101 Japanese syllables in this example for example), at a relatively low speed. This synthesis unit is analysed (Step 10). The resultant analysis data is divided into M consecutive synthesis frame periods, each having a time length T₀, for example, as shown in line l₁ in Fig. 3. The total time duration t₀ of this analysis data is (M × T₀) . Next, further items for the synthesis parameter file are obtained by speaking the same synthesis unit at a relatively high speed. This synthesis unit is analysed (Step 11). The resultant analysis data is divided into N consecutive synthesis frame periods, each having a time length T₀, for example, as shown in the line l₂ in Fig. 3. The total time duration t₁ of this analysis data is (N × T₀) . Then, the analysis data in the lines l₁ and l₂ are matched by the DP matching (Step 12). This is illustrated in Fig. 4. A path P which has the smallest cumulative distance between the frame periods is obtained by the DP matching, and the frame periods in the lines l₁ and l₂ are matched in accordance with the path P. In practice, the DP matching can move only in two directions, as illustrated in Fig. 5. Since one of the frame periods in the speech item spoken at the lower speed should not correspond to more than one of the frame periods in the speech item spoken at the higher speed, such a matching is prohibited by the rules illustrated in Fig. 5. Thus, similar frame periods have been matched between the lines l₁ and l₂, as illustrated in Fig. 3. Namely, p₁ ←→ q₁, p₂ ←→ q₂, p₃ ←→ q₂, have been matched as similar frame periods. A plurality of the frame periods in line l₁ may correspond to only one frame period in line l₂. In such a case, the frame period in the line l₂ is equally divided into portions and one of these portions is deemed to correspond to each of the plurality of frame periods in line l₁. For example, in Fig. 3, the second frame period in line l₁ corresponds to a half portion of the second frame period in line l₂. As the result, the M frame periods in line l₁ correspond to the N frame period portions in line l₂, on a one to one basis. It is apparent that the frame period portions in line l₂ do not always have the same time lengths. An item of synthesised speech, extending over a time duration t between the time durations t₀ and t₁, is illustrated by line l₃ in Fig. 3. This item of synthesised speech is divided into M frame periods, each corresponding to one frame period in line l₁ and to one frame period portion in line l₂. Accordingly, each of the frame periods in the item of synthesised speech has a time length interpolated between the time length of the corresponding frame period in line l₁, i.e., T₀, and the time length of the corresponding frame period portion in line l₂. The synthesis parameters r i of each of the frame periods in line l₃ are parameters interpolated between the corresponding synthesis parameters p i and q j of lines l₁ and l₂. After the DP matching, a frame period time length variation Δ T i and a parameter variation Δ p i for each of the frame periods are to be obtained (Step 13). The frame period time length variation Δ T i indicates a variation from the frame period length of the "i"th frame period in line l₁, i.e., T₀, to the frame period length of the frame period portion in the line l₂ corresponding to the "i"th frame period in line l₁. In Fig. 3, Δ T₂ is shown as an example thereof. When the frame in the line l₂ corresponding to the "i"th frame period in line l₁ is denoted as the "j"th frame period in line l₂, Δ T i may be expressed as where n j denotes the number of frame periods in line l₁ corresponding to the "j"th frame period in line l₂. When the total time duration t of the item of synthesised speech is expressed by linear interpolation between t₀ and t₁, with t₀ selected as the origin for interpolation, the following expression may be obtained. t = t₀ + x ( t₁ - t₀ ) where 0 ≦ x ≦ 1 . The x in the above expression is hereinafter referred to as an interpolation variable. As the interpolation variable approaches 0, the time duration t approaches the origin for interpolation. When expressed with the interpolation variable x and the variation Δ T i , the time length T i of each of the frame periods in the item of synthesised speech may be expressed by the following interpolation expression with the frame period length T₀ selected as the origin for interpolation. T i = T₀ - x Δ T i Thus, by obtaining Δ T i , the length T i of each of the frame periods in the item of synthesised speech, extending over any duration between t₀ - t₁, can be obtained. On the other hand, the synthesis parameter variation Δ p i is ( p i q j ) and the synthesis parameters r i of each of the frame periods in the item of synthesised speech may be obtained by the following expression. r i = p i - x Δ p i Accordingly, by obtaining Δ p i , the synthesis parameters r i of each of the frame periods in the item of synthesised speech, extending over any duration between t₀ - t₁, can be obtained. The variations Δ T i and Δ p i thus obtained are stored in the auxiliary storage 4 together with p i with a format such as illustrated in Fig. 7. The above processing is performed for each of the synthesis units for speech synthesis to constitute a composite parameter file ultimately. With the synthesis parameter file constituted, a text-to-speech synthesis operation can be started, and an item of text is input (Step 14). This item of text is input at the workstation 1 and the text data is transferred to the host computer 3, as stated before. A sentence analysis function 15 in the host computer 3 performs Kanji-Kana conversions, determinations of prosodic parameters, and determinations of durations of synthesis units. This is illustrated in the following Table 1 showing the flow chart of the function and a specific example thereof. In this example, the duration of each of the phonemes (consonants and vowels) is first obtained and then the duration of a syllable, i.e., a synthesis unit, is obtained by summing up all the durations of the phonemes. Thus, with the duration of each of the synthesis units in the text obtained by the sentence analysis function, the period length and synthesis parameters of each of the frame periods corresponding to the item of text are next to be obtained by interpolation for each of the synthesis units (Step 16, Fig. 2), as illustrated in detail in Fig. 6. An interpolation variable x is first obtained. Since t = t₀ + x ( t₁ - t₀ ) , the following expression is obtained (Step 161). From the above expression, it can be seen to what extent each of the synthesis units is near to the origin for interpolation. Next, the length T i and the synthesis parameters r i of each of the frame periods in each of the synthesis units are obtained from the following expressions, respectively, with reference to the parameter file (Steps 162 and 163). T i = T₀ - x Δ T i r i = p i - x Δ p i Thereafter, an item of synthesised speech is based on the period length T i and the synthesis parameters r i (Step 17 in Fig. 2). The speech synthesis function may typically be implemented as schematically illustrated in Fig. 8 by a sound source 18 and a filter 19. Signals indicating whether a sound is voiced (pulse train) or unvoiced (white noise) (indicated with U and V, respectively) are supplied as sound source control data, and line spectrum pair parameters, etc., are supplied as filter control data. As the result of the above processing, items of text, for example shown in Table 1, are synthesised and are outputted through the loud speaker 9. The following Tables 2 through 5 show, as an example, the processing of the syllable "WA" into synthesised speech extending over the duration of 172 ms decided as shown in Table 2. Table 2 shows the analysis of an item of synthesised speech representing the syllable "WA" having the analysis frame period of 10 ms and extending over a duration of 200 ms (the item of speech is spoken at a lower speed), and Table 3 shows the analysis of the item of synthesised speech representing the syllable "WA" having the same frame period and extending over a duration of 150 ms (the item of speech is spoken at a higher speed). Table 4 shows the correspondence between these items of speech by the DP matching. The portion of "WA" in the synthesis parameter file prepared according to Tables 2 to 4 is shown in Table 5 (the line spectrum pair parameters are shown only as to the first parameters). Table 5 shows also the time length and synthesis parameters (the first parameters) of each of the frame periods in the items of synthesised speech representing the syllable "WA" extending over a duration of 172 ms. In Table 5, p i , Δ p i , q j , and r i are shown only as to the first parameters. While the invention has been described above with respect to the speech synthesis system illustrated in Fig. 1, it is alternatively possible to implement the invention with a small system by employing a signal processing board 20 as illustrated in Fig. 9. In the system illustrated in Fig. 9, a workstation 1A performs the functions of editing a sentence, analysing the sentence, calculating variations, interpolation, etc. In Fig. 9, the portions having the functions equivalent to those illustrated in Fig. 1 are illustrated with the same reference numbers. The detailed explanation of this example is therefore not needed. Next, two modifications of the above described system will be explained. In one of the modifications, training of the synthesis parameter file is introduced. First, a consideration is made as to errors which would be caused when such a training is not performed. Fig. 10 illustrates the relations between synthesis parameters and durations of items of synthesised speech. As illustrated in Fig. 10, to generate the synthesis parameters r i from the synthesis parameters p i for an item of speech spoken at the lower speed (extending for a time duration t₁) and the synthesis parameters q j for an item of speech spoken at the higher speed, interpolation is performed by using a line OA₁, as shown by a broken line (a). To generate synthesis parameters r i ′ from synthesis parameters s k for another item of speech spoken at another higher speed (extending for a time duration t₂) and the synthesis parameters p i , interpolation is performed by using a line OA₂, as shown by a broken line (b). It will be seen that the synthesis parameters r i and r i ′ are different from each other. This is due to the errors, etc., caused in matching by the DP matching operation. In the modification, the synthesis parameters r i are generated by using a line OA′ which is obtained by averaging the lines OA₁ and OA₂, so that there will be a high probability that the errors of the lines OA₁ and OA₂ will be offset by each other, as seen from Fig. 10. Although the training is performed once in the example shown in Fig. 10, it is obvious that training of this type would result in smaller errors, as in this modification. Fig. 11 illustrates the operation of this modification, with functions similar to those in Fig. 2 illustrated with similar numbers. The operation need not therefore be explained here in detail. As illustrated in Fig. 11, the synthesis parameter file is updated in Step 21, and the need for training is judged in Step 22 so that the Steps 11, 12, and 21 can be repeated as requested. In Step 21, Δ T i ′ and Δ p i ′ are obtained according to the following expressions, Δ p i ' = Δ p i + ( p i - q j ) It is obvious that a processing similar to the Steps in Fig. 2 is performed since Δ T i ′ = 0 and Δ p i ′ = 0 in the initial stage. When the parameter values after training corresponding to those before a training are denoted, respectively, with dashes attached thereto, as the following expressions are obtained (See Fig. 10). Accordingly, when the parameter values after training corresponding to those before training, Δ p i and Δ T i , are denoted as Δ p i ′ and Δ T i ′, respectively, the following expressions are obtained. Δ p i ' = ( p i - q j )' = Δ p i + ( p i - s k ) Further, when an interpolation variable after training is denoted as x′, the following expressions are obtained. In Step 21 in Fig. 11 k and s are replaced with j and q, respectively, since there is no possibility of causing any confusion thereby in expressions. Another modification will now be explained. In the above described system, the synthesis parameters obtained by analysing an item of speech spoken at a lower speed are used as the origin for interpolation. Therefore, an item of synthesised speech to be produced at a speed near that of the item of speech spoken at the lower speed would be of high quality since synthesis parameters near the origin for interpolation can be employed. On the other hand, the higher the production speed of an item of synthesised speech is, the more the quality would be deteriorated. Accordingly, it would be quite effective, for improving the quality of an item of synthesised speech in the applications to the text-to-speech synthesis, etc., to employ synthesis parameters obtained by analysing an item of speech spoken at such a speed as is used most frequently (this speed is hereinafter referred to as "a standard speed") as the origin for interpolation. In that case, as to an item of synthesised speech to be produced at a speaking speed higher than the standard speed, the above-stated embodiment itself may be applied thereto by employing the synthesis parameters obtained by analysing an item of speech spoken at the standard speed as the origin for interpolation. On the other hand, as to an item of synthesised speech to be produced at a speaking speed lower than the standard speed, a plurality of frames in the item of speech spoken at the lower speed may correspond to one frame in the item of speech spoken at the standard speed, as illustrated in Fig. 12, and in such a case, the average of the synthesis parameters of the plurality of frame periods is employed as the origin for interpolation on the side of the item of speech spoken at the lower speed. More specifically, when the duration of the item of speech spoken at the standard speed is denoted as t₀ ( t₀ = MT₀ ) and the duration of the item of speech spoken at the lower speed is denoted as t₁ ( t₁ = NT₀ , N > M ), the synthesis parameters of each of the M frame periods in the items of synthesised speech, extending over the duration t ( t₀ ≦ t ≦ t₁ ), are obtained (See Fig. 12). When t = t₀ + x (t₁ - t₀ ) , the frame period duration T i and the synthesis parameters ri of the "i"th frame period are respectively expressed as T i = T₀ + x T₀ ( n i - 1 ) where p i denotes the synthesis parameters of the "i"th frame period in the item of speech spoken at the standard speed, q j denotes the synthesis parameters of the "j"th frame period in the item of speech spoken at the lower speed, J i denotes a set of the frame periods in the item of speech spoken at the lower speed corresponding to the "i"th frame period in the item of speech spoken at the standard speed, and n i denotes the number of elements of J i . Thus, by determining uniquely the synthesis parameters of each of the frame periods in the item of speech spoken at the lower speed, corresponding to each of the frame periods in the item of speech spoken at the standard speed, in accordance with the expression it is possible to determine the synthesis parameters for an item of synthesised speech to be produced at a lower speed than the standard speed by interpolation. Of course, it is also possible to perform the training of the synthesis parameters in this case. A speed synthesis system as described above can produce items of synthesised speech extending over a variable duration by interpolating the synthesis parameters obtained by analysing items of speech spoken at different speeds. The interpolation operation is convenient and can add the characteristics of the original synthesis parameters. Therefore, it is possible to produce an item of synthesised speech extending over a variable time duration conveniently without deteriorating the phonetic characteristics of the synthesised speech. Further, since training is possible, the quality of the item of synthesised speech can be improved more as required. The system can be applied to any language. The synthesis parameter file may be provided as a package.

Claims: A speech synthesis system comprising synthesis parameter generating means (5, 6, 7, 8, 10, 11) for generating reference synthesis parameters (p, q) corresponding to synthesis units, storage means (4) for storing said reference synthesis parameters, input means (1) for receiving text to be synthesised, analysis means (15) for analysing said text, calculating means (13, 16) utilising said stored reference synthesis parameters and the results of the analysis of said text to create a set of operational synthesis parameters corresponding to synthesis units representing said text, and synthetic speech generating means (6, 7, 9, 17) utilising said created set of operational synthesis parameters to generate synthesised speech representing said text, characterised in that said synthesis parameter generating means comprises means for generating a first set of reference synthesis parameters (p) in response to the receipt of natural speech spoken at a relatively high speed and corresponding to one synthesis unit, means for generating a second set of reference synthesis parameters (q) in response to the receipt of natural speech spoken at a relatively low speed and corresponding to another synthesis unit, and in that said calculating means comprises means for interpolating between said first and second sets of reference synthesis parameters in order to create said set of operational synthesis parameters (r) for said synthesis units representing said text, means for calculating an interpolation variable based on the required duration of said synthesised speech, and means for utilising said interpolation variable to control the creation of said set of operational synthesis parameters so that said synthesised speech is generated at the required speed between said relatively high speed and said relatively low speed. A speech synthesis system as claimed in Claim 1 characterised in that said synthesis parameter generating means comprises means for generating a third set of reference synthesis parameters in response to the receipt of natural speech spoken at a normal speed and corresponding to a further synthesis unit, and in that said calculating means comprises means for utilising any two of said first, second and third sets of reference synthesis parameters in order to create said set of operational synthesis parameters. A speech synthesis system as claimed in either of the preceding claims characterised in that said synthesis parameter generating means comprises means for subdividing said received natural speech into a set of time periods, and means for generating reference synthesis parameters for each of said time periods. A speech synthesis system as claimed in any one of the preceding claims characterised in that said synthesis parameter generating means comprises means for comparing said sets of reference synthesis parameters with each other in order to obtain a parameter variation factor, and said calculating means utilises said parameter variation factor to control the creation of said set of operational synthesis parameters. A speech synthesis system as claimed in any one of the preceding claims characterised in that said synthesis parameter generating means comprises means for training said sets of reference synthesis parameters in order to avoid errors in the creation of said set of operational synthesis parameters. A method of generating synthesised speech comprising generating reference synthesis parameters (p, q) corresponding to synthesis units, storing said reference synthesis parameters, receiving text to be synthesised, analysing said text, utilising said stored reference synthesis parameters and the results of the analysis of said text to create a set of operational synthesis parameters corresponding to synthesis units representing said text, and utilising said created set of operational synthesis parameters to generate synthesised speech representing said text, characterised in that said synthesis parameters are generated by generating a first set of reference synthesis parameters (p) in response to the receipt of natural speech spoken at a relatively high speed and corresponding to one synthesis unit, generating a second set of reference synthesis parameters (q) in response to the receipt of natural speech spoken at a relatively low speed and corresponding to another synthesis unit, and in that said stored reference synthesis parameters are utilised by interpolating between said first and second sets of reference synthesis parameters in order to create said set of operational synthesis parameters (r) for said synthesis units representing said text, calculating an interpolation variable based on the required duration of said synthesised speech, and utilising said interpolation variable to control the creation of said set of operational synthesis parameters so that said synthesised speech is generated at the required speed between said relatively high speed and said relatively low speed.

Inventors: KANEKO, HIROSHI
Applicants: INTERNATIONAL BUSINESS MACHINES CORPORATION
Application year: 1987
Main classification: G10L5
IPC codes: G10L13, G10L21