Patent Publication Number: US-11398219-B2

Title: Speech synthesizer using artificial intelligence and method of operating the same

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to International Patent Application No. 10-2019-0113337 filed on Sep. 16, 2019 in Korea, the entire contents of which is hereby incorporated by reference in its entirety. 
     BACKGROUND 
     The present disclosure relates to a speech synthesizer and, more particularly, a speech synthesizer capable of improving reading break prediction performance. 
     Competition for speech recognition technology which has started in smartphones is expected to become fiercer in the home with diffusion of the Internet of things (IoT). 
     In particular, an artificial intelligence (AI) device capable of issuing a command using speech and having a talk is noteworthy. 
     A speech recognition service has a structure for selecting an optimal answer to a user&#39;s question using a vast amount of database. 
     A speech search function refers to a method of converting input speech data into text in a cloud server, analyzing the text and retransmitting a real-time search result to a device. 
     The cloud server has a computing capability capable of dividing a large number of words into speech data according to gender, age and intonation and storing and processing the speech data in real time. 
     As more speech data is accumulated, speech recognition will be accurate, thereby achieving human parity. 
     Recently, services for providing a synthesized speech in specific speaker&#39;s voice using a synthesized speech model have appeared. 
     For reading break learning of the synthesized speech model, a training set including one sentence (training data) and labeling data for labeling words configuring the sentence with reading break is required. 
     The reading break may be classified into first reading break, second reading break greater than the first reading break and third reading break greater than the second reading break. 
     When data having imbalance such as the count of specific reading break less than that of other reading break is used upon outputting the synthesized speech of one sentence, performance of the synthesized speech model may deteriorate. 
     In addition, in an existing synthesized speech model, only a uniformly synthesized speech in which words are read with break according to the utterance style of a specific voice actor without considering the utterance style of a user was provided. Therefore, sometimes, the user may feel uncomfortable when listening to the synthesized speech. 
     SUMMARY 
     Another object of the present disclosure is to provide a speech synthesizer capable of improving reading break prediction performance using artificial intelligence when a synthesized speech is output. 
     Another object of the present disclosure is to provide a speech synthesizer capable of outputting a synthesized speech optimized for a reading break style of a person using artificial intelligence. 
     According to an embodiment, provided is a speech synthesizer using artificial intelligence including a memory, a communication processor configured to receive utterance information of words uttered by a user from a terminal, and a processor configured to acquire a plurality of utterance intonation phrase (IP) ratios respectively corresponding to a plurality of words uttered by the user based on the utterance information, compare a plurality of IP ratio tables respectively corresponding to a plurality of voice actors with the plurality of utterance IP ratios, acquire a plurality of non-utterance IP ratios respectively corresponding to a plurality of unuttered words based on a result of comparison, and generate a personalized synthesized speech model based on the plurality of utterance IP ratios and the plurality of non-utterance IP ratios, wherein a plurality of classes indicating reading break of a word includes a first class corresponding to first reading break, a second class corresponding to second reading break greater than the first reading break and a third class corresponding to third reading break greater than the second reading break, wherein a minor class has a smallest count among the first to third classes, and wherein each of the utterance IP ratios and the non-utterance IP ratios is a ratio in which a word is classified as the minor class. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram illustrating a terminal according to the present disclosure. 
         FIG. 2  is a diagram illustrating a speech system according to an embodiment of the present disclosure. 
         FIG. 3  is a diagram illustrating a process of extracting utterance features of a user from a speech signal according to an embodiment of the present disclosure. 
         FIG. 4  is a diagram illustrating an example of converting a speech signal into a power spectrum according to an embodiment of the present disclosure. 
         FIG. 5  is a block diagram illustrating the configuration of a speech synthesis server according to an embodiment of the present disclosure. 
         FIGS. 6 and 7  are views illustrating a class imbalance problem when reading break is predicted through a conventional synthesized speech. 
         FIG. 8  is a flowchart illustrating a method of operating a speech synthesis server according to an embodiment of the present disclosure. 
         FIG. 9  is a flowchart illustrating a process of performing data augmentation of a word based on prior information according to an embodiment of the present disclosure. 
         FIG. 10  is a view showing an IP frequency number and a non-IP frequency number of each word stored in a database according to an embodiment of the present disclosure. 
         FIG. 11  is a view illustrating an oversampling rate determined according to a ratio of a non-IP frequency number to an IP frequency number. 
         FIG. 12  is a ladder diagram illustrating a method of operating a system according to an embodiment of the present disclosure. 
         FIG. 13  is a view illustrating a basic structure of a recurrent neural network. 
         FIG. 14  is a view illustrating a process of classifying words configuring a sentence into classes using a synthesized speech model according to an embodiment of the present disclosure. 
         FIG. 15  is a flowchart illustrating a method of operating a speech synthesis server using artificial intelligence according to another embodiment of the present disclosure. 
         FIG. 16  is a diagram illustrating prior information of each word stored in a database according to an embodiment of the present disclosure. 
         FIG. 17  is a diagram illustrating an example of a synthesized speech model according to an embodiment of the present disclosure. 
         FIGS. 18 to 21  are diagrams illustrating an example of correcting a class classification probability set in consideration of an IP ratio and a non-IP ratio of a word according to an embodiment of the present disclosure. 
         FIG. 22  is a diagram illustrating a process of learning a synthesized speech model using a newly acquired second class classification probability set based on an IP ratio according to an embodiment of the present disclosure. 
         FIG. 23  is a ladder diagram illustrating a method of operating another system of the present disclosure. 
         FIG. 24  is a view illustrating a plurality of IP tables respectively corresponding to a plurality of voice actors according to an embodiment of the present disclosure. 
         FIG. 25  is a view illustrating a normal IP ratio table according to an embodiment of the present disclosure. 
         FIG. 26  is a view illustrating a table including personalized IP ratios and normal IP ratios of acquired words based on utterance information of a user. 
         FIG. 27  is a view illustrating an IP ratio model according to an embodiment of the present disclosure. 
         FIG. 28  is a view illustrating a result of estimating personalized IP ratios of words unuttered by a user according to an embodiment of the present disclosure. 
         FIG. 29  is a flowchart illustrating a process of updating a personalized synthesized speech model at a speech synthesizer according to an embodiment of the present disclosure. 
         FIG. 30  is a ladder diagram illustrating a method of operating a system according to another embodiment of the disclosure. 
         FIG. 31  is a view illustrating a personalized IP ratio table acquired based on utterance information of a specific user according to an embodiment of the present disclosure. 
         FIG. 32  is a view illustrating a process of comparing a personalized IP ratio table with a plurality of IP ratio tables respectively corresponding to a plurality of voice actors according to an embodiment of the present disclosure. 
         FIG. 33  is a view illustrating an example of using IP ratios included in a closest IP ratio table as IP ratios of unuttered words according to an embodiment of the present disclosure. 
         FIG. 34  is a view illustrating a process of training a personalized synthesized speech model according to an embodiment of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
     Description will now be given in detail according to exemplary embodiments disclosed herein, with reference to the accompanying drawings. For the sake of brief description with reference to the drawings, the same or equivalent components may be provided with the same reference numbers, and description thereof will not be repeated. In general, a suffix such as “module” or “unit” may be used to refer to elements or components. Use of such a suffix herein is merely intended to facilitate description of the specification, and the suffix itself is not intended to have any special meaning or function. In the present disclosure, that which is well-known to one of ordinary skill in the relevant art has generally been omitted for the sake of brevity. The accompanying drawings are used to help easily understand various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. As such, the present disclosure should be construed to extend to any alterations, equivalents and substitutes in addition to those which are particularly set out in the accompanying drawings. 
     While ordinal numbers including ‘first’, ‘second’, etc. may be used to describe various components, they are not intended to limit the components. These expressions may be used to distinguish one component from another component 
     When it is said that a component is ‘coupled with/to’ or ‘connected to’ another component, it should be understood that the one component is connected to the other component directly or through any other component in between. On the other hand, when it is said that a component is ‘directly connected to’ or ‘directly coupled to’ another component, it should be understood that there is no other component between the components. 
     The terminal described in this specification may include cellular phones, smart phones, laptop computers, digital broadcast terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigators, portable computers (PCs), slate PCs, tablet PCs, ultra books, wearable devices (for example, smart watches, smart glasses, head mounted displays (HMDs)), and the like. 
     However, the artificial intelligence device  100  described in this specification is applicable to stationary terminals such as smart TVs, desktop computers or digital signages. 
     In addition, the terminal  100  according to the embodiment of the present disclosure is applicable to stationary or mobile robots. 
     In addition, the terminal  100  according to the embodiment of the present disclosure may perform the function of a speech agent. The speech agent may be a program for recognizing the speech of a user and audibly outputting a response suitable to the recognized speech of the user. 
     The terminal  100  may include a wireless communication unit  110 , an input unit  120 , a learning processor  130 , a sensing unit  140 , an output unit  150 , an interface  160 , a memory  170 , a processor  180  and a power supply  190 . 
     The wireless communication unit  110  may include at least one of a broadcast reception module  111 , a mobile communication module  112 , a wireless Internet module  113 , a short-range communication module  114  and a location information module  115 . 
     The broadcast reception module  111  receives broadcast signals and/or broadcast associated information from an external broadcast management server through a broadcast channel. 
     The mobile communication module  112  may transmit and/or receive wireless signals to and from at least one of a base station, an external terminal, a server, and the like over a mobile communication network established according to technical standards or communication methods for mobile communication (for example, Global System for Mobile Communication (GSM), Code Division Multi Access (CDMA), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), Wideband CDMA (WCDMA), High Speed Downlink Packet access (HSDPA), HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE) LTE-A (Long Term Evolution-Advanced), and the like). 
     The wireless Internet module  113  is configured to facilitate wireless Internet access. This module may be installed inside or outside the terminal  100 . The wireless Internet module  113  may transmit and/or receive wireless signals via communication networks according to wireless Internet technologies. 
     Examples of such wireless Internet access include Wireless LAN (WLAN), Wireless Fidelity (Wi-Fi), Wi-Fi Direct, Digital Living Network Alliance (DLNA), Wireless Broadband (WiBro), Worldwide Interoperability for Microwave Access (WiMAX), High Speed Downlink Packet Access (HSDPA), HSUPA (High Speed Uplink Packet Access), Long Term Evolution (LTE), LTE-A (Long Term Evolution-Advanced), and the like. 
     The short-range communication module  114  is configured to facilitate short-range communication and to support short-range communication using at least one of Bluetooth™, Radio Frequency IDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand (UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity (Wi-Fi), Wi-Fi Direct, Wireless USB (Wireless Universal Serial Bus), and the like. 
     The location information module  115  is generally configured to acquire the position (or the current position) of the mobile terminal. Representative examples thereof include a Global Position System (GPS) module or a Wi-Fi module. As one example, when the terminal uses a GPS module, the position of the mobile terminal may be acquired using a signal sent from a GPS satellite. 
     The input unit  120  may include a camera  121  for receiving a video signal, a microphone  122  for receiving an audio signal, and a user input unit  123  for receiving information from a user. 
     Voice data or image data collected by the input unit  120  may be analyzed and processed as a control command of the user. 
     The input unit  120  may receive video information (or signal), audio information (or signal), data or user input information. For reception of video information, the terminal  100  may include one or a plurality of cameras  121 . 
     The camera  121  may process image frames of still images or moving images obtained by image sensors in a video call more or an image capture mode. The processed image frames can be displayed on the display  151  or stored in memory  170 . 
     The microphone  122  processes an external acoustic signal into electrical audio data. The processed audio data may be variously used according to function (application program) executed in the terminal  100 . Meanwhile, the microphone  122  may include various noise removal algorithms to remove noise generated in the process of receiving the external acoustic signal. 
     The user input unit  123  receives information from a user. When information is received through the user input unit  123 , 
     The processor  180  may control operation of the terminal  100  in correspondence with the input information. 
     The user input unit  123  may include one or more of a mechanical input element (for example, a mechanical key, a button located on a front and/or rear surface or a side surface of the terminal  100 , a dome switch, a jog wheel, a jog switch, and the like) or a touch input element. As one example, the touch input element may be a virtual key, a soft key or a visual key, which is displayed on a touchscreen through software processing, or a touch key located at a location other than the touchscreen. 
     The learning processor  130  may be configured to receive, classify, store and output information to be used for data mining, data analysis, intelligent decision, mechanical learning algorithms and techniques. 
     The learning processor  130  may include one or more memory units configured to store data received, detected, sensed, generated or output in a predetermined manner or another manner by the terminal or received, detected, sensed, generated or output in a predetermined manner or another manner by another component, device, terminal or device for communicating with the terminal. 
     The learning processor  130  may include a memory integrated with or implemented in the terminal. In some embodiment, the learning processor  130  may be implemented using the memory  170 . 
     Selectively or additionally, the learning processor  130  may be implemented using a memory related to the terminal, such as an external memory directly coupled to the terminal or a memory maintained in a server communicating with the terminal. 
     In another embodiment, the learning processor  130  may be implemented using a memory maintained in a cloud computing environment or another remote memory accessible by the terminal through the same communication scheme as a network. 
     The learning processor  130  may be configured to store data in one or more databases in order to identify, index, categorize, manipulate, store, retrieve and output data to be used for supervised or unsupervised learning, data mining, predictive analysis or other machines. 
     Information stored in the learning processor  130  may be used by one or more other controllers of the terminal or the processor  180  using any one of different types of data analysis algorithms and machine learning algorithms. 
     Examples of such algorithms include k-nearest neighbor systems, fuzzy logic (e.g., possibility theory), neural networks, Boltzmann machines, vector quantization, pulse neural networks, support vector machines, maximum margin classifiers, hill climbing, inductive logic system Bayesian networks, Petri Nets (e.g., finite state machines, Mealy machines or Moore finite state machines), classifier trees (e.g., perceptron trees, support vector trees, Marcov trees, decision tree forests, random forests), betting models and systems, artificial fusion, sensor fusion, image fusion, reinforcement learning, augmented reality, pattern recognition, and automated planning. 
     The processor  180  may make a decision using data analysis and machine learning algorithms and determine or predict at least one executable operation of the terminal based on the generated information. To this end, the processor  180  may request, retrieve, receive or use the data of the processor  130  and control the terminal to execute preferable operation or predicted operation of at least one executable operation. 
     The processor  180  may perform various functions for implementing intelligent emulation (that is, a knowledge based system, an inference system and a knowledge acquisition system). This is applicable to various types of systems (e.g., a fussy logic system) including an adaptive system, a machine learning system, an artificial neural system, etc. 
     The processor  180  may include a sub module for enabling operation involving speech and natural language speech processing, such as an I/O processing module, an environmental condition module, speech-to-text (STT) processing module, a natural language processing module, a workflow processing module and a service processing module. 
     Each of such sub modules may have an access to one or more systems or data and models at the terminal or a subset or superset thereof. In addition, each of the sub modules may provide various functions including vocabulary index, user data, a workflow model, a service model and an automatic speech recognition (ASR) system. 
     In another embodiment, the other aspects of the processor  180  or the terminal may be implemented through the above-described sub modules, systems or data and models. 
     In some embodiments, based on the data of the learning processor  130 , the processor  180  may be configured to detect and sense requirements based on the context condition or user&#39;s intention expressed in user input or natural language input. 
     The processor  180  may actively derive and acquire information necessary to fully determine the requirements based on the context condition or user&#39;s intention. For example, the processor  180  may actively derive information necessary to determine the requirements, by analyzing historical data including historical input and output, pattern matching, unambiguous words, and input intention, etc. 
     The processor  180  may determine a task flow for executing a function for responding to the requirements based on the context condition or the user&#39;s intention. 
     The processor  180  may be configured to collect, sense, extract, detect and/or receive signals or data used for data analysis and machine learning operations through one or more sensing components at the terminal, in order to collect information for processing and storage from the learning processor  130 . 
     Information collection may include sensing information through a sensor, extracting information stored in the memory  170 , or receiving information from another terminal, an entity or an external storage device through a communication unit. 
     The processor  180  may collect and store usage history information from the terminal. 
     The processor  180  may determine the best match for executing a specific function using the stored usage history information and predictive modeling. 
     The processor  180  may receive or sense surrounding environment information or other information through the sensing unit  140 . 
     The processor  180  may receive broadcast signals and/or broadcast related information, wireless signals or wireless data through the wireless communication unit  110 . 
     The processor  180  may receive image information (or signals corresponding thereto), audio signal (or signals corresponding thereto), data or user input information from the input unit  120 . 
     The processor  180  may collect information in real time, process or classify the information (e.g., a knowledge graph, a command policy, a personalization database, a dialog engine, etc.), and store the processed information in the memory  170  or the learning processor  130 . 
     When the operation of the terminal is determined based on data analysis and machine learning algorithms and techniques, the processor  180  may control the components of the terminal in order to execute the determined operation. The processor  180  may control the terminal according to a control command and perform the determined operation. 
     When the specific operation is performed, the processor  180  may analyze historical information indicating execution of the specific operation through data analysis and machine learning algorithms and techniques and update previously learned information based on the analyzed information. 
     Accordingly, the processor  180  may improve accuracy of future performance of data analysis and machine learning algorithms and techniques based on the updated information, along with the learning processor  130 . 
     The sensing unit  140  may include one or more sensors configured to sense internal information of the mobile terminal, the surrounding environment of the mobile terminal, user information, and the like. 
     For example, the sensing unit  140  may include at least one of a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a G-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, an infrared (IR) sensor, a finger scan sensor, an ultrasonic sensor, an optical sensor (for example, a camera  121 ), a microphone  122 , a battery gauge, an environment sensor (for example, a barometer, a hygrometer, a thermometer, a radiation detection sensor, a thermal sensor, and a gas sensor), and a chemical sensor (for example, an electronic nose, a health care sensor, a biometric sensor, and the like). The mobile terminal disclosed in this specification may be configured to combine and utilize information obtained from at least two sensors of such sensors. 
     The output unit  150  is typically configured to output various types of information, such as audio, video, tactile output, and the like. The output unit  150  may include a display  151 , an audio output module  152 , a haptic module  153 , and a light output unit  154 . 
     The display  151  is generally configured to display (output) information processed in the terminal  100 . For example, the display  151  may display execution screen information of an application program executed by the terminal  100  or user interface (UI) and graphical user interface (GUI) information according to the executed screen information. 
     The display  151  may have an inter-layered structure or an integrated structure with a touch sensor in order to realize a touchscreen. The touchscreen may provide an output interface between the terminal  100  and a user, as well as function as the user input unit  123  which provides an input interface between the terminal  100  and the user. 
     The audio output module  152  is generally configured to output audio data received from the wireless communication unit  110  or stored in the memory  170  in a call signal reception mode, a call mode, a record mode, a speech recognition mode, a broadcast reception mode, and the like. 
     The audio output module  152  may also include a receiver, a speaker, a buzzer, or the like. 
     A haptic module  153  can be configured to generate various tactile effects that a user feels. A typical example of a tactile effect generated by the haptic module  153  is vibration. 
     A light output unit  154  may output a signal for indicating event generation using light of a light source of the terminal  100 . Examples of events generated in the terminal  100  may include message reception, call signal reception, a missed call, an alarm, a schedule notice, email reception, information reception through an application, and the like. 
     The interface  160  serves as an interface with external devices to be connected with the terminal  100 . The interface  160  may include wired or wireless headset ports, external power supply ports, wired or wireless data ports, memory card ports, ports for connecting a device having an identification module, audio input/output (I/O) ports, video I/O ports, earphone ports, or the like. The terminal  100  may perform appropriate control related to the connected external device in correspondence with connection of the external device to the interface  160 . 
     The identification module may be a chip that stores a variety of information for granting use authority of the terminal  100  and may include a user identity module (UIM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), and the like. In addition, the device having the identification module (also referred to herein as an “identifying device”) may take the form of a smart card. Accordingly, the identifying device can be connected with the terminal  100  via the interface  160 . 
     The memory  170  stores data supporting various functions of the terminal  100 . 
     The memory  170  may store a plurality of application programs or applications executed in the terminal  100 , data and commands for operation of the terminal  100 , and data for operation of the learning processor  130  (e.g., at least one piece of algorithm information for machine learning). 
     The processor  180  generally controls overall operation of the terminal  100 , in addition to operation related to the application program. The processor  180  may process signals, data, information, etc. input or output through the above-described components or execute the application program stored in the memory  170 , thereby processing or providing appropriate information or functions to the user. 
     In addition, the processor  180  may control at least some of the components described with reference to  FIG. 1  in order to execute the application program stored in the memory  170 . Further, the processor  180  may operate a combination of at least two of the components included in the terminal  100 , in order to execute the application program. 
     The power supply  190  receives external power or internal power and supplies the appropriate power required to operate respective components included in the terminal  100 , under control of the controller  180 . The power supply  190  may include a battery, and the battery may be a built-in or rechargeable battery. 
     Meanwhile, as described above, the processor  180  controls operation related to the application program and overall operation of the terminal  100 . For example, the processor  180  may execute or release a lock function for limiting input of a control command of the user to applications when the state of the mobile terminal satisfies a set condition. 
       FIG. 2  is a diagram illustrating a speech system according to an embodiment of the present disclosure. 
     Referring to  FIG. 2 , the speech system  1  includes an terminal  100 , a speech-to-text (STT) server  10 , a natural language processing (NLP) server  20  and a speech synthesis server  30 . 
     The terminal  100  may transmit speech data to the STT server  10 . 
     The STT server  10  may convert the speech data received from the terminal  100  into text data. 
     The STT server  10  may increase accuracy of speech-text conversion using a language model. 
     The language model may mean a model capable of calculating a probability of a sentence or a probability of outputting a next word is output when previous words are given. 
     For example, the language model may include probabilistic language models such as a unigram model, a bigram model, an N-gram model, etc. 
     The unigram model refers to a model that assumes that use of all words is completely independent of each other and calculates the probability of a word string by a product of the probabilities of words. 
     The bigram model refers to a model that assumes that use of words depends on only one previous word. 
     The N-gram model refers to a model that assumes that use of words depends on (n−1) previous words. 
     That is, the STT server  10  may determine when the speech data is appropriately converted into the text data using the language model, thereby increasing accuracy of conversion into the text data. 
     The NLP server  20  may receive the text data from the STT server  10 . The NLP server  20  may analyze the intention of the text data based on the received text data. 
     The NLP server  20  may transmit intention analysis information indicating the result of performing intention analysis to the terminal  100 . 
     The NLP server  20  may sequentially perform a morpheme analysis step, a syntax analysis step, a speech-act analysis step, a dialog processing step with respect to text data, thereby generating intention analysis information. 
     The morpheme analysis step refers to a step of classifying the text data corresponding to the speech uttered by the user into morphemes as a smallest unit having a meaning and determining the part of speech of each of the classified morphemes. 
     The syntax analysis step refers to a step of classifying the text data into a noun phrase, a verb phrase, an adjective phrase, etc. using the result of the morpheme analysis step and determines a relation between the classified phrases. 
     Through the syntax analysis step, the subject, object and modifier of the speech uttered by the user may be determined. 
     The speech-act analysis step refers to a step of analyzing the intention of the speech uttered by the user using the result of the syntax analysis step. Specifically, the speech-act step refers to a step of determining the intention of a sentence such as whether the user asks a question, makes a request, or expresses simple emotion. 
     The dialog processing step refers to a step of determining whether to answer the user&#39;s utterance, respond to the user&#39;s utterance or question about more information. 
     The NLP server  20  may generate intention analysis information including at least one of the answer to, a response to, or a question about more information on the intention of the user&#39;s utterance, after the dialog processing step. 
     Meanwhile, the NLP server  20  may receive the text data from the terminal  100 . For example, when the terminal  100  supports the speech-to-text conversion function, the terminal  100  may convert the speech data into the text data and transmit the converted text data to the NLP server  20 . 
     The speech synthesis server  30  may synthesize prestored speech data to generate a synthesized speech. 
     The speech synthesis server  30  may record the speech of the user selected as a model and divide the recorded speech into syllables or words. The speech synthesis server  30  may store the divided speech in an internal or external database in syllable or word units. 
     The speech synthesis server  30  may retrieve syllables or words corresponding to the given text data from the database and synthesize the retrieved syllables or words, thereby generating the synthesized speech. 
     The speech synthesis server  30  may store a plurality of speech language groups respectively corresponding to a plurality of languages. 
     For example, the speech synthesis server  30  may include a first speech language group recorded in Korean and a second speech language group recorded in English. 
     The speech synthesis server  30  may translate text data of a first language into text of a second language and generate a synthesized speech corresponding to the translated text of the second language using the second speech language group. 
     The speech synthesis server  30  may transmit the synthesized speech to the terminal  100 . 
     The speech synthesis server  30  may receive the intention analysis information from the NLP server  20 . 
     The speech synthesis server  30  may generate the synthesized speech including the intention of the user based on the intention analysis information. 
     In one embodiment, the STT server  10 , the NLP server  20  and the speech synthesis server  30  may be implemented as one server. 
     The respective functions of the STT server  10 , the NLP server  20  and the speech synthesis server  30  may also be performed in the terminal  100 . To this end, the terminal  100  may include a plurality of processors. 
       FIG. 3  is a diagram illustrating a process of extracting utterance features of a user from a speech signal according to an embodiment of the present disclosure. 
     The terminal  100  shown in  FIG. 1  may further include an audio processor  181 . 
     The audio processor  181  may be implemented as a chip separated from the processor  180  or a chip included in the processor  180 . 
     The audio processor  181  may remove noise from the speech signal. 
     The audio processor  181  may convert the speech signal into text data. To this end, the audio processor  181  may include an STT engine. 
     The audio processor  181  may recognize a wake-up word for activating speech recognition of the terminal  100 . The audio processor  181  may convert the wake-up word received through the microphone  121  into text data and determine that the wake-up word is recognized when the converted text data corresponds to the prestored wake-up word. 
     The audio processor  181  may convert the speech signal, from which noise is removed, into a power spectrum. 
     The power spectrum may be a parameter indicating a frequency component included in the waveform of the speech signal varying with time, and a magnitude thereof. 
     The power spectrum shows a distribution of an amplitude squared value according to the frequency of the waveform of the speech signal. 
     This will be described with reference to  FIG. 4 . 
       FIG. 4  is a diagram illustrating an example of converting a speech signal into a power spectrum according to an embodiment of the present disclosure. 
     Referring to  FIG. 4 , the speech signal  410  is shown. The speech signal  410  may be received through the microphone  121  or prestored in the memory  170 . 
     The x-axis of the speech signal  410  denotes a time and the y-axis denotes an amplitude. 
     The audio processor  181  may convert the speech signal  410 , the x-axis of which is a time axis, into a power spectrum  430 , the x-axis of which is a frequency axis. 
     The audio processor  181  may convert the speech signal  410  into the power spectrum  430  using Fast Fourier transform (FFT). 
     The x-axis of the power spectrum  430  denotes a frequency and the y-axis of the power spectrum  430  denotes a squared value of an amplitude. 
       FIG. 3  will be described again. 
     The processor  180  may determine utterance features of a user using at least one of the power spectrum  430  or the text data received from the audio processor  181 . 
     The utterance features of the user may include the gender of the user, the pitch of the user, the tone of the user, the topic uttered by the user, the utterance speed of the user, the volume of the user&#39;s voice, etc. 
     The processor  180  may acquire the frequency of the speech signal  410  and the amplitude corresponding to the frequency using the power spectrum  430 . 
     The processor  180  may determine the gender of the user who utters a speech, using the frequency band of the power spectrum  430 . 
     For example, the processor  180  may determine the gender of the user as a male when the frequency band of the power spectrum  430  is within a predetermined first frequency band range. 
     The processor  180  may determine the gender of the user as a female when the frequency band of the power spectrum  430  is within a predetermined second frequency band range. Here, the second frequency band range may be larger than the first frequency band range. 
     The processor  180  may determine the pitch of the speech using the frequency band of the power spectrum  430 . 
     For example, the processor  180  may determine the pitch of the speech according to the amplitude within a specific frequency band range. 
     The processor  180  may determine the tone of the user using the frequency band of the power spectrum  430 . For example, the processor  180  may determine a frequency band having a certain amplitude or more among the frequency bands of the power spectrum  430  as a main register of the user and determines the determined main register as the tone of the user. 
     The processor  180  may determine the utterance speed of the user through the number of syllables uttered per unit time from the converted text data. 
     The processor  180  may determine the topic uttered by the user using a Bag-Of-Word Model scheme with respect to the converted text data. 
     The Bag-Of-Word Model scheme refers to a scheme for extracting mainly used words based on the frequency of words in a sentence. Specifically, the Bag-Of-Word Model scheme refers to a scheme for extracting unique words from a sentence, expressing the frequency of the extracted words by a vector and determining the uttered topic as a feature. 
     For example, when words &lt;running&gt;, &lt;physical strength&gt;, etc. frequently appears in the text data, the processor  180  may classify the topic uttered by the user into an exercise. 
     The processor  180  may determine the topic uttered by the user from the text data using a known text categorization scheme. The processor  180  may extract keywords from the text data and determine the topic uttered by the user. 
     The processor  180  may determine the volume of user&#39;s voice in consideration of the amplitude information in an entire frequency band. 
     For example, the processor  180  may determine the volume of user&#39;s voice based on an average or weighted average of amplitudes in each frequency band of the power spectrum. 
     The functions of the audio processor  181  and the processor  180  described with reference to  FIGS. 3 and 4  may be performed in any one of the NLP server  20  or the speech synthesis server  30 . 
     For example, the NLP server  20  may extract the power spectrum using the speech signal and determine the utterance features of the user using the extracted power spectrum. 
       FIG. 5  is a block diagram illustrating the configuration of a speech synthesis server according to an embodiment of the present disclosure. 
     The speech synthesis server  30  is a device or server disposed outside the terminal  100  and may perform the same function as the learning processor  130  of the terminal  100 . 
     That is, the speech synthesis server  30  may be configured to receive, classify, store and output information to be used for data mining, data analysis, intelligent decision, mechanical learning algorithms. Here, the machine learning algorithms may include a deep learning algorithm. 
     The speech synthesis server  30  may communicate with at least one terminal  100  and derive a result by analyzing or learning data instead of or in aid of the terminal  100 . Aiding another device may mean distribution of computing power through distribution processing. 
     The speech synthesis server  30  is a variety of devices for learning an artificial neural network, may generally mean a server, and may be referred to as a learning device or a learning server. 
     In particular, the speech synthesis server  30  may be implemented not only as a single server but also as a plurality of server sets, a cloud server or a combination thereof. 
     That is, a plurality of speech synthesis servers  30  may configure a learning device set (or a cloud server) and at least one speech synthesis server  30  included in the learning device set may derive a result by analyzing or learning data through distribution processing. 
     The speech synthesis server  30  may transmit a model learned by machine learning or deep learning to the terminal  100  periodically or according to a request. 
     Referring to  FIG. 5 , the speech synthesis server  30  may include a communication unit  210 , an input unit  220 , a memory  230 , a learning processor  240 , a power supply  250  and a processor  260 . 
     The communication unit  210  may correspond to a component including the wireless communication unit  110  and the interface  160  of  FIG. 1 . That is, data may be transmitted to and received from another device through wired/wireless communication or an interface. 
     The input unit  220  may correspond to the input unit  120  of  FIG. 1  and acquire data by receiving data through the communication unit  210 . 
     The input unit  220  may acquire input data for acquiring output using training data for model learning or a trained model. 
     The input unit  220  may acquire raw input data. In this case, the processor  260  may preprocess the acquired data to generate training data or preprocessed input data capable of being input to model learning. 
     At this time, preprocessing of the input data performed by the input unit  220  may mean extraction of input features from the input data. 
     The memory  230  may correspond to the memory  170  of  FIG. 1 . 
     The memory  230  may include a model storage unit  231  and a database  232 . 
     The model storage unit  231  stores a model (or an artificial neural network  231   a ) which is learned or being learned through the learning processor  240  and stores an updated model when the model is updated through learning. 
     At this time, the model storage unit  231  may classify and store the trained model into a plurality of versions according to a learning time point or learning progress, as necessary. 
     The artificial neural network  231   a  shown in  FIG. 5  is merely an example of the artificial neural network including a plurality of hidden layers and the artificial neural network of the present disclosure is not limited thereto. 
     The artificial neural network  231   a  may be implemented in hardware, software or a combination of hardware and software. When some or the whole of the artificial neural network  231   a  is implemented in software, one or more commands configuring the artificial neural network  231   a  may be stored in the memory  230 . 
     The database  232  stores the input data acquired by the input unit  220 , learning data (or training data) used for model learning, or a learning history of a model. 
     The input data stored in the database  232  may be not only data processed to suit model learning but also raw input data. 
     The learning processor  240  corresponds to the learning processor  130  of  FIG. 1 . 
     The learning processor  240  may train or learn the artificial neural network  231   a  using training data or a training set. 
     The learning processor  240  may immediately acquire data obtained by preprocessing the input data acquired by the processor  260  through the input unit  220  to learn the artificial neural network  231   a  or acquire the preprocessed input data stored in the database  232  to learn the artificial neural network  231   a.    
     Specifically, the learning processor  240  may determine the optimized model parameters of the artificial neural network  231   a , by repeatedly learning the artificial neural network  231   a  using the above-described various learning schemes. 
     In this specification, the artificial neural network having parameters determined through learning using training data may be referred to as a training model or a trained model. 
     At this time, the training model may infer a result value in a state of being installed in the speech synthesis server  30  of the artificial neural network and may be transmitted to and installed in another device such as the terminal  100  through the communication unit  210 . 
     In addition, when the training model is updated, the updated training model may be transmitted to and installed in another device such as the terminal  100  through the communication unit  210 . 
     The power supply  250  corresponds to the power supply  190  of  FIG. 1 . 
     A repeated description of components corresponding to each other will be omitted. 
       FIGS. 6 and 7  are views illustrating a class imbalance problem when a reading break is predicted through a conventional synthesized speech. 
       FIG. 6  is a view showing a result of performing reading with break through a synthesized speech at a synthesized speech engine with respect to one sentence  600 . 
     The synthesized speech engine may convert text into speech and output the speech. 
     The synthesized speech engine may be provided in the terminal  100  or the speech synthesis server  30 . 
     A space bar  601  indicates that reading break is 1, &lt;/&gt;  603  indicates that the reading break is 2 and &lt;//&gt;  605  indicates that the reading break is 3. 
     The reading break may indicate a time interval when text is read. That is, as the reading break increases, the time interval when text is read may increase. In contrast, as the reading break decreases, the time interval when text is read may decrease. 
       FIG. 7  shows a class table  700  indicating a result of analyzing the reading break with respect to the sentence  600  of  FIG. 6 . 
     The class table  700  may include a word phrase (WP) class, an accentual phrase (AP) class and an intonation phrase (IP) class. 
     The word phrase class indicates that reading break is 1 and may indicate a class that words are read without break. 
     The accentual phrase class indicates that reading break is 2 and may indicate that break between words is small. 
     The intonation phrase class indicates that reading break is 3 and may indicate that break between words is large. 
     In the sentence  600  of  FIG. 6 , the count of word phrase classes is 7, the count of accentual phrase classes is 19 and the count of intonation phase classes is 4. 
     A class with a smallest count is called a minor class and a class with a largest count is called a major class. 
     In  FIG. 7 , the intonation phrase class may be the minor class and the accentual phrase class may be the major class. 
     When class imbalance in which the count of intonation phrase classes is less than the count of the other classes occurs, in a machine learning process of reading with break through a synthesized speech, the intonation phrase class may be determined as being less important and reading break performance of the synthesized speech model may deteriorate. 
     Specifically, for reading break learning of the synthesized speech model, a training set including one sentence (training data) and labeling data for labeling words configuring the sentence with reading breaks is required. 
     When data with class imbalance is used as labeling data, performance of the synthesized speech model may deteriorate. 
     When performance of the synthesized speech model deteriorates, reading with break may become unnatural when the synthesized speech is output and thus users may feel uncomfortable when listening to the synthesized speech. 
     In order to solve such a problem, in the present disclosure, the counts of classes are adjusted in a balanced way, thereby improving reading break prediction performance. 
       FIG. 8  is a flowchart illustrating a method of operating a speech synthesis server according to an embodiment of the present disclosure. 
     The processor  260  of the speech synthesis server  30  acquires prior information of each of a plurality of words corresponding to the minor class (S 801 ). 
     Hereinafter, assume that the minor class is the intonation phrase class of  FIG. 7 . 
     A word belonging to (or being classified as) the intonation phrase class means that a word located before &lt;//&gt; indicating reading break of 3, such as &lt;government&#39;s&gt; shown in  FIG. 6 , belongs to the intonation phrase class. 
     In one embodiment, the prior information may include one or more of an intonation phrase (hereinafter referred to as IP) ratio of a word, an IP frequency number, a non-IP ratio, a non-IP frequency number, or a ratio of the non-IP frequency number to the IP frequency number. 
     The IP ratio may indicate a ratio in which a word is classified as the IP class, in the database  232 . Specifically, in 10000 sentences in the database  232 , when the number of times of classifying a first word into an IP class is 100, the IP ratio of the first word may be 1%(100/10000×100). 
     In the 10000 sentences, when the number of times of classifying a second word into the IP class is 200, the IP ratio of the second word may be 2%. 
     Of course, only some of the 10000 sentences may include the first word or the second word. 
     The IP frequency number may indicate the number of times of classifying a word into the IP class in the database  232 . In the above example, the IP frequency number of the first word may be 100 and the IP frequency number of the second word may be 200. 
     The non-IP ratio may indicate a ratio of a word classified as a class other than the IP class in the database  232 . 
     For example, in 10000 sentences of the database  232 , when the number of times of classifying the first word into a class other than the IP class is 500, the non-IP ratio of the first word may be 5%(500/10000×100). 
     The non-IP frequency number may indicate the number of times in which the word is not classified as the IP class in the database  232 . 
     For example, in 10000 sentences of the database, when the number of times in which the first word is not classified as the IP class is 300, the non-IP ratio of the first word may be 3%(300/10000×100). 
     The processor  260  of the speech synthesis server  30  performs data augmentation with respect to each data based on the acquired prior information (S 803 ). 
     In one embodiment, data augmentation may be a process of increasing a frequency number in which a word belongs to a specific class in order to increase a probability that the word belongs to the specific class. 
     Increasing the frequency number in which the word belongs to the specific class may indicate that the number of sentences including the word belonging to the specific class increases. 
     This may be interpreted as increasing a training set for learning of the synthesized speech model. 
     This will be described in detail below. 
     The processor  260  of the speech synthesis server  30  stores a result of performing data augmentation in the database  232  (S 805 ). 
     The processor  260  of the speech synthesis server  30  or the learning processor  240  performs machine learning for reading with break using the stored result of performing data augmentation (S 807 ). 
     Machine learning for reading with break may be a process of determining with which break the words configuring a sentence is read when the sentence is input. 
     That is, machine learning for reading with break may be learning for classifying one sentence into a word phrase class, an accentual phrase class and an intonation phrase class. 
     A synthesized speech model may be generated according to machine learning for reading with break. 
     The synthesized speech model may refer to a model for receiving one sentence as input data and outputting synthesized speech data in which words configuring one sentence are classified into three optimized reading break classes. 
     The processor  260  of the speech synthesis server  30  may transmit the generated synthesized speech model to the terminal  100  through the communication unit  210 . 
       FIG. 9  is a flowchart illustrating a process of performing data augmentation of a word based on prior information according to an embodiment of the present disclosure. 
     In particular,  FIG. 9  is a view illustrating steps S 803  and S 805  shown in  FIG. 8  in detail. 
     The processor  260  of the speech synthesis server  30  determines the oversampling rate of each word based on the prior information of the word (S 901 ) 
     In one embodiment, the processor  260  may determine the oversampling rate of the word based on the ratio of the non-IP frequency number to the IP frequency number of the word classified as the minor class. 
     The oversampling rate may indicate a rate at which the word belongs to the IP class in the database  232 . 
     The processor  260  may increase the oversampling rate as the ratio of the non-IP frequency number to the IP frequency number of the word increases. 
     The processor  260  may decrease the oversampling rate as the ratio of the non-IP frequency number to the IP frequency number of the word increases. 
     This will be described with reference to  FIG. 10 . 
       FIG. 10  is a view showing an IP frequency number and a non-IP frequency number of each word stored in a database according to an embodiment of the present disclosure. 
       FIG. 10  shows a result obtained by measuring reading break after uttering a specific word when a voice actor utters a large number of sentences, in order to generate a synthesized speech. 
     For example, assume that the frequency number in which a word &lt;but&gt; is classified as the IP class in the database  232  is 60 and the frequency number in which the word &lt;but&gt; is classified as the non-IP class instead of the IP class is 10. 
     Since the ratio of the non-IP frequency number to the IP frequency number is 1:6, the processor  260  may determine that the oversampling rate of the word &lt;but&gt; is 60% (6/1×0.1). 
     For example, the processor  260  may increase the existing frequency number, in which the word &lt;but&gt; is classified as the IP class, to  96  which is greater than 60 by 60%. 
     In another example, assume that the frequency number in which a word &lt;can&gt; is classified as the IP class in the database  232  is 30 and the frequency number in which the word &lt;can&gt; is classified as the non-IP class instead of the IP class is 120. 
     Since the ratio of the non-IP frequency number to the IP frequency number is 4:1, the processor  260  may determine that the oversampling rate of the word &lt;can&gt; is 2.5% (¼×0.1). 
     For example, the processor  260  may increase the existing frequency number, in which the word &lt;can&gt; is classified as the IP class, to 3.075 which is greater than 30 by 2.5%. 
     In another example, the processor  260  may increase the oversampling rate only when the IP frequency number of the word is greater than the non-IP frequency number of the word. 
     In contrast, the processor  260  may not perform oversampling of the word when the IP frequency number of the word is less than the non-IP frequency number of the word. That is, the processor  260  may fix the oversampling rate when the IP frequency number of the word is less than the non-IP frequency number of the word. 
       FIG. 11  is a view illustrating an oversampling rate determined according to a ratio of a non-IP frequency number to an IP frequency number. 
       FIG. 11  shows the oversampling rate determined according to the ratio of the non-IP frequency number to the IP frequency number of the word of  FIG. 10  in the 10000 sentences stored in the database  232 . 
     That is,  FIG. 11  shows the non-IP frequency number in which each word is not classified as the IP class, the IP frequency number in which each word is classified as the IP class, the relative ratio of non-IP frequency number to the IP frequency number, and the oversampling rate determined according to the relative ratio. 
     As can be seen from  FIG. 11 , as the relative ratio increases, the oversampling rate increases. As the relative ratio decreases, the oversampling rate decreases. 
     When the oversampling rate increases, a probability that the word is classified as the IP class may increase. 
     When the probability that the word is classified as the IP class increases, the class imbalance can be solved and the reading break performance of the synthesized speech model may increase. 
       FIG. 9  will be described again. 
     The processor  260  of the speech synthesis server  30  determines the number of times of oversampling of the word using the determined oversampling rate (S 903 ). 
     In one embodiment, the number of times of oversampling of the word may indicate the IP frequency number to be increased based on the determined oversampling rate of the word. 
     The IP frequency number to be increased may indicate the number of sentences including the word classified as the IP class. 
     That is, increasing the number of times of oversampling of the word may indicate that the number of sentences including the word classified as the IP class increases. 
     In one embodiment, the processor  260  may determine the number of times of oversampling of the word based on the oversampling rate determined in step S 901 . 
     In another embodiment, the processor  260  may determine the number of times of oversampling the word, based on the oversampling rate, the number of words classified as the major class in the database  232 , the number of words classified as the minor class, the number of times of labeling the word with the minor class, a probability that the word belongs to the minor class, and the number of times in which the word appears in the database  232 . 
     Specifically, the processor  260  may determine the number of times of oversampling as shown in Equation 1 below. 
     
       
         
           
             
               
                 
                   
                     word 
                     
                       i 
                       ⁢ 
                       
                         : 
                       
                       ⁢ 
                       over 
                     
                   
                   = 
                   
                     SamplingRate 
                     ⋆ 
                     
                       
                          
                         
                           Class 
                           Major 
                         
                          
                       
                       
                          
                         
                           Class 
                           minor 
                         
                          
                       
                     
                     ⋆ 
                     
                        
                       
                         
                           word 
                           i 
                         
                         = 
                         minor 
                       
                       ⁢ 
                       
                           
                       
                        
                     
                     ⋆ 
                     
                       P 
                       ( 
                       
                         word 
                         
                           i 
                           = 
                           minor 
                         
                       
                     
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     1 
                   
                   ] 
                 
               
             
           
         
       
     
     where, word i  may indicate a specific word present in the database  232 , 
     word i:over  may indicate the number of times of oversampling of word i , 
     Sampling Rate may be a constant determined in step S 901  and may have a value of 10% to 100%, but this is merely an example, 
     |Class Major | may indicate the number of words in the major class, 
     |Class Minor | may indicate the number of words in the minor class, and 
     P(word i=minor ) may indicate a probability that a specific word belongs to the minor class. 
     P(word i=minor ) may be expressed by Equation 2 below. 
     
       
         
           
             
               
                 
                   
                      
                     
                       
                         word 
                         i 
                       
                       = 
                       minor 
                     
                     ⁢ 
                     
                         
                     
                      
                   
                   
                      
                     
                       word 
                       i 
                     
                      
                   
                 
               
               
                 
                   [ 
                   
                     Equation 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     2 
                   
                   ] 
                 
               
             
           
         
       
     
     where, |Word i =minor| may indicate the number of times in which word i  is labeled with the minor class in the database  232 . 
     Labeling the word with the minor class may mean that words &lt;government&#39;s&gt;, &lt;year&gt;, &lt;rain&gt; and &lt;Monday&gt; which are used as criteria used to determine the count of the IP class which is the minor class are classed into the IP class, in  FIGS. 6 and 7 . 
     |word i | may indicate the number of times in which word i  appears in the database  232 . 
     That is, |word i | may indicate the number of times in which word i  appears in a plurality of sentences of the database  232 . 
     The processor  260  of the speech synthesis server  30  stores the determined number of times of oversampling in the database  232  (S 905 ). 
     The processor  260  may generate sentences including the words, the number of which correspond to the determined number of times of oversampling the word. 
     The processor  260  may label the word with the IP class and generate sentences including the word labeled with reading break of 3 such that the number of sentences corresponds to the number of times of oversampling. 
     The processor  260  may learn the synthesized speech model using the sentences including the word and the labeling data of labeling the word with reading break. 
       FIG. 12  is a ladder diagram illustrating a method of operating a system according to an embodiment of the present disclosure. 
     Referring to  FIG. 12 , the speech synthesis server  30  acquires sentences including the word classified as the IP class by the number of times of oversampling of the word (S 1201 ). 
     The speech synthesis server  30  may generate arbitrary sentences including the word. 
     The arbitrary sentences may be training data for learning of the synthesized speech model. 
     The speech synthesis server  30  learns the synthesized speech model using the acquired sentences (S 1203 ). 
     The word classified as the IP class may be labeled with reading break of 3. 
     The speech synthesis server  30  may learn the synthesized speech model using the arbitrary sentences (training data) and the labeling data of labeling the word in the arbitrary sentences with the reading break. 
     In one embodiment, the processor  260  of the speech synthesis server  30  may learn the synthesized speech model using a recurrent neural network (RNN). 
     The recurrent neural network is a kind of artificial neural network in which a hidden layer is connected to a directional edge to form a recurrent structure. 
     A process of learning the synthesized speech model using the recurrent neural network will be described with reference to  FIG. 13 . 
       FIG. 13  is a view illustrating a basic structure of a recurrent neural network. 
     Xt denotes input data, Ht denotes current hidden data, H(t−1) denotes previous hidden data, and Yt denotes output data. 
     The input data, the hidden data and the output data may be expressed by feature vectors. 
     Parameters learned by the RNN include a first parameter W 1  for converting the previous hidden data into the current hidden data, a second parameter W 2  for converting the input data into the hidden data and a third parameter W 3  for converting the current hidden data into the output data. 
     The first, second and third parameters W 1 , W 2  and W 3  may be expressed by a matrix. 
     According to the present disclosure, the input data may be a feature vector indicating a word, and the output data may be a feature vector indicating a first probability that an input word belongs to a WP class, a second probability that the input word belongs to an AP class and a third probability that the input word belongs to an IP class. 
     The previous hidden data may be hidden data of a previously input word, and the current hidden data may be data generated using the hidden data of the previously input word and a feature vector of a currently input word. 
       FIG. 14  is a view illustrating a process of classifying words configuring a sentence into classes using a synthesized speech model according to an embodiment of the present disclosure. 
     Referring to  FIG. 14 , a plurality of words  1310  configuring one sentence is sequentially input to the synthesized speech model  1330 . 
     The terminal  100  or the speech synthesis server  30  may output a first probability ( 1351 ) that each of the sequentially input words  1310  is classified as the WP class, a second probability ( 1353 ) that each of the sequentially input words  1310  is classified as the AP class and a third probability ( 1355 ) that each of the sequentially input words  1310  is classified as the IP class, using the synthesized speech model  1330 . 
     The terminal  100  or the speech synthesis server  30  may classify a probability having the largest value among the first to third probabilities into the class of the input word. 
       FIG. 12  will be described again. 
     The speech synthesis server  30  transmits the learned synthesized speech model to the terminal  100  (S 1205 ). 
     The terminal  100  outputs the synthesized speech according to the request of the user through the audio output unit  152  using the synthesized speech model received from the speech synthesis server  30  (S 1207 ). 
     The request of the user may be the speech command of the user, such as &lt;Read news article&gt;. 
     The terminal  100  may receive the speech command of the user and grasp the intention of the received speech command. 
     The terminal  100  may output, through the audio output unit  152 , the synthesized speech of the text corresponding to the news article suiting the grasped intention using the synthesized speech model. 
       FIG. 15  is a flowchart illustrating a method of operating a speech synthesis server using artificial intelligence according to another embodiment of the present disclosure. 
     Referring to  FIG. 15 , the processor  260  of the speech synthesis server  30  acquires prior information of each of a plurality of words corresponding to a minor class (S 1501 ). 
     Hereinafter, assume that the minor class is the intonation phrase class of  FIG. 7 . 
     A word belonging to (or being classified as) the intonation phrase class means that a word located before &lt;//&gt; indicating reading break of 3, such as &lt;government&#39;s&gt; shown in  FIG. 6 , belongs to the intonation phrase class. 
     In one embodiment, the prior information may include one or more of an intonation phrase (hereinafter referred to as IP) ratio of a word, an IP frequency number, a non-IP ratio in which the word does not belong to the intonation phrase class (hereinafter referred to as a non-IP ratio), a non-IP frequency number, or a ratio of the non-IP frequency number to the IP frequency number. 
     The IP ratio may indicate a ratio in which a word is classified as the IP class, in the database  232 . Specifically, in 10000 sentences in the database  232 , when the number of times of classifying a first word into an IP class is 100, the IP ratio of the first word may be 1%(100/10000×100). 
       FIG. 16  is a diagram illustrating prior information of each word stored in a database according to an embodiment of the present disclosure. 
       FIG. 16  shows an IP table  1600  showing an IP ratio in which each word is classified as the IP class and a non-IP ratio in which each word is not classified as the IP class. 
     The IP table  1600  shows the IP ratio which is the ratio in which each word is classified as the IP class in the sentences stored in the database  232  and the non-IP ratio obtained by subtracting the IP ratio from 1, with respect to each of the plurality of words. 
     The non-IP ratio may be a sum of the ratio in which the word is classified as the WP class and the ratio in which the word is classified as the AP class. 
     For example, the IP ratio of the word “Because” is 0.84 and the non-IP ratio thereof is 0.16 (1−0.84). 
     In addition, the IP ratio of the word “This” is 0.15 and the non-IP ratio thereof is 0.85. 
       FIG. 15  will be described again. 
     The processor  260  of the speech synthesis server  30  acquires a first class classification probability set indicating the per-class classification probability of each word used as labeling data of the synthesized speech model (S 1503 ). 
     The synthesized speech model may be an artificial neural network based model learned by a machine learning algorithm or a deep learning algorithm. 
     For example, the synthesized speech model may be a recurrent neural network based model shown in  FIG. 13 . 
     The first class classification probability set may include a probability that the word is classified as the WP class, a probability that the word is classified as the AP class and a probability that the word is classified as the IP class, within one sentence. 
     Specifically, a first probability that the word is classified as the WP class may indicate a probability of reading a word with break of 1 within one sentence. 
     Similarly, a second probability that the word is classified as the AP class may indicate a probability of reading a word with break of 2 within one sentence, and a third probability that the word is classified as the IP class may indicate a probability of reading a word with break of 3 within one sentence. 
     The processor  260  may infer a probability that each word is classified as the WP class, a probability that each word is classified as the AP class and a probability that each word is classified as the IP class, using the synthesized speech model. 
     The synthesized speech model may be a model composed of an artificial neural network learned to infer a first class classification probability set indicating the output feature point using a sentence including a plurality of words as input data. 
     For example, the synthesized speech model may be learned through supervised learning. Specifically, learning data used for the synthesized speech model may be labeled with the probability that the word is classified as the WP class, the probability that the word is classified as the AP class and the probability that the word is classified as the IP class. 
     The synthesized speech model may be learned using the labeled learning data. 
     The synthesized speech model may be learned with the goal of accurately inferring the probability that the word included in text data is classified as the labeled WP class, the probability that the word included in text data is classified as the AP class and the probability that the word included in text data is classified as the IP class, from text data for learning. 
     The weighted cost function of the synthesized speech model may be expressed by a squared mean of a difference between each class probability corresponding to each learning data and each class probability inferred from learning data. 
     Through learning of the synthesized speech model, model parameters included in the artificial neural network may be determined to minimize the value of the weighted cost function. 
       FIG. 17  is a diagram illustrating an example of a synthesized speech model according to an embodiment of the present disclosure. 
     Referring to  FIG. 17 , a synthesized speech model  1700  composed of an artificial neural network is shown. 
     Text data which is learning data and a per-class probability which is labeling data may be input to the synthesized speech model  1700  as one training set. 
     As a result of inference, the per-class probability may be output. 
     The synthesized speech model  1700  may be learned to minimize the cost function corresponding to a difference between the output per-class probability and the labeled per-class probability. 
     The synthesized speech model  1700  may be learned, such that a result of inferring each per-class classification probability is output as a target feature vector and a cost function corresponding to a difference between the output per-class classification probability set and the labeled per-class classification probability set is minimized, when an input feature vector is extracted from the text data including words and input to the synthesized speech model. 
     The output result of the synthesized speech model  1700  may be composed of an output layer having a plurality of output nodes indicating the per-class classification probability of the word. 
     Each of the plurality of output nodes may indicate the probability that the word is classified as the WP class, the probability that the word is classified as the AP class and the probability that the word is classified as the IP class. 
     Each of the plurality of output nodes may be expressed by a target feature vector such as (0.3, 0.3, 0.4). Each of the values of the elements configuring the target feature vector may have a value of 0 to 1. 
       FIG. 15  will be described again. 
     The processor  260  of the speech synthesis server  30  acquires a second class classification probability set indicating a corrected per-class classification probability of each word based on the prior information and first class classification probability set of each word (S 1505 ). 
     The processor  260  may correct the first class classification probability set to the second class classification probability set, based on the IP ratio and the first class classification probability set of the word. 
     The processor  260  may adjust the class classification probability set to be used as labeling data in the synthesized speech model  1700 , based on the IP ratio and the first class classification probability set of the word. 
     This is because labeling data is corrected by applying the IP ratio in which a specific word belongs to the IP class and a non-IP ratio in which the specific word belongs to the non-IP ratio in the plurality of sentences stored in the database. 
     When labeling data, to which the IP ratio and the non-IP ratio are applied, is used to learn the synthesized speech model, since the distribution of the word classified as the IP class is applied, it is possible to improve class classification accuracy. 
     Therefore, it is possible to solve a class imbalance problem. 
     A process of correcting the first class classification probability set in consideration of the IP ratio and the non-IP ratio of the word will be described with reference to the following drawings. 
       FIGS. 18 to 21  are diagrams illustrating an example of correcting a class classification probability set in consideration of an IP ratio and a non-IP ratio of a word according to an embodiment of the present disclosure. 
     First,  FIGS. 18 and 19  will be described based on the word &lt;Because&gt;. 
     In addition, for the IP ratio of the word &lt;Because&gt;, refer to the IP table  1600  of  FIG. 16 . 
     Referring to  FIG. 18 , the first class classification probability set  1800  which is labeling data corresponding to the word &lt;Because&gt; is shown. 
     As the labeling data of &lt;Because&gt;, a probability that the word is classified as the WP class is 0.3, a probability that the word is classified as the AP class is 0.6, and a probability that the word is classified as the IP class is 0.1. 
     In  FIG. 18 , each of the probability that the word is classified as the WP class, the probability that the word is classified as the AP class and the probability that the word is classified as the IP class may be a mathematical probability. 
     The processor  260  may acquire the second class classification probability set  1900  shown in  FIG. 19 , based on the IP ratio, non-IP ratio and first class classification probability set  1800  of &lt;Because&gt;. 
     The corrected WP class classification probability may be expressed as shown in Equation 3 below.
 
Corrected WP class classification probability=(WP class classification probability before correction)*(Non-IP ratio)*(probability that the word is classified as the WP class/probability that the word is not classified as the IP class)  [Equation 3]
 
     Accordingly, a value of 0.3*0.16*(0.3/0.9)=0.016 may be obtained as the corrected WP class classification probability by Equation 3. 
     The corrected AP class classification probability may be expressed by Equation 4 below.
 
Corrected AP class classification probability=(AP class classification probability before correction)*(Non-IP ratio)*(probability that the word is classified as the AP class/probability that the word is not classified as the IP class)  [Equation 4]
 
     Accordingly, a value of 0.6*0.16*(0.6/0.9)=0.064 may be obtained as the corrected AP class classification probability by Equation 4. 
     The corrected IP class classification probability may be expressed by Equation 5 below.
 
Corrected IP class classification probability=(IP class classification probability before correction)*(IP ratio)  [Equation 5]
 
     Accordingly, a value of 0.1*0.84=0.084 may be obtained as the corrected IP class classification probability by Equation 5. 
     Meanwhile, the processor  260  may change each of the corrected class classification probabilities to a normalized value based on  1 . 
     The probability that the word &lt;Because&gt; is classified as the corrected WP class is 0.97, the probability that the word &lt;Because&gt; is classified as the corrected AP class is 0.390, and the probability that the word &lt;Because&gt; is classified as the corrected IP class is 0.513. 
     That is, the first class classification probability set (0.3, 0.6, 0.1) may be changed to the second class classification probability set (0.097, 0.390, 0.513). 
     That is, the WP class classification probability decreased from 0.3 to 0.097, the AP class classification probability decreased from 0.6 to 0.390, and the IP class classification probability decreased from 0.1 to 0.513. 
     The processor  260  may increase the probability that the word is classified as the IP class as the IP ratio of the word increases and decreases the probability that the word is classified as the IP class as the IP ratio decreases. 
     The probability that the word is classified as the IP class may increase by correcting the labeling data in consideration of the IP ratio of the word stored in the database  232 . 
     Therefore, when the synthesized speech is generated, a possibility that the word belongs to the IP class increases, thereby solving class imbalance. 
     Next,  FIGS. 20 and 21  will be described. 
       FIGS. 20 and 21  will be described based on the word &lt;This&gt;. 
     In addition, for the IP ratio of the word &lt;This&gt;, refer to the IP table  1600  of  FIG. 16 . 
     Referring to  FIG. 20 , the first class classification probability set  2000  which is labeling data corresponding to the word &lt;This&gt; is shown. 
     As the labeling data of &lt;This&gt;, a probability that the word is classified as the WP class is 0.3, a probability that the word is classified as the AP class is 0.3, and a probability that the word is classified as the IP class is 0.4. 
     The processor  260  may acquire the second class classification probability set  2100  shown in  FIG. 21 , based on the IP ratio, non-IP ratio and first class classification probability set  2000  of &lt;This&gt;. 
     The corrected WP class classification probability may be expressed as shown in Equation 3 above. 
     A value of 0.3*0.85*(0.3/0.6)=0.127 may be obtained as the corrected WP class classification probability by Equation 3. 
     The corrected AP class classification probability may be expressed by Equation 4 above. 
     A value of 0.3*0.85*(0.3/0.6)=0.127 may be obtained as the corrected AP class classification probability by Equation 4. 
     The corrected IP class classification probability may be expressed by Equation 5 above. 
     A value of 0.4*0.15=0.06 may be obtained as the corrected IP class classification probability by Equation 5. 
     Meanwhile, the processor  260  may change each of the corrected class classification probabilities to a normalized value based on  1 . 
     The probability that the word &lt;This&gt; is classified as the corrected WP class is 0.4, the probability that the word &lt;This&gt; is classified as the corrected AP class is 0.4, and the probability that the word &lt;This&gt; is classified as the corrected IP class is 0.2. 
     That is, the first class classification probability set (0.3, 0.3, 0.4) may be changed to the second class classification probability set (0.4, 0.4, 0.2). 
     That is, the WP class classification probability increased from 0.3 to 0.4, the AP class classification probability increased from 0.3 to 0.4, and the IP class classification probability decreased from 0.4 to 0.2. 
     The probability that the word is classified as the IP class may decrease by correcting the labeling data in consideration of the IP ratio of the word stored in the database  232 . 
     Therefore, when the synthesized speech is generated, a possibility that the word belongs to the IP class decreases, thereby solving class imbalance. 
     That is, according to the embodiment of the present disclosure, it is possible to improve reading break prediction performance of the synthesized speech model. 
       FIG. 15  will be described again. 
     The processor  260  of the speech synthesis server  30  or the learning processor  240  learns the synthesized speech model using the acquired second class classification probability set (S 1507 ). 
     The processor  260  may learn the synthesized speech model using the acquired second class classification probability set as new labeling data. 
     A process of learning the synthesized speech model using the second class classification probability set will be described with reference to  FIG. 22 . 
       FIG. 22  is a diagram illustrating a process of learning a synthesized speech model using a newly acquired second class classification probability set based on an IP ratio according to an embodiment of the present disclosure. 
     Referring to  FIG. 22 , a training set including text data including words and class classification probability set of each word may be input to the synthesized speech model  1700 . 
     The synthesized speech model  1700  may output a first class classification probability set  2210  including a probability that each word is classified as the WP class, a probability that each word is classified as the AP class and a probability that each word is classified as the IP class, with respect to the training set. 
     The processor  260  may acquire a second class classification probability set  2230  based on the output first class classification probability set  2210  and the IP ratio and non-IP ratio of each word. 
     That is, the processor  260  may perform probability correction operation to change the first class classification probability set  2210  to the second class classification probability set  2230 , as shown in  FIGS. 18 to 21 . 
     The processor  260  may use the newly acquired second class classification probability set  2230  as new labeling data of the synthesized speech model  1700 . 
     Meanwhile, the synthesized speech models described with reference to  FIGS. 15 to 22  may be transmitted to the terminal. 
     The terminal may obtain the per-class classification probability of each word included in text data corresponding to a sentence using the synthesized speech mode. 
       FIG. 23  is a ladder diagram illustrating a method of operating another system of the present disclosure. 
     In particular,  FIG. 23  relates to a method of outputting a synthesized speech in consideration of a reading-with-break style of a user. 
     The processor  260  of the speech synthesizer  30  acquires an IP ratio table of each of a plurality of voice actors based on the plurality of voice actors stored in the database  232  (S 2301 ). 
     The database  232  stores speech data uttered by each of the plurality of voice actors. 
     The processor  260  may acquire the IP ratio table based on the speech data uttered by each voice actor. The IP ratio table may include an IP ratio in which the plurality of words is classified as an IP class and a non-IP ratio which the plurality of words is not classified as an IP class. 
     This will be described with reference to  FIG. 24 . 
       FIG. 24  is a view illustrating a plurality of IP tables respectively corresponding to a plurality of voice actors according to an embodiment of the present disclosure. 
     The processor  260  may acquire the IP ratio of each word based on the speech data corresponding to the speech uttered by each voice actor. 
     Each of the plurality of voice actors  2401  to  2403  may utter words with different breaks according to the unique utterance style thereof, even if the words are the same. 
     Referring to  FIG. 24 , a first IP ratio table  2410  obtained based on the speech uttered by the first voice actor  2401 , a second IP ratio table  2430  obtained based on the speech uttered by a second voice actor  2403  and a third IP table  2450  obtained based on the speech uttered by a third voice actor  2405  are shown. 
     The first IP ratio table  2410  includes an IP ratio in which the first voice actor  2401  reads each of a plurality of words with break corresponding to the IP class and a non-IP ratio in which the first voice actor  2401  reads each of a plurality of words with break corresponding to the non-IP class. 
     The second IP ratio table  2430  includes an IP ratio in which the second voice actor  2403  reads each of a plurality of words with break corresponding to the IP class and a non-IP ratio in which the first voice actor  2403  reads each of a plurality of words with break corresponding to the non-IP class. 
     The second IP ratio table  2450  includes an IP ratio in which the second voice actor  2405  reads each of a plurality of words with break corresponding to the IP class and a non-IP ratio in which the first voice actor  2405  reads each of a plurality of words with break corresponding to the non-IP class. 
     In  FIG. 24 , the IP ratio tables of three voice actors are described but this is merely an example and IP ratio tables of more voice actors may be obtained. 
       FIG. 23  will be described. 
     The processor  260  of the speech synthesizer  30  acquires a normal IP ratio table indicating a normal IP ratio of each word using the plurality of acquired IP ratio tables (S 2303 ). 
     The processor  260  may calculate an average of the IP ratios of each word included in each of the plurality of IP ratio tables. 
     For example, when the IP ratios of three voice actors for each word are a1, a2 and a3, the normal IP ratio may be (a1+a2+a3)/3. 
     This will be described with reference to  FIG. 25 . 
       FIG. 25  is a view illustrating a normal IP ratio table according to an embodiment of the present disclosure. 
       FIG. 24  is used to describe  FIG. 25 . 
     Referring to  FIG. 25 , the normal IP ratio table  2500  is shown. 
     The IP ratio of each word included in the normal IP ratio table  2500  may be obtained by the average of the IP ratios of each word included in each of the IP ratio tables shown in  FIG. 24 . 
     For example, the normal IP ratio of a word &lt;Because&gt; will be described. 
     The IP ratio of &lt;Because&gt; is 0.84 in the first IP ratio table  2410 , is 0.80 in the second IP ratio table  2430 , and is 0.70 in the third IP ratio table  2450 . 
     The normal IP ratio of &lt;Because&gt; is 0.780, by calculating (0.84+0.80+0.70)/3. 
     The normal non-IP ratio of &lt;Because&gt; may be obtained by subtracting the normal IP ratio from 1. 
     In this manner, the normal IP ratio of each of the plurality of words may be calculated and the normal IP ratio table  2500  which is a combination of the normal IP ratios may be obtained. 
     As described below, the normal IP ratio table  2500  may be used to train the normal synthesized speech model. 
       FIG. 23  will be described again. 
     The processor  260  of the speech synthesizer  30  generates a normal synthesized speech model based on the acquired normal IP ratio table (S 2305 ). 
     The normal synthesized speech model may be a model for determining a probability that a word is classified as an IP class, a probability that a word is classified as a WP class and a probability that a word is classified as an AP class, using the normal IP ratio of each word included in the normal IP ratio table. 
     After the normal synthesized speech model is trained, the reading break of the word may be determined according to the class having the highest probability among the determined probabilities when the synthesized speech is generated. 
     The normal synthesized speech model may be an artificial neural network based model learned by a deep learning algorithm or a machine learning algorithm. 
     The normal synthesized speech model may be learned through supervised learning. 
     Learning data used for the normal synthesized speech model may include text data corresponding to a sentence including a plurality of words, a normal IP ratio of each of the plurality of words, and an IP class classification probability labeled with each word. 
     Detailed description of the normal synthesized speech model may be replaced with the embodiment of  FIG. 17 . 
     The processor  260  of the speech synthesizer  30  transmits the generated normal synthesized speech model to the terminal  100  through the communication unit  210  (S 2307 ). 
     The terminal  100  stores the normal synthesized speech model received from the speech synthesizer  30  in the memory  170  and collect utterance information of the user (S 2309 ). 
     The terminal  100  may output a speech based on the normal synthesized speech model. 
     The terminal  100  may receive the speech data uttered by the user and acquire an intention of the received speech data. 
     The terminal  100  may acquire text corresponding to the acquired intention and output a synthesized speech corresponding to the acquired text using the normal synthesized speech model. The terminal  100  may output the synthesized speech, to which the reading break of each word included in the synthesized speech is applied. 
     The terminal  100  may collect utterance information based on the speech data uttered by the user. 
     The utterance information may include the number of times of reading words configuring a sentence uttered by the user in the IP class, the part of speech of the word, and the position of the word in the sentence. 
     The processor  260  of the speech synthesizer  30  receives the utterance information of the user collected by the terminal  100  through the communication unit  210  (S 2311 ). 
     The processor  260  may use the received utterance information to acquire a personalized IP ratio table. 
     The personalized IP ratio table may be data used to train the personalized synthesized speech model. The personalized IP ratio table may include IP ratios of each word applied to a specific person. 
     The processor  260  of the speech synthesizer  30  acquires the utterance IP ratio of each of a plurality of uttered words based on the received utterance information of the user (S 2313 ). 
     The utterance IP ratio may be an IP ratio correspond to each of the plurality of words uttered by the user. 
     The utterance IP ratio may be a ratio in which the words uttered by the user are read with break corresponding to the IP class. 
     The processor  260  may change the normal IP ratio of each word used in the normal synthesized speech model to a newly acquired utterance IP ratio. 
     This will be described with reference to  FIG. 26 . 
       FIG. 26  is a view illustrating a table including personalized IP ratios and normal IP ratios of acquired words based on utterance information of a user. 
     Referring to  FIG. 26 , a table  2600  includes a normal IP ratio  2610  and a personalized IP ratio  2630  of each of a plurality of words. 
     For example, the normal IP ratio of a word &lt;Because&gt; obtained based on utterance of the plurality of voice actors may be 0.78 and the personalized IP ratio of the word &lt;Because&gt; obtained based on utterance of a specific person may be 0.75. 
     For example, the personalized IP ratio may be obtained through the number of times that the user reads the word &lt;Because&gt; included in 100 sentences with break corresponding to the IP class. 
     In addition, the normal IP ratio of a word &lt;For&gt; obtained based on utterance of the plurality of voice actors may be 0.697 and the personalized IP ratio of the word &lt;For&gt; obtained based on utterance of a specific person may be 0.65. 
     That is, the IP ratio may be changed to generate the personalized synthesized speech model. 
     The personalized synthesized speech model may be a personalized model for outputting a synthesized speech, to which the reading-with-break style of the user is applied. 
     The personalized synthesized speech model may be a model for outputting a synthesized speech optimized for the user who receives the speech recognition service through the terminal  100 . 
     Meanwhile, the collected utterance information of the user requires personalized IP ratios of words unuttered by the user. 
     That is, referring to  FIG. 26 , words &lt;We&gt;, &lt;And&gt; and &lt;This&gt; are unuttered by the user and thus the personalized IP ratios thereof may not be directly calculated. 
     To this end, the processor  260  may estimate the normal IP ratios of the unuttered words. 
       FIG. 24  will be described again. 
     The processor  260  of the speech synthesizer  30  estimates the non-utterance IP ratios respectively corresponding to the unuttered words based on the utterance information and the acquired utterance IP ratios (S 2315 ). 
     The utterance information may include the part of speech of the word uttered by the user, the length of the uttered sentence and the position of the uttered word in the sentence. 
     The processor  260  may estimate the non-utterance IP ratios of the unuttered words based on the utterance information and the personalized IP ratios of the uttered words. 
     The processor  260  may determine the IP ratio of the unuttered word based on the IP ratio model. 
     The IP ratio model may be a model for determining a probability that the unuttered word is classified as an IP class, using the unuttered word, the property of the unuttered word, an IP ratio of the uttered word having a property similar to that of the unuttered word, and labeling data (a probability of being classified as the IP class). 
     The uttered word having the property similar to that of the unuttered word may be a word similar to the unuttered word. 
     The processor  260  may determine the probability that the word is classified as the IP class as the IP ratio of the unuttered word. 
     This will be described with reference to  FIG. 27 . 
       FIG. 27  is a view illustrating an IP ratio model according to an embodiment of the present disclosure. 
     The IP ratio model  2700  may be a model for estimating the IP ratio of a word unuttered by the user. 
     The IP ratio model may be an artificial neural network based model learned by a deep learning algorithm or a machine learning algorithm. 
     Learning data used to train the IP ratio model may include an unuttered word, a property of the unuttered word, and labeling data (a probability of being classified as the IP class). 
     The property of the unuttered word may include one or more of the part of speech of the unuttered word or the position of the unuttered word in the sentence. 
     The IP ratio model  2700  may be a model for accurately inferring the probability that the unuttered word is classified as the IP class, using, as input data, an unuttered word, a property of the unuttered word, and the probability that the word is classified as the IP class. 
     The labeling data may be a personalized IP ratio of an uttered word having the same part of speech as an unuttered word and having a position in the sentence similar to that of the unuttered word. 
     The processor  260  may acquire the personalized IP ratio of the word unuttered by the user using the IP ratio model  2700 . 
       FIG. 28  is a view illustrating a result of estimating personalized IP ratios of words unuttered by a user according to an embodiment of the present disclosure. 
     Referring to the table  2800  of  FIG. 28 , personalized IP ratios  2810  (non-utterance IP ratios) of the words &lt;We&gt;, &lt;And&gt; and &lt;This&gt; unuttered by the user, which are estimated using the IP ratio model  2700 , may be included. 
     The personalized IP ratios respectively corresponding to the words unuttered by the user may be used to train the personalized synthesized speech model. 
       FIG. 23  will be described again. 
     The processor  260  of the speech synthesizer  30  generates a personalized synthesized speech model based on the utterance IP ratios and the estimated non-utterance IP ratios (S 2317 ). 
     The processor  260  may generate the personalized synthesized speech model based on the IP ratios of the uttered words and the non-utterance IP ratios. 
     The personalized synthesized speech model may be a model for outputting the synthesized speech, to which the reading-with-break style of the user is applied. 
     The method of training the personalized synthesized speech model may be equal to the method of training the synthesized speech model shown in  FIG. 17 . However, as the IP ratios used as the learning data, the personalized IP ratios shown in  FIG. 28  may be used. 
     The personalized synthesized speech model may be an artificial neural network based model learned by a deep learning algorithm or a machine learning algorithm. 
     The processor  260  may apply the oversampling scheme described with reference to  FIGS. 8 and 9  to the personalized synthesized speech model, in order to increase reading break prediction performance. 
     In another example, the processor  260  may apply the embodiment of  FIG. 15  to the personalized synthesized speech model in order to increase reading break prediction performance. 
     The processor  260  of the speech synthesizer  30  transmits the personalized synthesized speech model to the terminal  100  through the communication unit  210  (S 2319 ). 
     The terminal  100  may store the personalized synthesized speech model in the memory  170 . 
     The terminal  100  may output the synthesized speech suiting the reading-with-break style of the user when the speech recognition service is provided to the user. 
     Therefore, the user can listen to the speech suiting the utterance style thereof, thereby greatly improving satisfaction with the speech recognition service. 
     Meanwhile, when utterance information of words unuttered by the user are collected, the processor  260  may update the personalized synthesized speech model. 
     This will be described with reference to  FIG. 29 . 
       FIG. 29  is a flowchart illustrating a process of updating a personalized synthesized speech model at a speech synthesizer according to an embodiment of the present disclosure. 
     Referring to  FIG. 29 , the processor  260  of the speech synthesizer  30  collects the utterance information of an unuttered word (S 2901 ). 
     The processor  260  may receive the utterance information of the unuttered word from the terminal  100 . 
     The utterance information of the unuttered word may include the number of times of reading a word in the IP class, the part of speech of the word, and the position of the word in the sentence. 
     The processor  260  of the speech synthesizer  30  acquires the IP ratio of the unuttered word based on the collected utterance information of the unuttered word (S 2903 ). 
     The processor  260  may acquire the IP ratio based on the number of times of reading the unuttered word in the IP class. 
     The processor  260  of the speech synthesizer  30  determines whether a difference between the acquired IP ratio and the IP ratio estimated in step S 2315  is equal to or greater than a predetermined ratio (S 2905 ). 
     The predetermined ratio may be 0.05 but this is merely an example. 
     The processor  260  of the speech synthesizer  30  retrains the personalized synthesized speech model when the difference between the acquired IP ratio and the estimated IP ratio is equal to or greater than the predetermined ratio (S 2907 ). 
     When the difference between the acquired IP ratio and the estimated IP ratio is large, the processor  260  may retrain the personalized synthesized speech model in order to accurately generate the personalized synthesized speech model. 
     That is, the processor  260  may train the personalized synthesized speech model by changing the estimated IP ratio of the unuttered word to the IP ratio of the newly uttered word. 
     The processor  260  of the speech synthesizer  30  acquires the personalized synthesized speech model updated by retraining and transmits the updated personalized synthesized speech model to the terminal  100  (S 2909 ). 
     The terminal  100  may store the updated personalized synthesized speech model in the memory  170  and provide the speech service through the updated personalized synthesized speech model. 
     Therefore, it is possible to provide the synthesized speech, to which the reading-with-break style of the user is applied, thereby greatly improving user&#39;s satisfaction with the speech recognition service. 
       FIG. 30  is a ladder diagram illustrating a method of operating a system according to another embodiment of the disclosure. 
     In particular,  FIG. 30  relates to a method of generating a synthesized speech model by reflecting a user&#39;s reading break style. 
     The processor  180  of the terminal  100  collects utterance information of a user (S 3001 ). 
     The processor  180  of the terminal  100  may receive speech uttered by a specific user through the microphone  122  and collect utterance information from the received speech. 
     An object of the present disclosure is to generate a synthesized speech model capable of outputting a synthesized speech with an optimized reading break to the specific user. The utterance information may be collected based on the speech uttered by the user. 
     The utterance information may include the number of times of reading words configuring a sentence uttered by the user in the IP class, the part of speech of each word and the position of a word in the sentence. The processor  180  may acquire the utterance information using an automatic speech recognition (ASR) model. The ASR model may extract utterance features from the speech uttered by the user. 
     The processor  180  may collect the utterance information for a certain period. 
     The wireless communication unit  110  of the terminal  100  transmits the collected utterance information of the user to the communication unit  210  of the speech synthesizer  30  (S 3003 ). 
     The processor  260  may use the received utterance information to acquire a personalized IP ratio table. 
     The personalized IP ratio table may be used to train a personalized synthesized speech model. The personalized IP ratio table may include IP ratios of each word applied to the specific user. 
     The processor  260  of the speech synthesizer  30  acquires utterance IP ratios of a plurality of uttered words based on the utterance information collected from the terminal  100  (S 3005 ). 
     The utterance IP ratios may be IP ratios respectively corresponding to a plurality of words uttered by the user. 
     The utterance IP ratio may be a ratio of reading words uttered by the user with break in the IP class. 
     The processor  260  may acquire the utterance IP ratio of each word based on the number of times of classifying each word included in the utterance information as the IP class. 
     This will be described with reference to  FIG. 31 . 
       FIG. 31  is a view illustrating a personalized IP ratio table acquired based on utterance information of a specific user according to an embodiment of the present disclosure. 
     Referring to  FIG. 31 , the personalized IP ratio table  3100  including a plurality of utterance IP ratios of a plurality of acquired words based on the utterance information of a specific user is shown. 
     The personalized IP ratio of each word may be obtained through the number of times that a user reads the word included in 100 sentences with break in an IP class. 
     For example, the IP ratio (0.75) of a word &lt;Because&gt; may indicate that a specific user reads the word &lt;Because&gt; in the IP class 75 times in 100 sentences including &lt;Because&gt;. 
     Similarly, the IP ratio (0.65) of a word &lt;For&gt; may indicate that a specific user reads the word &lt;For&gt; in the IP class 65 times in 100 sentences including &lt;For&gt;. 
     The personalized synthesized speech model may be a personalized model for outputting a synthesized speech, to which the reading-with-break style of the user is applied. The personalized IP ratios of words which are not uttered by the specific user are required to train the personalized synthesized speech model. 
     Referring to  FIG. 31 , words &lt;We&gt;, &lt;And&gt; and &lt;This&gt; are not uttered by the user and thus the personalized IP ratios thereof cannot be directly calculated. 
     To this end, the processor  260  may estimate the IP ratios of the unuttered words using the IP ratios of the words uttered by a plurality of voice actors. 
       FIG. 30  will be described again. 
     The processor  260  of the speech synthesizer  30  compares the plurality of acquired utterance IP ratios with a plurality of IP ratio tables respectively corresponding to the plurality of voice actors (S 3007 ). 
     The database  232  of the speech synthesizer  30  may store speech data uttered by the plurality of voice actors. 
     The processor  260  may acquire an IP ratio table based on the speech data uttered by each voice actor. 
     For this, refer to the description of  FIG. 24 . 
     The processor  260  may compare the personalized IP ratio table including the plurality of acquired utterance IP ratios with the plurality of IP ratios respectively corresponding to the plurality of voice actors, based on the utterance information of the specific user. 
     The processor  260  may perform comparison operation in order to acquire the respective IP ratios of the unuttered words which are not uttered by the specific user. 
     The processor  260  of the speech synthesizer  30  determines an IP ratio table closest to the plurality of utterance IP ratios among the plurality of IP ratio tables according to the result of comparison (S 3009 ). 
     This will be described with reference to  FIG. 32 . 
       FIG. 32  is a view illustrating a process of comparing a personalized IP ratio table with a plurality of IP ratio tables respectively corresponding to a plurality of voice actors according to an embodiment of the present disclosure. 
     Referring to  FIG. 32 , the personalized IP ratio table  3100  and the plurality of IP ratio tables  2410  to  2450  are shown. 
     The processor  260  of the speech synthesizer  30  may compare the personalized IP ratio table  3100  with the plurality of IP ratio tables  2410  to  2450 . 
     Specifically, the processor  260  may compare the personalized IP ratios corresponding to the uttered words included in the personalized IP ratio table  3100  with the IP ratios of the same words included in each IP ratio table. 
     The personalized IP ratio of the word &lt;Because&gt; is 0.75. 
     The IP ratio of the word &lt;Because&gt; is 0.84 in the first IP ratio table  2410 , is 0.8 in the second IP ratio table  2430 , and is 0.7 in the third IP ratio table  2450 . 
     The processor  260  may calculate differences between 0.75 which is the personal IP ratio of the word &lt;Because&gt; and the above-described IP ratios of the word &lt;Because&gt;. 
     The results of calculation may be 0.09(0.84−0.75), 0.05(0.8−0.75) and 0.05(0.75−0.7). 
     The personalized IP ratio of the word &lt;For&gt; is 0.65. 
     The IP ratio of the word &lt;For&gt; is 0.79 in the first IP ratio table  2410 , is 0.6 in the second IP ratio table  2430 , and is 0.69 in the third IP ratio table  2450 . 
     The processor  260  may calculate differences between 0.65 which is the personal IP ratio of the word &lt;For&gt; and the above-described IP ratios of the word &lt;For&gt;. 
     The results of calculations may be 0.14(0.79−0.65), 0.05(0.65−0.6) and 0.04(0.69−0.65). 
     The processor  260  may sum up the differences between the IP ratios of the uttered words. The summed result may be obtained as 0.23 by comparison with the first IP ratio table  2410 ,  0 . 1  by comparison with the second IP ratio table  2430 , and 0.09 by comparison with the third IP ratio table  2450 . 
     The processor  260  may determine the third IP ratio table  2450  having a smallest summed result value as a closest IP ratio table. 
     The processor  260  may determine the IP ratios included in the third IP ratio table  2450  as the IP ratios of the unuttered words. 
     This will be described with reference to  FIG. 33 . 
       FIG. 33  is a view illustrating an example of using IP ratios included in a closest IP ratio table as IP ratios of unuttered words according to an embodiment of the present disclosure. 
     As described with reference to  FIG. 32 , it is assumed that the third IP ratio table  2450  is determined as the IP ratio table closest to the personalized IP ratio table  3100 . 
     The IP ratio of the unuttered word &lt;We&gt; may be determined as 0.68 which is the IP ratio of the word &lt;We&gt; included in the third IP ratio table  2450 . 
     Similarly, the IP ratio of the unuttered word &lt;And&gt; may be determined as 0.8 which is the IP ratio of the word &lt;And&gt; included in the third IP ratio table  2450 . 
     Similarly, the IP ratio of the unuttered word &lt;This&gt; may be determined as 0.15 which is the IP ratio of the word &lt;This&gt; included in the third IP ratio table  2450 . 
     In the same manner, the IP ratios of the unuttered words of the specific user may be determined. Finally, the completed personalized IP ratio table  3100  of  FIG. 33  may be obtained. 
     As such, when the IP ratio according to the speech generated by the voice actor is used, it is possible to solve initial data lack of the words which are not uttered by the specific user. 
       FIG. 30  will be described again. 
     The processor  260  of the speech synthesizer  30  generates a personalized synthesized speech model based on the determined IP ratio table (S 3011 ). 
     The personalized synthesized speech model may determine a probability of classifying a word as an IP class, a probability of classifying a word as a WP class and a probability of classifying a word as an AP class, using the IP ratio of each word included in the personalized IP ratio table. 
     The personalized synthesized speech model may be an artificial neural network based model trained by a deep learning algorithm or a machine learning algorithm. 
     The personalized synthesized speech model may be trained through supervised learning. 
     Training data used in the personalized synthesized speech model may include text data corresponding to a sentence including a plurality of words and a personalized IP ratio labeled in each word. 
     The personalized synthesized speech model will be described with reference to  FIG. 34 . 
       FIG. 34  is a view illustrating a process of training a personalized synthesized speech model according to an embodiment of the present disclosure. 
     Referring to  FIG. 34 , the personalized synthesized speech model  3400  may be trained through supervised learning. 
     The personalized synthesized speech model  3400  may infer the reading break of each word with respect to the specific user. The inferred reading break of the word may be used when the synthesized speech of the word is output. 
     The training data set of the personalized synthesized speech model  3400  may include text data corresponding to a sentence including a plurality of words, an IP ratio labeled in a word configuring the text data and an IP probability which is labeling data. 
     The IP probability is a probability of being classed as an IP class and may be labeling data. 
     The personalized synthesized speech model  3400  may infer the IP probability of classifying text data corresponding to a word as the IP class. 
     The processor  260  may apply the oversampling scheme described with reference to  FIGS. 8 and 9  to the personalized synthesized speech model, in order to improve reading break prediction performance. 
     In another example, the processor  260  may apply the embodiment of  FIG. 15  to the personalized synthesized speech model, in order to improve reading break prediction performance. 
       FIG. 30  will be described again. 
     The processor  260  of the speech synthesizer  30  transmits the generated personalized synthesized speech model to the terminal  100  through the communication unit  210  (S 3013 ). 
     The terminal  100  may store the personalized synthesized speech model in the memory  170 . 
     The terminal  100  may output the synthesized speech suiting the reading-with-break style of the user, when the speech recognition service is provided to the user. 
     Therefore, the user can listen to the speech suiting the utterance style thereof, thereby greatly improving satisfaction with the speech recognition service. 
     According to the embodiment of the present disclosure, as performance of a synthesized speech model is improved, it is possible to naturally output a synthesized speech. Therefore, a listener may not feel uncomfortable when listening to the synthesized speech. 
     According to the embodiment of the present disclosure, as a personalized synthesized speech model is provided, a synthesized speech suiting the utterance style of a user is output, thereby improving user&#39;s satisfaction with a speech recognition service. 
     The present disclosure mentioned in the foregoing description can also be embodied as computer readable codes on a computer-readable recording medium. Examples of possible computer-readable mediums include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The computer may include the processor  180  of the terminal.