Patent ID: 12217016

DETAILED DESCRIPTION

The terms used in describing various example embodiments will be briefly explained, and various example embodiments will be described in greater detail with reference to the accompanying drawings.

Terms used in the disclosure are selected as general terminologies currently widely used in consideration of the configuration and functions of the disclosure, but may be different depending on intention of those skilled in the art, a precedent, appearance of new technologies, or the like. Further, in specific cases, terms may be arbitrarily selected. In this case, the meaning of the terms will be described in the description of the corresponding embodiments. Accordingly, the terms used in the description should not necessarily be construed as simple names of the terms, but be defined based on meanings of the terms and overall contents of the disclosure.

The example embodiments may vary, and may be provided in different example embodiments. Various example embodiments will be described with reference to accompanying drawings. However, this does not necessarily limit the scope of the exemplary embodiments to a specific embodiment form. Instead, modifications, equivalents and replacements included in the disclosed concept and technical scope of this specification may be employed. While describing exemplary embodiments, based on it being identified that the specific description regarding a known technology obscures the gist of the disclosure, the specific description is omitted.

The terms such as “first,” “second,” and so on may be used to describe a variety of elements, but the elements should not be limited by these terms. The terms used herein are solely intended to explain specific example embodiments, and not to limit the scope of the disclosure.

Singular forms are intended to include plural forms unless the context clearly indicates otherwise. The terms “include”, “comprise”, “is configured to,” etc., of the description are used to indicate that there are features, numbers, steps, operations, elements, parts or combination thereof, and they should not exclude the possibilities of combination or addition of one or more features, numbers, steps, operations, elements, parts or a combination thereof.

The example embodiments of the disclosure will be described in greater detail below in a manner that will be understood by one of ordinary skill in the art. However, exemplary embodiments may be realized in a variety of different configurations, and not limited to descriptions provided herein. Also, well-known functions or constructions may not be described in detail where they would obscure the disclosure with unnecessary detail.

FIG.1is a block diagram illustrating an example configuration of an electronic apparatus according to an embodiment.

Referring toFIG.1, the electronic apparatus100may include a microphone110, a display120, a speaker130, a communication interface140, a memory150, and a processor160. For example, the electronic apparatus100may be a user terminal.

The microphone110may acquire a user voice, for example as a user voice input. For example, the microphone110may be formed integrally with an upper side, a front direction, a side direction, or the like of the electronic apparatus100. The microphone110may include various configurations such as a microphone that collects analog user voice, an amplifier circuit that amplifies the collected user voice input, an A/D conversion circuit that samples the amplified user voice input and converts the user voice input into a digital signal, a filter circuit for removing noise components from the converted digital signal, or the like.

The display120may display text corresponding to the user voice. For example, the display120may display a first text corresponding to a user voice in a first language. Alternatively, the display120may display a second text of a second language in which the first text is translated. The display120may be implemented as a liquid crystal display panel (LCD), organic light emitting diodes (OLED), a flexible display, a transparent display, or the like. However, the display120according to the disclosure is not limited to a specific type.

The speaker130may output audio under the control of the processor160. For example, the speaker130may output a voice message corresponding to an output sentence. The output sentence may be a sentence of the second language (or target language) acquired based on an input sentence of the first language (or source language).

The communication interface140may include at least one communication circuit and may communicate with various types of external devices or external servers. For example, the communication interface140may transmit a feature vector corresponding to the user voice of the electronic apparatus100to the external device or receive a feature vector corresponding to the user voice of the external device from the external device. The communication interface140may include at least one of a Wi-Fi communication module, a cellular communication module, a 3rd generation (3G) mobile communication module, a 4th generation (4G) mobile communication module, and a 4th generation Long term evolution (LTE) mobile communication module, a 5th generation (5G) mobile communication module.

The memory150may store an operating system (OS) for controlling the overall operation of components of the electronic apparatus100and commands or data related to the components of the electronic apparatus100. For example, the memory150may store information on a first neural network model and information on a second neural network model. The first neural network model may be learned, or trained, to output an output sentence of the second language based on the input sentence of the first language. The first neural network model may be a monotonic multi-head attention (MMA) model. The second neural network model may be trained to output information on a subsequent token predicted to be acquired after an output token acquired at a previous point in time. The second neural network model may be a language model (LM). The memory150may be implemented as a non-volatile memory (e.g., a hard disk, a solid state drive (SSD), a flash memory), a volatile memory, or the like.

The processor160may be electrically connected to the memory150to control overall functions and operations of the electronic apparatus100. The processor160may acquire a first user voice in the first language acquired through the microphone110. For example, the first language may be English. The processor160may acquire a first token corresponding to the first user voice. A token means a character string having a meaning. For example, a token may mean a word or a sentence.

The processor160may acquire a first text of the second language by inputting the first token into the first neural network model. For example, the second language may be Korean. The first neural network model may be trained to acquire text in the second language corresponding to the input token or to identify an additional input token in addition to the input token, based on the input token of the first language input to the first neural network model.

The first neural network model may include a first encoder and a first decoder. The first decoder may be trained to acquire a text of the second language corresponding to the input token based on a context vector acquired by inputting the input token to the first encoder. The first decoder may be trained to acquire the text of the second language corresponding to the input token in response to a probability value being acquired based on the context vector is greater than a predetermined value, and to identify additional tokens in addition to the input token based on the probability value being less than the predetermined value. The probability value may be acquired based on a function defined based on the context vector acquired through the first encoder.

The first neural network model may include a plurality of modules learned to acquire vectors corresponding to a plurality of features among a plurality of input tokens. For example, the first neural network model may include a first attention module for acquiring a first vector corresponding to a first feature between the plurality of input tokens. The first feature may be a grammatical relationship between the plurality of input tokens. The first neural network model may include a second attention module for acquiring a second vector corresponding to a second feature between the plurality of input tokens. The second feature may mean a semantic relationship between the plurality of input tokens. The first neural network model may be trained to acquire a second text in a second language based on the first vector and the second vector.

The processor160may output at least one text acquired by using the first neural network model. For example, the processor160may control the display120to display at least one text. Alternatively, the processor160may control the speaker130to output a voice message corresponding to at least one text.

Functions related to artificial intelligence according to the disclosure may operate through the processor160and the memory150. The processor160may include one or a plurality of processors. In this example, one or the plurality of processors may include, for example, and without limitation, a general-purpose processor such as a CPU, AP, or a digital signal processor (DSP), a graphics-only processor such as a GPU, a vision processing unit (VPU), or an artificial intelligence-only processor such as an NPU. One or more processors control to process input data according to a predefined operation rule or artificial intelligence model stored in the memory150. Based on one or the plurality of processors being artificial intelligence-only processors, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

The predefined operation rule or artificial intelligence model is characterized in that it is generated through learning. Being generated through learning may refer, for example, to a basic artificial intelligence model being learned using a plurality of learning data by a learning algorithm, such that the predefined operation rule or artificial intelligence model set to perform a desired feature (or purpose) is generated. Such learning may be performed in a device itself on which the artificial intelligence according to the disclosure is performed, or may be performed through a separate server and/or system. Examples of the learning algorithm include, for example, and without limitation, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

The artificial intelligence model may be generated through learning. The artificial intelligence model may be composed of a plurality of neural network layers. Each of the plurality of neural network layers may have a plurality of weight values, and perform a neural network operation through an operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, the plurality of weights may be updated to reduce or minimize a loss value or a cost value acquired from the artificial intelligence model during the learning process.

The artificial neural network may include, for example, and without limitation, a deep neural network (DNN), such as convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), generative adversarial network (GAN), restricted Boltzmann machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), or deep Q-Networks, or the like, but is not limited to the embodiment described above.

Meanwhile, the electronic apparatus100may include an adaptive echo canceller (AEC) module, a noise suppression (NS) module, and an end-point detection (EPD) module for pre-processing the user voice, or may include an automatic gain control (AGC) module. Also, the electronic apparatus100may include a digital signal processor (DSP) that converts an analog audio signal into a digital signal or converts a stored digital image or digital audio signal into an analog signal.

FIG.2is a view illustrating a method of acquiring output text according to an embodiment.

Referring toFIG.2, the electronic apparatus100may acquire a feature value ŷicorresponding to a present point in time by inputting an output text yi-n, . . . , yi-1up to a time point before the present point in time into a second neural network model, which may be for example the LM. The feature value ŷimay correspond to a subsequent word (or token) predicted to be uttered at the present point in time.

The electronic apparatus100may acquire an output text yicorresponding to the present point in time or identify an input token xjcorresponding to the present point in time by inputting the output text yi-n, . . . , yi-1up to the previous point in time, the input token xj-n, . . . , xj-1up to the previous point in time, and the feature value ŷito a first neural network model, which may be, for example, the MMA Model. The first neural network model (for example the MMA Model) may perform a read operation of identifying an input token based on input data and loading it into a memory, or a write operation of generating an output token by translating the loaded input token.

For example, in response to a probability value being acquired based on a context vector identified based on the input token xj-n, . . . , xj-1is greater than a predetermined value, the first neural network model (for example the MMA Model) may generate the output text by performing the write operation. Based on the probability value being smaller than the predetermined value, the first neural network model (for example the MMA Model) may identify the input token xjand load it into the memory by performing the read operation. The probability value may be related to the amount of information input to the first neural network model (for example the MMA Model). For example, based on information input to the first neural network model (for example the MMA Model) at the present point in time being sufficient to generate output text (i.e., when translation quality is identified to be above a predetermined level), the first neural network model (MMA) may generate output text. Meanwhile, based on the information input to the first neural network model (for example the MMA Model) at the present point in time being insufficient to generate the output text (i.e., when the translation quality is identified to be lower than the predetermined level), the first neural network model (for example the MMA Model) may identify additional input tokens.

The existing translation model does not consider the feature value ŷiacquired through the second neural network model (for example the LM) when generating output text. However, the electronic apparatus100according to the disclosure acquires the output text yiby using the feature value ŷi corresponding to a word predicted by the second neural network model (for example the LM). Accordingly, the output text yiacquired through the first neural network model (for example the MMA Model) may have superior quality compared to the output text acquired through the existing translation model.

Meanwhile, the first neural network model (for example the MMA Model) and the second neural network model (for example the LM) may be integrated into one neural network model. In addition, data type of the input token may be text, but is not limited thereto, and may be audio data type.

FIG.3is a view illustrating a method of acquiring output text according to an embodiment.

Referring toFIG.3, at a first time point t1, the first neural network model (for example the MMA Model) may identify a first token x1by performing a read operation

At a second time point t2, the electronic apparatus100may acquire a first text y1by inputting the first token x1into the first neural network model (for example the MMA Model). In this case, the first neural network model (for example the MMA Model) may generate the first text y1by performing a write operation.

At a third time point t3, the electronic apparatus100may acquire a first feature value ŷ1by inputting the first text y1into the second neural network model LM. The electronic apparatus100may input the first token x1, the first text y1, and the first feature value ŷ1into the first neural network model (for example the MMA Model). In this case, the first neural network model (for example the MMA Model) may identify the second token x2by performing a read operation.

At a fourth time point t4, the electronic apparatus100may acquire a second text y2by inputting the first token x1, the second token x2, the first text y1, and the first feature value ŷ1into the first neural network model LM. In this case, the first neural network model (for example the MMA Model) may generate the second text y2by performing a write operation.

At a ninth time point t9, the electronic apparatus100may identify a fourth feature value ŷ4by inputting the first text y1, the second text y2, and the third text y3into the second neural network model LM. The electronic apparatus100may acquire the fourth feature value y4by inputting the first token x1, the second token x2, a third token x3, a fourth token x4, a fifth token x5, the first text y1, the second text y2, a third text y3, and the fourth feature value ŷ4into the first neural network model MMA Model.

In this way, the electronic apparatus100may acquire text of the second language from an input token of the first language.

FIG.4is a view illustrating a first neural network model according to an embodiment.

Referring toFIG.4, the first neural network model (for example the MMA Model) may include a first attention module and a second attention module. The first attention module and the second attention module are modules for acquiring information on features between input tokens x1, x2, . . . , xn. For example, the first attention module may output a first vector v1indicating a grammatical relationship between the input tokens x1, x2, . . . , xn. The second attention module may output a second vector v2indicating a semantic relationship between x1, x2, . . . , xn. The first neural network model (for example the MMA Model) may acquire output text y based on the first vector v1and the second vector v2.

FIG.5is a view illustrating a method of acquiring output text according to an embodiment.

Referring toFIG.5, the electronic apparatus100may acquire output texts61to64of a second language based on input tokens51to58of the first language acquired in real time using the first neural network model (for example the MMA Model) and the second neural network model (for example the LM).

At a first time point t1, the electronic apparatus100may acquire a first input token51based on a user voice in the first language. The electronic apparatus100may acquire a first output text61by inputting the first input token51into the first neural network model (for example the MMA Model). In this case, the first neural network model (for example the MMA Model) may generate the first output text61by performing a write operation.

At a second time point t2, the electronic apparatus100may acquire a second input token52following the first input token51. The electronic apparatus100may acquire a first feature value ŷ1by inputting the first output text61into the second neural network model LM. The electronic apparatus100may input the first input token51, the second input token52, the first output text61, and the first feature value ŷ1into the first neural network model (for example the MMA Model). The first neural network model (for example the MMA Model) may identify a third input token53following the second input token52by performing a read operation without generating new output text.

At a third time point t3, the electronic apparatus100may acquire a second output text62by inputting the first input token51, the second input token52, the third input token53, the first output text61, and the first feature value ŷ1into the first neural network model (for example the MMA Model). In this case, the first neural network model (for example the MMA Model) may generate the second output text62by performing a write operation.

At a fourth time point t4, the electronic apparatus100may acquire a fourth input token54following the third input token53. The electronic apparatus100may acquire the second feature value ŷ2by inputting the first output text61and the second output text62into the second neural network model LM. The electronic apparatus100may input the first input token51, the second input token52, the third input token53, the fourth input token54, the first output text61, the second output text62, and the second feature value ŷ2into the first neural network model (for example the MMA Model). The first neural network model (for example the MMA Model) may identify a fifth input token55by performing a read operation without generating new output text.

At a fifth time point t5, the electronic apparatus100may input the first input token51, the second input token52, the third input token53, the fourth input token54, the fifth input token55, the first output text61, the second output text62, and the second feature value ŷ2into the first neural network model (for example the MMA Model). The first neural network model (for example the MMA Model) may identify a sixth input token56following the fifth input token55by performing a read operation without generating new output text.

At a sixth time point t6, the electronic apparatus100may acquire a third output text63by inputting the first input token51, the second input token52, the third input token53, the fourth input token54, and the fifth input token55, the sixth input token56, the first output text61, the second output text62, and the second feature value ŷ2. The first neural network model (for example the MMA Model) may generate the third output text63by performing a write operation.

At a seventh time point t7, the electronic apparatus100may acquire a seventh input token57following the sixth input token56. The electronic apparatus100may acquire the third feature value ŷ3by inputting the first output text61, the second output text62, and the third output text63into the second neural network model LM The electronic apparatus100may input the first input token51, the second input token52, the third input token53, the fourth input token54, the fifth input token55, the sixth input token56, the seventh input token57, the first output text61, the second output text62, the third output text63, and the third feature value ŷ3. The first neural network model (for example the MMA Model) may identify an eighth input token58following the seventh input token57by performing a read operation without generating new output text.

At an eighth time point t8, the electronic apparatus100may acquire a fourth output text64by inputting the first input token51, the second input token52, the third input token53, the fourth input token54, and the fifth input token55, the sixth input token56, the seventh input token57, an eighth input token58, the first output text61, the second output text62, the third output text63, and the third feature value ŷ3into the first neural network model (for example the MMA Model). The first neural network model (for example the MMA Model) may generate the fourth output text64by performing a write operation.

FIGS.6and7are views illustrating a text output method of an electronic apparatus according to an embodiment.

Referring toFIG.6, the electronic apparatus100may acquire a user voice71of a first language uttered by a user1. The electronic apparatus100may acquire text72of the second language based on the user voice71. The electronic apparatus100may display the acquired text72.

Referring toFIG.7, the electronic apparatus100may output a voice message73corresponding to the text72acquired based on the user voice71.

FIG.8is a flowchart illustrating a method of controlling an electronic apparatus according to an embodiment.

Referring toFIG.8, the electronic apparatus100may acquire a first token corresponding to a first user voice in a first language (S810). The first token may be audio data or text corresponding to the first user voice.

The electronic apparatus100may acquire a first text of a second language by inputting the first token into a first neural network model (S820). The first neural network model may be a neural network model learned to acquire text in a second language based on an input token in the first language or to identify an additional input token in addition to the input token. The first neural network model may include a first encoder that outputs a context vector based on the input token and a first decoder that generates an output text based on the context vector. The first decoder may generate an output text or identify the additional input token according to a probability value acquired based on the context vector. For example, based on the probability value being greater than a predetermined value, the first decoder may acquire text of the second language corresponding to the input token. Meanwhile, based on the probability value being smaller than the predetermined value, the first decoder may identify an additional token in addition to the input token.

The first neural network model may include a first attention module for acquiring a first vector corresponding to a first feature between a plurality of input tokens and a second attention module for acquiring a second vector corresponding to a second feature between the plurality of input tokens. For example, the first feature may be a grammatical relationship between the plurality of input tokens. The second feature may mean a semantic relationship between the plurality of input tokens. The first neural network model may generate a second text in a second language based on the first vector and the second vector.

The electronic apparatus100may acquire a feature value corresponding to a subsequent token predicted to be uttered after the first token by inputting the first text into the second neural network model (S830). The second neural network model may be trained to predict text that will follow the input text based on the input text.

Based on a second token following the first token being acquired, the electronic apparatus100may acquire the second text of the second language by inputting the first token, the second token, the first text and the feature value into the first neural network model (S840). The electronic apparatus100may input the input token acquired up to the present point in time, output text acquired up to the present point in time, and the feature value into the first neural network model. The first neural network model may perform a write operation of generating output text or a read operation of identifying an additional input token based on input information.

Various example embodiments described above may be embodied in a recording medium that may be read by a computer or a similar apparatus to the computer using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software configuration, various embodiments described in the specification such as a procedure and a function may be implemented as separate software modules. The software modules may respectively perform one or more functions and operations described in the disclosure

According to various embodiments described above, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer-readable medium may cause a particular device to perform processing operations on the device according to the various embodiments described above based on being executed by the processor of the particularly device.

The non-transitory computer-readable medium does not refer to a medium that stores data for a short period of time, such as a register, cache, memory, etc., but semi-permanently stores data and is available of reading by the device. For example, the non-transitory computer-readable medium may include, for example, and without limitation, a CD, DVD, a hard disc, Blu-ray disc, USB, a memory card, ROM, or the like.

The machine-readable storage media may be provided in a form of a non-transitory storage media. The term “non-transitory storage medium may refer to a tangible device and does not include a signal (e.g., electromagnetic wave), and the term does not distinguish between the case that the data is permanently stored in the storage medium and the case that the data is temporarily stored in the storage medium. For example, the “non-transitory storage medium” may include a buffer in which data is temporarily stored.

According to an embodiment, the method according to various embodiments disclosed in the disclosure may be provided as being included in a computer program product. The computer program product may be traded between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or through application stores (e.g., Play Store™), or may be distributed (e.g., downloaded or uploaded) directly or online between two user devices (e.g., smartphones). In the case of online distribution, at least some of the computer program products (e.g. downloadable apps) may be temporarily stored on a storage medium readable by a device, such as a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.

While the disclosure has been illustrated and described with reference to various example embodiments, it will be understood that the various example embodiments are intended to be illustrative, not limiting. It will be understood by those skilled in the art that many alternatives, modifications, and variations may be made without departing from the true spirit and full scope of the disclosure, including the appended claims and their equivalents.