Patent ID: 12190907

DETAILED DESCRIPTION

Hereinafter, the disclosure will be described in detail with reference to the accompanying drawings.

Embodiments of the disclosure propose a method and a system for automatic back-channel generation in an interactive agent system. The back-channel refers to an audio signal of a short and quick response that is given to a speaker by a listener when the speaker is saying in a dialogue, such as “well”, “yes, yes.”

According to embodiments of the disclosure, in a system which automatically generates a dialogue response and exchanges an utterance with a user, a back-channel may be automatically generated and expressed in order to implement a natural interaction while a user is speaking.

Specifically, in an embodiment of the disclosure, a back-channel may be predicted based on a language model which learns linguistic knowledge from a large-capacity corpus, not based on a rule, and a natural dialogue interaction may be implemented.

In addition, an embodiment of the disclosure proposes a model that exactly predicts a time to generate a back-channel, based on a user utterance, and appropriately predicts a back-channel category.

In addition, in an embodiment of the disclosure, performance of a back-channel prediction model may be reinforced by utilizing a sentiment feature directly/indirectly from an utterance text.

FIG.1is a view illustrating a configuration of an automatic back-channel generation system according to an embodiment of the disclosure.

As shown in the drawing, the automatic back-channel generation system according to an embodiment may be established by including a voice input unit110, a voice recognition module120, an utterance voice feature extraction module130, an utterance text feature extraction module140, an utterance sentiment feature extraction module150, a back-channel prediction module160, a back-channel generation module170, and a voice output unit180.

The voice input unit110may be a means for receiving an utterance of a user, and may transmit an utterance voice of the user which is inputted to the voice recognition module120and the utterance voice feature extraction module130.

The utterance voice feature extraction module130may extract an utterance voice feature from the user's utterance voice which is inputted through the voice input unit110. The voice recognition module120may be a speech-to-text (STT) conversion module to convert the user's utterance voice, which is inputted through the voice input unit110, into an utterance text. The utterance text feature extraction module140may extract an utterance text feature from the utterance text converted at the voice recognition module120.

The utterance sentiment feature extraction module150may extract an utterance sentiment feature from the utterance text converted at the voice recognition module120. Specifically, the utterance sentiment feature extraction module150may extract sentiment distribution information by counting the number of sentiment dictionary words existing in the utterance text.

The back-channel prediction module160may predict a back-channel to express by using a back-channel prediction model.

The back-channel prediction model may be an AI model that is trained to receive the utterance voice feature extracted at the utterance voice feature extraction module130, and the utterance text feature extracted at the utterance text feature extraction module140, to analyzes the same, and to predict a back-channel.

By the back-channel prediction model, it may be predicted whether a back-channel will be expressed based on the user's utterance, that is, whether a back-channel will be generated at a present time, and a category (type) of the back-channel to be generated may be predicted.

The category of the back-channel may include a back-channel which expresses ‘continual’, a back-channel which expresses ‘understanding,’ a back-channel which expresses ‘agreement,’ a back-channel which expresses ‘emotional response,’ a back-channel which expresses ‘empathic response.’

As shown inFIG.1, the back-channel prediction model performing such functions may separately process user's utterance input by using layers161for learning the utterance voice feature and layers162for learning the utterance text feature. Hidden representations generated on the layers161,162may be combined and inputted to and processed at linear layers163. An output from the linear layers163may be a prediction value indicating whether a back-channel will be generated and a category.

Accordingly, by the back-channel prediction model, it may be predicted whether a back-channel will be expressed from the user's utterance, that is, whether a back-channel will be generated at the present time, and a category (type) of the back-channel to be generated may be predicted.

In addition, the text feature learning layers162of the back-channel prediction model may be trained to receive the utterance text feature extracted at the utterance text feature extraction module140, to analyze, and to predict sentiment distribution information.

To achieve this, the back-channel prediction model may be implemented by a deep learning-based model, and may utilize a multi-tasking learning method when learning. That is, at a step of learning by the back-channel prediction model, a main-task of predicting whether a back-channel will be generated and a category of the back-channel, and a sub-task of predicting a distribution of utterance sentiment features may be performed simultaneously, and learning by the back-channel prediction model may proceed.

The main task may be inputting the utterance voice feature and the utterance text feature to the back-channel prediction model, and training the back-channel prediction model so as to reduce a loss between a predicted back-channel value and a ground truth (GT) value.

The sub-task may be inputting the utterance text feature to the text feature learning layers162of the back-channel prediction model, and training the back-channel prediction model so as to reduce a loss between predicted sentiment distribution information and sentiment distribution information which is extracted by the utterance sentiment feature extraction module150.

Accordingly, parameters of a language model which becomes a backbone of the text feature learning layers162may be fine-tuned by learning by both the two tasks simultaneously, so that an ability to utilize knowledge on utterance sentiment information may be enhanced in prediction of a back-channel.

A total learning loss (LT) of the back-channel prediction model may include LMwhich is a main task loss and LSwhich is a sub task loss, thereby being expressed by LT=αLM+(1−α)LS. A ratio of two losses may be adjusted through a value of a and features of training data may be reflected.

The back-channel generation module170may generate a back-channel signal which is predicted by the back-channel prediction module160. The back-channel generation module170may generate a back-channel signal of a category that is predicted by the back-channel prediction module160at a time of generation of the back-channel that is predicted by the back-channel prediction module160.

The voice output unit180may output the back-channel signal which is generated by the back-channel generation module170as a voice signal.

FIG.2is a flowchart provided to explain an automatic back-channel generation method according to another embodiment of the disclosure.

When the voice input unit110receives an utterance of a user (S210), the utterance voice feature extraction module130may extract an utterance voice feature from an utterance voice of the user, which is inputted through step S210, in order to automatically generate a back-channel (S220).

The voice recognition module120may convert the utterance voice of the user, which is inputted through step S210, into an utterance text (S230), and the utterance text feature extraction module140may extract an utterance text feature from the utterance text converted at step S230(S240).

Next, the back-channel prediction module160may receive the utterance voice feature which is extracted at step S220, and the utterance text feature which is extracted at step S240, may analyze the same, and may predict a back-channel (S250).

Then, the back-channel generation module170may generate a back-channel signal which is predicted at step S250(S260), and the voice output unit180may output the back-channel signal which is generated at step S260as a voice signal (S270).

Hereinafter, another implementation method of an automatic back-channel generation system will be described in detail with reference toFIG.3. FIG.3is a view illustrating a configuration of an automatic back-channel generation system according to another embodiment of the disclosure.

As shown in the drawing, the automatic back-channel generation system according to an embodiment may be established by including a voice input unit310, a voice recognition module320, an utterance voice feature extraction module330, an utterance text feature extraction module340, an utterance sentiment feature extraction module350, a back-channel prediction module360, a back-channel generation module370, and a voice output unit380.

The voice input unit310, the voice recognition module320, the utterance voice feature extraction module330, the utterance text feature extraction module340, and the utterance sentiment feature extraction module350may be the same as the voice input unit110, the voice recognition module120, the utterance voice feature extraction module130, the utterance text feature extraction module140, and the utterance sentiment feature extraction module150shown inFIG.1in terms of functions, and thus a detailed description thereof is omitted.

The back-channel prediction module360may predict a back-channel to express by using a back-channel prediction model.

The back-channel prediction model may be an AI model that is trained to receive the utterance voice feature extracted at the utterance voice feature extraction module330, the utterance text feature extracted at the utterance text feature extraction module340, and the utterance sentiment feature extracted by the utterance sentiment feature extraction module350, to analyzes the same, and to predict a back-channel.

By the back-channel prediction model, it may be predicted whether a back-channel will be expressed based on the user's utterance, that is, whether a back-channel will be generated at a present time, and a category (type) of the back-channel to be generated may be predicted.

The back-channel prediction model performing such functions may separately process user's utterance input by using layers361for learning the utterance voice feature and layers362for learning the utterance text feature and the utterance sentiment feature. Hidden representations generated on the layers361,362may be combined and inputted to and processed at linear layers363. An output from the linear layers363may be a prediction value indicating whether a back-channel will be generated and a category.

The back-channel prediction model may be implemented by a deep learning-based model, and the utterance voice feature, the utterance text feature, and the utterance sentiment feature may be inputted to the back-channel prediction model, and the back-channel prediction model may be trained so as to reduce a loss between a predicted back-channel value and a ground truth (GT) value.

Functions of the back-channel generation module370and the voice output unit380may be the same as the functions of the back-channel generation module170and the voice output unit180ofFIG.1, and thus a detailed description thereof is omitted.

FIG.4is a flowchart provided to explain an automatic back-channel generation method according to still another embodiment of the disclosure.

When the voice input unit310receives an utterance of a user (S410), the utterance voice feature extraction module330may extract an utterance voice feature from an utterance voice of the user, which is inputted through step S410, in order to automatically generate a back-channel (S420).

The voice recognition module320may convert the utterance voice of the user, which is inputted through step S410, into an utterance text (S430), the utterance text feature extraction module340may extract an utterance text feature from the utterance text converted at step S430(S440), and the utterance sentiment feature extraction module350may extract an utterance sentiment feature from the utterance text converted at step S430(S450).

Next, the back-channel prediction module360may receive the utterance voice feature which is extracted at step S420, the utterance text feature which is extracted at step S440, and the utterance sentiment feature which is extracted at step S450, may analyze the same, and may predict a back-channel (S460).

Then, the back-channel generation module370may generate a back-channel signal which is predicted at step S460(S470), and the voice output unit380may output the back-channel signal which is generated at step S470as a voice signal (S480).

Up to now, the method and the system for automatic back-channel generation in the interactive agent system have been described in detail with reference to preferred embodiments.

Embodiments of the disclosure propose an automatic back-channel generation method for implementing a natural dialogue interaction with a user in an interactive agent system and for enhancing quality of a service provided to the user.

Accordingly, a rich dialogue interaction may be implemented by predicting various functional back-channel categories, and high performance may be guaranteed by utilizing a back-channel prediction module which is based on a language model, and a back-channel suitable to an utterance context may be generated by applying a sentiment feature.

The technical concept of the disclosure may be applied to a computer-readable recording medium which records a computer program for performing the functions of the apparatus and the method according to the present embodiments. In addition, the technical idea according to various embodiments of the present disclosure may be implemented in the form of a computer readable code recorded on the computer-readable recording medium. The computer-readable recording medium may be any data storage device that can be read by a computer and can store data. For example, the computer-readable recording medium may be a read only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical disk, a hard disk drive, or the like. A computer readable code or program that is stored in the computer readable recording medium may be transmitted via a network connected between computers.

In addition, while preferred embodiments of the present disclosure have been illustrated and described, the present disclosure is not limited to the above-described specific embodiments. Various changes can be made by a person skilled in the art without departing from the scope of the present disclosure claimed in claims, and also, changed embodiments should not be understood as being separate from the technical idea or prospect of the present disclosure.