Voice recognition apparatus and method for providing response information

A voice recognition apparatus and a method for providing response information are provided. The voice recognition apparatus according to the present disclosure includes an extractor configured to extract a first utterance element representing a user action and a second utterance element representing an object from a user's utterance voice signal; a domain determiner configured to detect an expansion domain related to the extracted first and second utterance elements based on a hierarchical domain model, and determine at least one candidate domain related to the detected expansion domain as a final domain; a communicator which performs communication with an external apparatus; and a controller configured to control the communicator to transmit information regarding the first and second utterance elements and information regarding the determined final domain.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Korean Patent Application No. 10-2013-0011292, filed on Jan. 31, 2013, the disclosure of which is incorporated herein by reference.

BACKGROUND

Methods and apparatuses consistent with exemplary embodiments relate to a voice recognition apparatus and method for providing response information, and more particularly, a voice recognition apparatus for providing response information corresponding to an utterance voice of a user, and a method for providing response information thereof.

2. Description of the Related Art

When an utterance voice of a user is received, a related art voice recognition apparatus analyzes the received utterance voice of the user to determine a domain intended by the user, and provides response information on the utterance voice of the user based on the determined domain.

However, as services of domains (related area) and services that should be provided in such a related art voice recognition apparatus becomes more diverse and complicated, there emerges a problem that the voice recognition apparatus determines a domain not intended by the user, and provides response information on the utterance voice of the user based on the determined unintended domain.

For example, when an utterance voice “what kinds of action movies are there?” is received from the user, a television (TV) program and/or a video on demand (VOD) related domain may be determined with reference to such an utterance voice, whereas a conventional voice recognition apparatus detects only one domain from among a plurality of domains without considering the possibility of such multiple domains. For example, the utterance voice “what kinds of action movies are there?” from the user may include the user's intention regarding the action movie provided in the TV program. However, the voice recognition apparatus may arbitrarily determine the VOD related domain from the user's utterance voice regardless of the user's intention, and may provide response information regarding the user's utterance voice based on the determined VOD related domain. As such, in a conventional voice recognition apparatus, there often occurs cases where response information not intended by the user is provided, and thus, the user must put up with the inconvenience of having to provide a further detailed utterance voice in order to be provided with the response information intended.

SUMMARY

One or more exemplary embodiments provide a voice recognition apparatus for providing response information corresponding to an utterance voice of a user, and a method for providing response information thereof. More particularly, one or more exemplary embodiments determine a domain in accordance with a user's intention, from among the multiple domains possible based on a user's utterance voice, in a voice recognition apparatus providing response information on the user's utterance voice in a dialogue type system, and provide response information on the user's utterance voice based on the determined domain.

According to an aspect of an exemplary embodiment, there is provided a dialogue type voice recognition apparatus including an extractor configured to extract a first utterance element representing a user action and a second utterance element representing an object from a user's utterance voice signal; a domain determiner configured to detect an expansion domain related to the extracted first and second utterance elements based on a predetermined hierarchical domain model, and determine at least one candidate domain related to the detected expansion domain as a final domain; and a communicator configured to perform communication with an external apparatus; and a controller configured to control the communicator to transmit information regarding the first and second utterance elements and information regarding the determined final domain.

The hierarchical domain model may consist of a candidate domain of a lowermost concept, where a main action corresponding to the first utterance element and a component slot corresponding to a second utterance element are matched to each other, and a virtual expansion domain predetermined as a superior concept of the candidate domain.

The domain determiner may use a multi classifier to detect a subordinate candidate domain related to the detected expansion domain, when the expansion domain where the main action corresponding to the first utterance element and the component slot corresponding to the second utterance element extracted from the extractor matched to each other is detected, and detect a candidate domain where a main action corresponding to the first utterance element and the component slot corresponding to the second utterance element matched to each other, when the expansion domain is not detected.

The domain determiner may use at least one binary classifier to detect at least one domain from among the candidate domain, where the main action corresponding to the first utterance element and the component slot corresponding to the second utterance element are matched to each other, and the expansion domain.

The domain determiner may understand a user's intention from at least one of the extracted first and second utterance elements and the detected candidate domain to determine whether or not a context initialization occurred, and when it is determined that a context initialization has not occurred, determine at least one candidate domain from among the detected candidate domains as a final domain based on a previous domain, the previous domain being the domain determined from the user's previous utterance voice before receiving the user's utterance voice signal.

The domain determiner may determine the previous domain as the final domain when one candidate domain from among the detected candidate domains and the previous domain correspond to each other.

The domain determiner may determine the candidate domain related to the first utterance element representing the user action from among at least one candidate domain related to the detected expansion domain as the final domain, when it is determined that the detected candidate domain and the previous domain are different from each other or the context initialization has occurred.

The domain determiner may determine all the detected candidate domains as the final domain when the detected candidate domain includes main action information corresponding to the first utterance element representing the user action.

The domain determiner may determine the candidate domain selected by the user as the final domain or arbitrarily select one candidate domain of among the detected candidate domains to determine as the final domain, when the detected candidate domain includes the main action information corresponding to the first utterance element representing the user action.

The domain determiner may determine the candidate domain corresponding to the previous domain from among the detected candidate domains as the final domain, when there are a plurality of previous domains, one of the plurality of previous domains and one of the candidate domains correspond to each other, and the candidate domain includes main action information corresponding to the first utterance element representing the user action.

According to an aspect of an exemplary embodiment, there is provided a method of providing response information corresponding to a user's utterance voice in a dialogue type voice recognition apparatus, the method including extracting a first utterance element representing a user action and a second utterance element representing an object from a user's utterance voice signal; detecting an expansion domain related to the extracted first and second utterance elements based on a predetermined hierarchical domain model; determining at least one candidate domain related to the detected expansion domain as a final domain; and transmitting information regarding the first and second utterance elements and information regarding the determined final domain to an external apparatus.

The hierarchical domain model may consist of a candidate domain of a lowermost concept, where a main action corresponding to the first utterance element and a component slot corresponding to a second utterance element are matched to each other, and a virtual expansion domain predetermined as a superior concept of the candidate domain.

The detecting the expansion domain may use a multi classifier to detect a subordinate candidate domain related to the detected expansion domain, when the expansion domain where the main action corresponding to the first utterance element and the component slot corresponding to the second utterance element extracted from the extractor are matched to each other is detected, and detect a candidate domain where a main action corresponding to the first utterance element and a component slot corresponding to the second utterance element are matched to each other, when the expansion domain is not detected.

The detecting the expansion domain may use at least one binary classifier to detect at least one domain from among the candidate domain, where the main action corresponding to the first utterance element and the component slot corresponding to the second utterance element are matched to each other, and the expansion domain.

The determining the final domain may understand a user's intention from at least one of the extracted first and second utterance elements and the detected candidate domain to determine whether or not a context initialization occurred, and when it is determined that a context initialization has not occurred, determines at least one candidate domain from among the detected candidate domains as a final domain based on a previous domain, the previous domain being the domain determined from the user's previous utterance voice before receiving the user's utterance voice signal.

The determining the final domain may incorporate determining the previous domain as the final domain when one candidate domain of among the detected candidate domains and the previous domain correspond to each other.

The determining the final domain may determine the candidate domain related to the first utterance element representing the user action from among at least one candidate domain related to the detected expansion domain as the final domain, when it is determined that the detected candidate domain and the previous domain are different from each other or the context initialization has occurred.

The determining the final domain may determine all the detected candidate domains as the final domain when the detected candidate domains includes main action information corresponding to the first utterance element representing the user action.

The determining the final domain may determine the candidate domain selected by the user as the final domain or arbitrarily select one candidate domain of among the detected candidate domains to determine as the final domain, when the detected candidate domain includes the main action information corresponding to the first utterance element representing the user action.

The determining the final domain may determine the candidate domain corresponding to the previous domain from among the detected candidate domains as the final domain, when there are a plurality of previous domains, one of the plurality of previous domains and one of the candidate domains correspond to each other, and the candidate domain includes main action information corresponding to the first utterance element representing the user action.

As aforementioned, according to various exemplary embodiments, the voice recognition apparatus in a dialogue type system may determine a domain appropriate to the user's intention in consideration of various number of cases regarding the user's utterance voice and may provide response information on the user's utterance voice based on the determined domain.

DETAILED DESCRIPTION

Certain exemplary embodiments are described in higher detail below with reference to the accompanying drawings.

FIG. 1is a view of a dialogue type system, according to an exemplary embodiment.

As illustrated inFIG. 1, a dialogue type system includes a voice recognition apparatus100and a display apparatus200. The voice recognition apparatus100receives a user's utterance voice signal (hereinafter referred to as an “utterance voice”) through a display apparatus200, and determines which domain the received user's utterance voice belongs to. Next, the voice recognition apparatus100may generate response information on the user's utterance voice based on a dialogue pattern of the determined domain. However, the present disclosure is not limited thereto, and the voice recognition apparatus100and the display apparatus200may be embodied as one apparatus, and may collect utterance voice of a user, determine a domain to which the collected utterance voice of the user belongs to, and generate response information on the utterance voice of the user based on the dialogue pattern of the determined domain.

The display apparatus200may be a smart TV, but this is merely an exemplary embodiment, and thus may be embodied as various electronic devices such as a mobile phone/smart phone, desktop PC, notebook, navigation system etc. The display apparatus200transmits the collected user's utterance voice to the voice recognition apparatus100. As mentioned before, the voice recognition apparatus100determines the domain to which the user's utterance voice, received from the display apparatus200, belongs to, generates response information on the user's utterance voice based on the determined dialogue pattern of the domain to which the user's utterance voice, received from the display apparatus200, belongs to, and transmits the generated response information to the display apparatus200. Accordingly, the display apparatus200may output the response information, received from the voice recognition apparatus100, through a speaker or display on a screen.

For example, when an utterance voice “show me an action movie!” is received from the user, the voice recognition apparatus100analyzes the received utterance voice and detects at least one candidate domain regarding the utterance voice. More specifically, the voice recognition apparatus100may extract a first utterance element which represents a user action “show me” and a second utterance element which represents an object “action movie” from the utterance voice “show me an action movie!”. Next, the voice recognition apparatus100may determine the user's intention “search_program(genre)” based on the extracted first and second utterance element, and determines whether or not there is an expansion domain which includes “search_program(genre)” based on a predetermined hierarchical domain module. Herein, the expansion domain is a virtual domain predetermined as a superior concept of the candidate domain for determining the expansion domain as the final domain.

When it is determined that the expansion domain including “search_program(genre)” is video contents, the voice recognition apparatus100detects a subordinate domain to which the expansion domain corresponding to “search_program(genre)” belongs to. When the subordinate candidate domain belonging to the video contents expansion domain is a TV program and VOD, the voice recognition apparatus100may detect the candidate domain corresponding to the TV program and VOD.

When such a candidate domain is detected, the voice recognition apparatus100may determine at least one candidate domain as the final domain based on a previous domain, predetermined according to previous dialogue contents and detected candidate domain.

For example, when the previous domain, predetermined according to previous dialogue contents and detected candidate domain, is VOD, the voice recognition apparatus100may determine the candidate domain corresponding to VOD as the final domain. When such a final domain is determined, the voice recognition apparatus100transmits information on the pre-extracted first and second utterance element and information on the predetermined final domain to the external apparatus (not illustrated). The external apparatus (not illustrated) may generate response information related to the user's utterance voice within the predetermined final domain corresponding to VOD and transmit the generated response information to the voice recognition apparatus100. The voice recognition apparatus100may then transmit the response information received from the external apparatus (not illustrated) to the display apparatus200. However, the present disclosures is not limited thereto, and when the candidate domain corresponding to VOD is determined as the final domain, the voice recognition apparatus100may generate response information regarding the user's utterance voice based on the pre-extracted first and second utterance element within the determined final domain, and transmit the generated response information to the display apparatus200. Accordingly, the display apparatus200may then output the response information received from the voice recognition apparatus100through the speaker or display the response information on the screen.

Herein below is a detailed explanation on the voice recognition apparatus100which provides response information to the user's utterance voice in a dialogue type system according to the present disclosure.

FIG. 2is a block diagram of a voice recognition apparatus according to an exemplary embodiment.

As illustrated inFIG. 2, the voice recognition apparatus100includes a communicator110, a voice recognizer120, an extractor130, a domain determiner140, a controller150, and a storage unit160.

The communicator110performs data communication with the display apparatus200and receives the user's utterance voice collected by the display apparatus200. In addition, the communicator110may perform communication with an external apparatus (not illustrated) and may transmit information on the determined final domain to generate response information on the user's utterance voice and information on the utterance element extracted from the user's utterance voice. The communicator110may include various communication modules such as a short distance wireless communicator module (not illustrated), etc. Herein, a short distance wireless communication module is a module for performing communication with an external device located nearby, according to a short distance wireless communication method such as Bluetooth, zigbee, etc.

Besides the above, mobile communication modules which access the mobile communication network and perform communication according to various mobile communication standards such as 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), etc. may be further included in the communicator110.

In addition, the communicator110may perform communication with a web server (not illustrated) through an Internet network, and transmit various search keywords to the web server (not illustrated) and receive web search results according thereto. Herein, a search keyword may include various keywords such as weather related keywords (for example, a name of area, a temperature, a rainfall probability, etc.), and contents related keywords (for example, a movie title, a movie release date, a music title, a singer, etc.). The various keywords may be pre-stored in the storage unit160.

The voice recognizer120recognizes the user's utterance voice received from the display apparatus200through the communicator110and converts the recognized utterance voice into text. According to exemplary embodiments, the voice recognizer120may use a speech-to-text (STT) algorithm to convert the user's received utterance voice to text. When the user's utterance voice is converted into text through the voice recognizer120, the extractor130extracts an utterance element from the user's utterance voice converted into text. More specifically, the extractor130may extract the utterance elements from the text, converted from the user's utterance voice, based on a corpus table pre-stored in the storage unit160. Herein, the utterance element is a keyword, within the user's utterance voice, for performing an operation that the user requested, and may be classified as a first utterance element which represents a user action and a second utterance element which represents major features. For example, in the case of the user's utterance voice “show me an action movie!”, the extractor130may extract the first utterance element which represents the user action “show me!” and the second utterance element which represents the object “action movie”.

When the first and second utterance elements are extracted, the domain determiner140detects the expansion domain related to the first and second utterance elements based on the hierarchical domain model, predetermined and stored in the storage unit160. Next, the domain determiner140determines at least one candidate domain related to the detected expansion domain as the final domain. Herein, the hierarchical domain model, predetermined and stored in the storage unit160, is a domain model hierarchically consisting of a candidate domain of a lowermost concept where a main action corresponding to the first utterance element extracted per utterance voice and a component slot corresponding to the second utterance element are matched to each other, and a virtual expansion domain predetermined as a superior concept of each candidate domain.

Therefore, the domain determiner140may detect an expansion domain related to the first and second utterance element based on such a hierarchical domain model, and when such an expansion domain is detected, the domain determiner140may determine at least one candidate domain related to the detected expansion domain as the final domain.

The controller150controls overall operations of each configuration of the voice recognition apparatus100. Especially, the controller150may control the communicator110to transmit information on the pre-extracted first and second utterance elements on the final domain determined through the determiner140. Therefore, the external device may generate response information related to the user's utterance voice within the predetermined final domain and transmit the generated response information to the voice recognition apparatus100, and the controller150may transmit the response information received from the external device to the display apparatus200through the communicator110. However, the present disclosure is not limited thereto, and the controller150may itself generate response information on the user's utterance voice based on the pre-extracted first and second utterance elements within the final domain determined by the domain determiner140and transmit the generated response information to the display apparatus200.

The hierarchical domain model, predetermined and stored in the storage unit160, may consist of a hierarchical structure as inFIG. 3.

FIG. 3is an exemplary view of a hierarchical domain model for detecting a candidate domain according to an exemplary embodiment.

As illustrated inFIG. 3, the hierarchical domain model for detecting a candidate domain consists of a standardized hierarchy structure showing the relationship between the user's intention and the processable domain. The hierarchical domain model may consist of a hierarchy structure having a tree shape including an uppermost hierarchy level, middle hierarchy level, and lowermost hierarchy level etc. The uppermost hierarchy level is a theme node corresponding to the virtual expansion domain predetermined as the uppermost concept corresponding to the candidate domain, which is the area generating response information on the user's utterance voice. The middle hierarchy level is a theme node regarding the virtual expansion domain predetermined as the subordinate concept to the uppermost concept. In addition, the lowermost hierarchy level is a theme node corresponding to the candidate domain predetermined as the subordinate concept to the middle concept.

For example, as illustrated inFIG. 3, at the lowermost hierarchy level, each candidate domain, TV program10, VOD20and TV device30may be predetermined. In such a case, at the middle hierarchy level, an expansion domain (TV channel40), which is the middle concept of each of the candidate domains (TV program10and TV device30), and an expansion domain (Video Content50), which is the middle concept of each of the candidate domains (TV program10and VOD20) may be predetermined.

That is, in the lower most hierarchy level of the hierarchical domain model, a candidate domain for determining an area for generating response information on each utterance voice of the user may be predetermined, and in the middle hierarchy level, an expansion domain of a middle concept which includes at least two candidate domains from among the candidate domains predetermined as the lowermost concept may be predetermined. In addition, in the uppermost hierarchy level, an expansion domain of an uppermost concept including all candidate domains predetermined as the lowermost concept may be predetermined. Each domain predetermined per hierarchy, as mentioned above, may be predetermined as a main action for determining each domain and each component slot per main action matches each other as inFIG. 4.

FIG. 4is an exemplary view illustrating a main action predetermined in the domain included in the hierarchical domain model and a component slot per main action according to an exemplary embodiment, andFIG. 5is an exemplary view illustrating a main action predetermined in all the domains included in the hierarchical domain model and a component slot per main action according to an exemplary embodiment.

As illustrated inFIG. 4, for example, the domain regarding the TV program10may be a candidate domain predetermined in the lowermost hierarchy level as explained inFIG. 3above. The candidate domain regarding the TV program10may include the component slot11corresponding to the second utterance element representing the object. Furthermore, the candidate domain regarding the TV program10may include a user action script12where the main action corresponding to the first utterance element regarding the user action and the component slot11are matched to each other. As illustrated, the candidate domain regarding the TV program10may include a user action script12such as “play_program(channel_name, channel_no, genre, time, title)”, “search_program(channel_name, channel_no, genre, time, title)”. Therefore, each domain per hierarchy included in the hierarchical domain model may include a user action script generated by combining different main actions and a component slot per main action.

Thus, the domain determiner140may detect at least one candidate domain which includes a user action script where a main action and a component slot corresponding to the first utterance element and second utterance element respectively, extracted from the extractor130, are matched to each other based on such a predetermined hierarchical domain model. For example, with reference toFIG. 5, the extractor130may extract the first utterance element “show me!” and the second utterance element “(title of broadcasting program)” from the text regarding the utterance voice “show me (title of broadcasting program)”.

In this case, the domain determiner140determines whether or not the user action script generated by combining the main action corresponding to the first utterance element “show me!” and the component slot corresponding to the second utterance element “(title of broadcasting program)” exists in the expansion domain. In the aforementioned example, the main action corresponding to the first utterance element “show me!” may be “play_program”, and the component slot corresponding to the second utterance element “(title of broadcasting program)” may be the “title”. Therefore, the user action script corresponding to the “show me (title of broadcasting program)” may be “play_program(title)”. Therefore, the domain determiner140determines whether or not the user action script is included in the expansion domain. When it is determined that the user action script regarding “play_program(title)” is included in the expansion domain of the middle hierarchy, the domain determiner140may detect the domain Video Content (50) as the expansion domain.

When an expansion domain related to the first and second utterance elements extracted from the user's utterance voice is not detected, the domain determiner140may detect the candidate domain related to the first and second utterance elements.

For example, the extractor130may extract the first utterance element “find me!” and second utterance elements “action movie” and “right now” from the text corresponding to the utterance voice “find me an action movie on air right now”. When the first and second utterance elements are extracted, the domain determiner140determines whether or not the user action script generated by combining the main action corresponding to the first utterance element “find me” and component slots corresponding to each second utterance element “right now” and “action movie” corresponding to the first utterance element exists in the expansion domain. In the aforementioned example, the main action corresponding to the first utterance element may be “search_program”, and component slots corresponding to each second utterance element “right now” and “action movie” may be “time” and “genre”, respectively. Therefore, the user action script corresponding to “find me an action movie on air right now” may be “search_program(time, genre)”.

Therefore, the domain determiner140determines whether or not such a user action script is included in the expansion domain. When it is determined that the user action script regarding “search_program(time, genre)” is not included in the expansion domain, the domain determiner140may detect a candidate domain including “search_program(time, genre)”, corresponding to the user action script, included in a plurality of candidate domains belonging to the lowermost hierarchy. When the candidate domain including “search_program(time, genre)” is a TV program10, the domain determiner140may detect the domain TV program10as the candidate domain.

When the candidate domain is detected, the domain determiner140may determine whether or not to determine the detected candidate domain as the final domain based on the previous domain. When detecting the expansion domain based on the main action and the component slot corresponding to the first utterance element and second utterance element respectively, extracted from the user's utterance voice, it is possible to detect the expansion domain by classifying the domain per hierarchy level belonging to the hierarchical domain model as inFIGS. 6 and 7.

FIG. 6is a first exemplary view which classifies the domain per hierarchy level, belonging to the hierarchical domain model, in the domain determiner according to an exemplary embodiment.

As illustrated inFIG. 6, the domain determiner140may include a multi classifier141and an expansion domain interpreter142. The multi classifier141recognizes the user's utterance voice per hierarchy level, belonging to the hierarchical domain model, and detects at least one candidate domain related to the first and second utterance elements extracted from the converted text. In addition, the expansion domain interpreter142determines the domain belonging to the superior concept of the candidate domain detected from the multi classifier141as the expansion domain.

As in the aforementioned example, when the first utterance element “find me!” and the second utterance element “action movie” are extracted from the user's utterance voice “find me an action movie!”, the multi classifier141may detect the candidate domain belonging to the lowermost hierarchy level, TV program10and VOD20, which include the user action script generated by combining the main action corresponding to the first utterance element and the component slot corresponding to the second utterance element. As mentioned above, when the candidate domains, TV program10and VOD20are detected, the expansion domain interpreter142determines whether or not there is an expansion domain having a superior concept with each candidate domain, TV program10and VOD10. When it is determined that each candidate domain of TV program10and VOD20consists of a hierarchy with the common superior concept with a domain “Video Content50”, the expansion domain interpreter142may determine the domain “Video Content50” as the expansion domain.

FIG. 7is a second exemplary view classifying the domain per hierarchy belonging to the hierarchical domain model in the domain determiner according to another exemplary embodiment.

As illustrated inFIG. 7, the domain determiner140may include a binary classifier143and a domain combiner144. There may be as many number of binary classifiers as the candidate domains in the lowermost hierarchy level belonging to the hierarchical domain model. That is, when the number of candidate domains of the lowermost hierarchy level, belonging to the hierarchical domain model, is n, the domain determiner140may include binary classifiers143-1-143-n. Therefore, the binary classifiers143-1-143-ncorresponding to the candidate domain of the lowermost hierarchy level, belonging to the hierarchical domain model, may detect the candidate domain related to the first utterance element and second utterance element extracted from the user's utterance voice and the domain of the superior hierarchy level corresponding to the candidate domain. Therefore, the domain combiner144may combine the domains detected through each binary classifier143-1-143-nand determine the candidate domain.

For example, when the first utterance element “find me!” and the second utterance element “action movie” are extracted from the user's utterance voice “find me an action movie!”, only the binary classifier143-2and143-3, corresponding to the candidate domain TV program10and VOD20including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element “action movie”, may detect the domain per hierarchy level.

That is, it is possible to detect the candidate domain regarding the TV program10of the lowermost hierarchy level, the expansion domain corresponding to the TV channel40, video channel50having a middle hierarchy consisting of a superior concept and the expansion domain corresponding to the root60of the uppermost hierarchy including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element. In addition, the binary classifier143-3may detect the candidate domain regarding the VOD20of the lowermost hierarchy, the expansion domain corresponding to the video channel50of the middle hierarchy consisting of the superior concept and the expansion domain corresponding to the root60of the uppermost hierarchy including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element. As such, when the domain per hierarchy is detected through the binary classifier143-1,2, . . . , the domain combiner144may determine the repeated domain of the detected domains as the expansion domain.

Accordingly, when the expansion domain related to the first and second utterance elements extracted from the user's utterance voice is detected, the domain determiner140may detect all domains of subordinate concept related to the pre-detected expansion domain based on the previously determined domain (hereinabove referred to as domain), and determine at least one candidate domain of all candidate domains detected as the final domain.

In addition, the domain determiner140may understand the user's intention from at least one of the first and second utterance elements corresponding to the user's utterance voice extracted from the extractor130and the pre-detected candidate domain. For example, when the second utterance elements “right now” and “action movie” are detected from the user's utterance voice “find me an action movie on air right now”, the domain determiner140may understand the user's intention from the component slot corresponding to the second utterance elements “right now” and “action movie”.

As such, when the user's intention is understood, the domain determiner140determines whether or not a context initialization has occurred from the understood user's intention. When it is determined that a context initialization has not occurred, the domain determiner140may determine at least one candidate domain of a plurality of candidate domains pre-detected as the final domain based on the previous domain, the previous domain being the domain determined from the user's previous utterance voice right before receiving the user's utterance voice.

More specifically, when at least one candidate domain of a plurality of pre-detected candidate domains correspond to the previous domain pre-stored in the storage unit160, the domain determiner140may determine the previous domain as the final domain. Herein, the previous domain is the domain determined from the user's previous utterance voice right before receiving the user's utterance voice. Such a previous domain may be a TV program10, and the currently detected candidate domains may be a TV program10and VOD20. In this case, the domain determiner140determines that the current user's utterance voice is being input within the category related to the previous utterance voice. Therefore, the domain determiner140does not determine the candidate domain as the final domain, but may maintain the TV program10, which is the previous domain, as the final domain.

However, the present disclosure is not limited thereto, and the domain determiner140may determine the candidate domain corresponding to the previous domain of the pre-detected plurality of candidate domains as the final domain.

When it is determined that the pre-detected plurality of candidate domains are different from the previous domain or a context initialization has occurred, the domain determiner140determines the candidate domain related to the first utterance element representing the user action from among at least one candidate domains related to the pre-detected expansion as the final domain. More specifically, the domain determiner140determines the candidate domain including the main action information corresponding to the first utterance element representing the user action from among the detected plurality of candidate domains as the final domain. Herein, when all main action information corresponding to the first utterance element representing the user action is included in the detected plurality of candidate domains, the domain determiner140may determine all pre-detected domains as the final domain.

For example, when the pre-detected expansion domain is an expansion domain regarding video content, the candidate domain regarding the expansion domain regarding the video content may be a domain regarding the TV program and VOD. The previous domain determined based on the previous utterance voice may be the TV program and VOD. That is, when the currently detected candidate domain and previous domain all correspond, the domain determiner140may determine the domain correspond to the TV program and VOD which is the previous domain as the final domain. When the candidate domain is a domain correspond to the TV program and VOD, and the previous domain is a domain correspond to the VOD, the domain determiner140may determine the VOD domain which is the previous domain as the final domain.

The candidate domain may be the domain correspond to the TV program and VOD, and the previous domain may be the domain correspond the TV apparatus. That is, when the detected candidate domain and the previous domain are different from each other, the domain determiner140determines that a domain conversion has occurred. Therefore, the domain determiner140may determine the candidate domain including the main action information corresponding to the first utterance element representing the user action from among the first and second utterance elements extracted from the user's utterance voice as the final domain.

For example, when the detected candidate domain is a domain related to a TV program and VOD, and the main action information corresponding to the first utterance element extracted from the user's utterance voice is a “search_program” searching contents, the domain determiner140determines whether or not the detected candidate domains includes the main action information. When the main action information “search_program” is included in a candidate domain corresponding to the TV program and VOD, the domain determiner140may determine all pre-detected candidate domains, that is, the domains related to the TV program and VOD as the final domain.

When the pre-detected plurality of candidate domains include the main action information corresponding to the first utterance element extracted from the user's utterance voice, the domain determiner140may determine the candidate domain selected by the user from among the detected plurality of candidate domains as the final domain or arbitrarily select one candidate domain from among the plurality of candidate domains and determine that as the final domain.

For example, the pre-detected candidate domain may be a domain related to a TV program and VOD, and the main action information corresponding to the first utterance element extracted from the user's utterance voice may be specific contents “play_program”. In this case, when the TV program and VOD related domains are all selected as the final domain and different response information may be generated regarding the user's utterance voice. That is, the response information generated in response to the user's utterance voice within the candidate domain corresponding to the TV program and the response information generated in response to the user's utterance voice within the candidate domain corresponding to the VOD may be different from each other. Therefore, the domain determiner140may determine one of the two selected by the user from among the TV program and VOD as the final domain.

There may be a plurality of previous domains, and one previous domain from among the plurality of previous domains and one candidate domain from among the pre-detected plurality of candidate domains may correspond to each other. In this case, the domain determiner140determines whether or not the candidate domain corresponding to the previous domain includes main action information corresponding to the first utterance element. When it is determined that the candidate domain corresponding to the previous domain includes the main action information corresponding to the first utterance element, the domain determiner140may determine the candidate domain corresponding to the previous domain as the final domain.

When one candidate domain, from among the pre-detected plurality of candidate domains, is determined as the final domain, the controller150may transmit information regarding the first and second utterance elements extracted from the user's utterance voice and information regarding the predetermined final domain to the external apparatus (not illustrated) through the communicator110. Therefore, the external apparatus (not illustrated) generates response information related to the user's utterance voice within the predetermined final domain and transmits the generated response information to the voice recognition apparatus100. Accordingly, the controller150may transmit the response information received from the external apparatus to the display apparatus200through the communicator110. However, the present disclosure is not limited thereto, and when the final domain is determined through the domain determiner140, the controller150may itself generate response information on the user's utterance voice based on the first and second utterance elements pre-extracted within that determined final domain, and transmit the generated response information to the display apparatus200through the communicator110.

Herein, the response information includes response message information corresponding to the user's utterance voice. Therefore, the controller150may receive response information including the response message information in a text format, being output on the display apparatus200, from the external apparatus or generate the response information internally. When such response information is received from the external apparatus or generated internally, the controller150transmits the response information to the display apparatus200through the communicator110. Accordingly, the display apparatus200may output the response message information, included in the response information received from the voice recognition apparatus100, on the screen in a text format or output as audio through the speaker.

A detailed explanation was provided above on each configuration of the voice recognition apparatus100providing response information to the user's utterance voice in a dialogue type system according to the exemplary embodiments. A detailed explanation on a method of providing response information corresponding to the user's utterance voice in a dialogue type voice recognition apparatus100according to exemplary embodiments will be provided below.

FIG. 8is a flowchart of a method for providing response information corresponding to the user's utterance voice in a voice recognition apparatus of a dialogue type system according to an exemplary embodiment.

As illustrated inFIG. 8, the voice recognition apparatus100receives a user's utterance voice signal (hereinafter referred to as an “utterance voice”) collected from the display apparatus200. When the user's utterance voice is received from the display apparatus200, the received user's utterance voice is recognized and converted into a text (S810). According to an exemplary embodiments, the voice recognition apparatus100may use an STT (Speech to Text) algorithm and convert the user's received utterance voice into text. When the user's utterance voice is converted into text, the voice recognition apparatus100extracts the first utterance element representing the user action and the second utterance element representing the object from the user's utterance voice converted into text (S820). For example, in the case of the user's utterance voice “find me an action movie!”, the voice recognition apparatus100may extract the first utterance element representing the user action “find me!” and the second utterance element representing the object “action movie”.

Next, the voice recognition apparatus100detects the expansion domain related to the first and second utterance elements extracted from the user's utterance voice based on the hierarchical domain model predetermined and stored (S830). Herein, the hierarchical domain model is a domain model hierarchically consisting of a candidate domain where the main action corresponding to the first utterance element extracted per utterance voice and the component slot corresponding to the second utterance element are matched to each other, and a virtual expansion domain predetermined as a superior concept of each candidate domain.

The hierarchical domain model may consist of a tree shaped hierarchical structure including an uppermost hierarchy level, a middle hierarchy level and a lowermost hierarchy level. The uppermost hierarchy level is a theme node regarding the virtual expansion domain predetermined as an uppermost concept related to the candidate domain which is an area generating the response information on the user's utterance voice, and the middle hierarchy level is a theme node predetermined as a subordinate concept of the uppermost concept. In addition, the lowermost hierarchy level is a theme node regarding the candidate domain predetermined as a subordinate concept of the middle concept. That is, the candidate domain belonging to the lowermost hierarchy level of the hierarchical domain model is predetermined as a candidate domain of the lowermost concept corresponding to a basic keyword, and the virtual expansion domain belonging to the middle hierarchy level may be predetermined as a domain of a middle concept including two or more candidate domains from among the plurality of candidate domains predetermined as a lowermost concept. In addition, the virtual expansion domain belonging to the uppermost hierarchy level may be determined as a domain of an uppermost concept including all candidate domains predetermined as a lowermost concept.

Each domain predetermined per hierarchy level may be predetermined as the main action for determining each domain and the component slot are matched to each other as illustrated inFIG. 4. That is, each domain predetermined per hierarchy may include the user action script generated by combining the main action corresponding to the first utterance element representing the user action and the second utterance element representing the object. As illustrated inFIG. 4, the candidate domain regarding the TV program10belonging to the lowermost hierarchy may include the user action script12such as “play_program(channel_name, channel_no, genre, time, title)”, “search_program(channel_name, channel_no, genre, title)”. Therefore, as illustrated inFIG. 4, each domain predetermined per hierarchy, that is each domain per hierarchy included in the hierarchical domain model may include the user action script generated by combining a different main action and component slot per main action as illustrated inFIG. 5.

Therefore, the voice recognition apparatus100may detect at least one candidate domain including the user action script, generated by combining the main action and the component slot corresponding to the first utterance element and second utterance element, respectively, extracted from the user's utterance voice based on the predetermined hierarchical domain model, and detect the expansion domain belonging to the superior concept of the corresponding candidate domain from the detected candidate domains.

According to an exemplary embodiment, the voice recognition apparatus100uses the multi classifier to detect at least one candidate domain related to the first and second utterance elements extracted from the user's utterance voice per hierarchy belonging to the hierarchical domain model. When at least one candidate domain is detected as such, the voice recognition apparatus100determines the domain belonging to the superior concept of the detected candidate domain as the expansion domain. For example, the voice recognition apparatus100may detect the candidate domain TV program10and VOD20including “search_program(genre)” which is the user action script generated by combining the main action corresponding to the first utterance element “find me!” and component slot corresponding to the second utterance element “action movie” from the user's utterance voice. As such, when each candidate domain (TV program10and VOD20) is detected, the voice recognition apparatus100determines whether or not there is an expansion domain having a superior concept common with each of the detected candidate domain. When it is determined that each of the candidate domain TV program10and VOD20consists of a hierarchy having a common superior concept with the domain “Video Content(50)”, the voice recognition apparatus100may determine the domain “Video Content(50)” as the expansion domain.

According to another exemplary embodiment, the voice recognition apparatus100uses at least one binary classifier to detect at least one domain, from among the candidate domains, related to the first utterance element and the second utterance elements extracted from the user's utterance voice and the expansion domain. Herein, the number of binary classifiers may be as many as the number of candidate domains in the lowermost hierarchy level belonging to the hierarchical domain model. That is, when there is n number of candidate domains in the lowermost hierarchy level belonging to the hierarchical domain model, the voice recognition apparatus100uses n number of binary classifiers to detect the candidate domain related to the first and second utterance elements extracted from the user's utterance voice and the expansion domain of the superior hierarchy level related to the corresponding candidate domain. Next, the voice recognition apparatus100may determine the candidate domain by combining the domains detected through each binary classifier. For example, when the first utterance element “find me!” and the second utterance element “action movie” are extracted from the user's utterance voice, only the binary classifier corresponding to the candidate domains TV program and VOD including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element “action movie”, from among each binary classifier corresponding to the TV program, VOD and TV device, may detect the domain per hierarchy.

That is, the voice recognition apparatus100may detect the candidate domain corresponding to the TV program10of the lowermost hierarchy level including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element “action movie”, the expansion domain corresponding to the TV channel and video content in the middle hierarchy level of the superior concept incorporating the corresponding candidate domain, and the expansion domain corresponding to the root in the uppermost hierarchy level. In addition, the voice recognition apparatus100may detect the candidate domain regarding the VOD in the lowermost hierarchy level including the user action script generated by combining the main action corresponding to the first utterance element “find me!” and the component slot corresponding to the second utterance element “action movie”, the expansion domain corresponding the video channel in the middle hierarchy level consisting of a superior concept incorporating the corresponding candidate domain, and the expansion domain corresponding to the root of the uppermost hierarchy level. As such, when the domain per hierarchy is detected through the binary classifier corresponding to each candidate domain TV program and VOD, the voice recognition apparatus100may determine the repeated domain from among the detected domains as the expansion domain.

When the expansion domain related to the first and second utterance elements extracted from the user's utterance voice is detected, the voice recognition apparatus100may detect all candidate domains of a subordinate concept related to the pre-detected candidate domain based on the previously determined domain (herein below referred to as the previous domain) and detect at least one of the plurality of detected candidate domains as the final domain (S840, S850). Next, the voice recognition apparatus100transmits the information regarding the determined final domain and the information regarding the first and second utterance elements extracted from the user's utterance voice to the external apparatus (not illustrated) (S860). Accordingly, the external apparatus (not illustrated) may generate response information related to the user's utterance voice within the predetermined final domain and transmit the generated response information to the voice recognition apparatus100. However, the present disclosure is not limited thereto and the voice recognition apparatus100may itself generate the response information on the user's utterance voice based on the first and second utterance elements extracted within the predetermined final domain. Herein, the response information may include a response message information corresponding to the user's utterance voice. Therefore, when such response information is received from the external apparatus or generated internally, the voice recognition apparatus100transmits the corresponding response information to the display apparatus200. Accordingly, the display apparatus200may output the response message information included in the response information received from the voice recognition apparatus100on the screen in a text format or output as audio through the speaker.

Herein below is a detailed explanation of a method for determining at least one candidate domain from among the plurality of candidate domains as the final domain in the aforementioned voice recognition apparatus100.

FIG. 9is a flowchart of a method for determining one of the candidate domains extracted in the voice recognition apparatus as the final domain according to an exemplary embodiment.

As illustrated inFIG. 9, the voice recognition apparatus100understands the user's intention from the first and second utterance elements extracted in operation S820and the pre-detected candidate domain. As mentioned before, when the second utterance elements “right now” and “action movie” are detected from the user's utterance voice “find me an action movie on air right now”, the voice recognition apparatus100may understand the user's intention from the detected component slots “right now” and “action movie”.

When the user's intention is understood, the voice recognition apparatus100determines whether or not a context initialization occurred based on the understood user's intention (S910). When it is determined that a context initialization has not occurred, the voice recognition apparatus100determines one candidate domain from among the pre-detected plurality of candidate domains as the final domain based on the previous domain. More specifically, when the one candidate domain from among the pre-detected plurality of candidate domains correspond to the pre-stored previous domain, the voice recognition apparatus100maintains the previous domain as the final domain (S920, S930). Herein, the previous domain is the domain determined from the user's previous utterance voice right before receiving the user's utterance voice. For Example, when such a previous domain is a TV program and the currently detected candidate domain is TV program and VOD, the voice recognition apparatus100determines that the user's current utterance voice is being input within the category related to the previous utterance voice. Therefore, the voice recognition apparatus100does not determine the pre-detected plurality of candidate domains as the final domain, but may maintain the TV program, which is the previous domain, as the final domain.

When it is determined that the pre-detected plurality of candidate domains are different from the previous domain or a context initialization has occurred, the voice recognition apparatus100obtains main action information corresponding to the first utterance element representing the user intention (S940). Next, the voice recognition apparatus100determines whether or not main action information corresponding to the first utterance element is included in the pre-detected plurality of candidate domains (S950). When it is determined that main action information corresponding to the first utterance element is included in the at least one candidate domain of among the plurality of candidate domains, the voice recognition apparatus100determines the candidate domain including the main action information, corresponding to the first utterance element, as the final domain (S960).

There may be a plurality of previous domains, and one previous domain from among the plurality of previous domains and one candidate domain from among the pre-detected plurality of candidate domains may correspond to each other. In this case, the voice recognition apparatus100determines whether or not the candidate domain corresponding to the previous domain includes the main action information corresponding to the first utterance element. When it is determined that the candidate domain corresponding to the previous domain includes the main action information corresponding to the first utterance element, the voice recognition apparatus100may determine the candidate domain corresponding to the previous domain as the final domain.

For example, when the detected candidate domain is a domain regarding a TV program and VOD, and the main action information corresponding to the first utterance element extracted from the user's utterance voice is “search_program” for searching contents, the voice recognition apparatus100determines whether or not the detected candidate domain includes the main action information “search_program”. When the candidate domain regarding the TV program and VOD includes the main action information “search_program”, the voice recognition apparatus100may determine the candidate domain regarding the TV program and VOD as the final domain.

When it is determined that all of the pre-detected plurality of candidate domains do not include the main action information, the voice recognition apparatus100may determine the candidate domain selected by the user from among the detected plurality of candidate domains as the final domain (S970) or arbitrarily select one candidate domain of among the plurality of candidate domains and determine that as the final domain.

For example, the pre-detected candidate domain may be a domain related to a TV program and VOD, and the main action information corresponding to the first utterance element extracted from the user's utterance voice may be “play_program”. In this case, when the candidate domain regarding the TV program and VOD is selected as the final domain, different response information may be generated regarding the user's utterance voice. Therefore, the voice recognition apparatus100may determine one candidate domain selected by the user of among the candidate domains regarding the TV program and VOD as the final domain.

As such, when at least one candidate domain from among the pre-detected plurality of candidate domains is determined as the final domain, the voice recognition apparatus100may generate the response information on the user's utterance voice within the determined final domain and output the generated response information. Accordingly, the display apparatus200may confirm the response message regarding the utterance voice of oneself by displaying the response information received from the voice recognition apparatus100or by outputting through the speaker.

The method of providing response information on the user's utterance voice in the voice recognition apparatus according to various exemplary embodiments may be embodied as a program code and be provided in each server or device stored in a non-transitory computer readable medium.

A non-transitory computer readable medium is a computer readable medium where data may be stored semi-permanently and not a medium where data is stored for a short period of time such as a resistor, cache, memory, etc. More specifically, the aforementioned various applications or programs may be stored in a non-transitory computer readable medium such as a CD, DVD, hard disk, Blu-ray, USB, memory card, ROM, etc.