Patent Description:
A speech dialogue service allows a voice-based interaction between a user and a voice agent (e.g. a device or system supporting the speech dialogue service). Currently, such a speech dialogue service is offered or supported by many interfacing devices (e.g. devices providing a user interface), including various types of portable terminal. However, often the speech dialogue service fails to sufficiently utilize any type of input/output interface other than voice, thus limiting expression, convenience and flexibility. Further, as the length of a voice input increases, a possibility of errors in voice recognition may also increase.

In view of the above issues, multi-modal interface techniques based on various interfaces have been introduced, see for example <CIT>.

It is an aim of certain embodiments of the present invention to address, solve, mitigate or obviate, at least partly, at least one of the problems and/or disadvantages associated with the related art.

Certain embodiments of the present invention aim to provide at least one advantage over the related art, for example at least one of the advantages described below.

According to the invention, there are provided a system as set forth in claim <NUM> and a method as set forth in claim <NUM>.

Other aspects, advantages, and salient features of the present invention will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the present invention.

The above and other aspects, and features and advantages of certain exemplary embodiments and aspects of the present invention will be more apparent from the following detailed description when taken in conjunction with the accompanying drawings, in which:.

Throughout the drawings, it should be noted that the same or similar reference numbers may be used to denote the same or similar elements, features, and structures.

The following description of exemplary embodiments of the present invention, with reference to the accompanying drawings, is provided to assist in a comprehensive understanding of the present invention, as defined by the claims. The description includes various specific details to assist in that understanding, but these are to be regarded as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the various embodiments described herein may be made without departing from the scope of the present invention.

Detailed descriptions of well-known functions, processes, operations, features, elements, structures and constructions may be omitted for clarity and conciseness, and to avoid obscuring the subject matter of the present invention.

The terms and words used in the following description are not limited to the bibliographical meanings, but are merely used to enable a clear and consistent understanding of the present invention. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the present invention is provided for illustration purpose only and not for the purpose of limiting the present invention, as defined by the appended claims.

Throughout the description and claims of this specification, the words "comprise", "include" and "contain" and variations of the words, for example "comprising" and "comprises", means "including but not limited to", and is not intended to (and does not) exclude other features, elements, components, integers, steps, processes, operations, characteristics, properties and/or groups thereof.

Throughout the description and claims of this specification, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a signal" includes reference to one or more of such signals.

Features, elements, components, integers, steps, processes, operations, functions, characteristics, properties and/or groups thereof described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example described herein unless incompatible therewith.

Throughout the description and claims of this specification, language in the general form of "X for Y" (where Y is some action, process, activity, operation or step and X is some means for carrying out that action, process, activity, operation or step) encompasses means X adapted, configured or arranged specifically, but not exclusively, to do Y.

In this disclosure, the term "multi-modal" refers to the use of two or more input/output mechanisms, including input/output mechanisms of two or more different types, in an interface between a human being (e.g. a user) and a machine (e.g. a device or system). For example, a multi-modal input may be made using one or more of voice recognition, a keyboard, a keypad, a touch sensor, an image capture, and the like. Additionally, the expression "voice input" refers to an input based on a user's voice received through a microphone. In certain embodiments, a voice input may be regarded as a normal or typical type of interfacing used in a speech dialogue service. On the other hand, the expression "non-voice input" refers to any suitable type of input other than a voice input. An input mode switch may be performed to switch between a first input mode, for receiving inputs of a first type (e.g. voice inputs) and a second input mode, for receiving inputs of a second type. The input mode switch may be performed according to an input of the first type in the first input mode.

<FIG> is a block diagram illustrating a system for supporting a speech dialogue service according to an exemplary embodiment of the present invention.

Referring to <FIG>, the system for supporting a speech dialogue service includes an interfacing device <NUM> and a server <NUM>.

The interfacing device <NUM> is a device that supports a speech dialogue service between a user and a voice agent. The interfacing device <NUM> has the ability to communicate with the server <NUM> via a network in order to derive a response to a user input from a voice agent. A voice agent may refer to an entity (e.g. a virtual entity) that executes or supports voice interaction with a user in a speech dialogue service.

The interfacing device <NUM> may include an input unit <NUM>, a control unit <NUM>, a communication unit <NUM>, and an output unit <NUM>, but is not limited thereto. The skilled person will appreciate that certain embodiments of the present invention may comprise one or more units in addition to those shown in <FIG>. In certain embodiments, one or more of the units shown in <FIG> may be omitted, and in certain embodiments, one or more of the units shown in <FIG> may be replaced with units having equivalent or corresponding functions.

The input unit <NUM> is configured to perform a function of receiving a user input in a speech dialogue service. The input unit <NUM> may include a voice input unit <NUM> for receiving a user's voice input, and a non-voice input unit <NUM> for receiving a non-voice input (i.e. an input other than a voice input). The voice input unit <NUM>, which may comprise a microphone for example, may receive a user's voice input and convert it into a voice input signal. The non-voice input unit <NUM> may receive a user's non-voice input, which, according to the claimed invention, includes image data. Other non-voice inputs are not covered by the claimed invention, for example text, music, and the like. Text may be received as a user input, for example through a keyboard input, a touch sensor gesture input, a text copy input, and the like. Images may be received as a user input, for example through a camera function, an image selection from a list of stored images, an image copy input, and the like. Music may be received as a user input, for example while being played through a music player. As will be understood by those skilled in the art, any other input mode may be applied as an input interfacing type, and any suitable type of input may be received by the input unit <NUM>, for example when operating in a corresponding input mode.

The control unit <NUM> controls the execution of a speech dialogue service. Specifically, the control unit <NUM> may process a user input signal received from the input unit <NUM> and then transmit the user input signal to the output unit <NUM> and the server <NUM>.

In the case that a voice input is received, the control unit <NUM> may convert a voice input signal into text and then output the text to the output unit <NUM> to display a user input. Additionally, the control unit <NUM> may transmit, as a request signal, the voice input signal, or a corresponding signal (e.g. a text signal), to the server <NUM>. When a result of dialogue recognition is received from the server <NUM> in response to the request signal, the control unit <NUM> may execute a particular function corresponding to the received result of dialogue recognition and then output a response to the output unit <NUM>. The result of dialogue recognition may comprise, for example, information related to the user input signal, or text extracted therefrom. For example, the information may comprise a response to a request for information or a service in which the user input signal or the text extracted therefrom (e.g. provided in the request signal) comprises a parameter of the request.

Further, in the case that a non-voice input is received, the control unit <NUM> may include a text extraction unit <NUM> for extracting text from a non-voice signal corresponding to the non-voice input. For example, if an image or music file (a music file is not in accordance with the claimed invention) containing metadata is inputted, the text extraction unit <NUM> may recognize or extract text of metadata contained in the image or music file and perform a filtering or selection process for selecting a part of the text (e.g. essential text). The filtering process may be performed according to a predefined rule or a user's selection. For example, when metadata comprises description information (e.g., one or more keywords), a saving or modification time (e.g. a time the file was last saved or modified), a timestamp, copyright holder information, and any other file information, the description information may be predefined to be selected as the selected part of the text (e.g. essential text) in the filtering or selection process. The preceding example is, however, exemplary only and not to be considered as a limitation of the present invention.

In the case of an image file containing characters, the text extraction unit <NUM> may recognize or extract text contained in the image file through image processing (e.g. an Optical Character Recognition (OCR) process). A specific part of the extracted text (e.g. a part desired by a user) may be selected in a filtering or selection process. The filtering process may be performed according to a predefined rule or a user's selection.

The control unit <NUM> may output the extracted text of the non-voice input to the output unit <NUM> to display a user input. Additionally, the control unit <NUM> may transmit, as a request signal, the extracted text of the non-voice input to the server <NUM>. When a result of dialogue recognition is received from the server <NUM> in response to the request signal, the control unit <NUM> may execute a particular function corresponding to the received result of dialogue recognition and then output a response to the output unit <NUM>.

After the transmission of a request signal to the server <NUM>, the control unit <NUM> may receive from the server <NUM> a message that inquires whether to apply an associated content provider (e.g. whether it is desired to use a service or content provided by the associated content provider). The associated content provider may comprise, for example, an entity providing content or a service related to the extracted text. Determination of an associated content provider will be described below.

The communication unit <NUM> is configured to support a communication between the interfacing device <NUM> and the server <NUM> through a wired or wireless network.

The output unit <NUM> may receive a dialogue content signal (i.e., a processed user input signal and a response signal of a voice agent), from the control unit <NUM> and then output the dialogue content signal while a speech dialogue service is executed. For example, the output unit <NUM> may include a display <NUM> and a speaker <NUM>. In the case that a visual signal is not required for a dialogue content signal, a response of a voice agent may be outputted by means of sound only. In this case, the display <NUM> may be turned off and the speaker <NUM> may be turned on. Alternatively, both the speaker <NUM> and the display <NUM> may be turned on so as to display as text a user input and a response of a voice agent. If a user input is received as a non-voice input, the display <NUM> may be turned on to display text of the extracted non-voice input. In addition, the output unit <NUM> may display various functions and contents offered in a speech dialogue service.

The server <NUM> is configured to receive a user input (e.g., a request signal), from the interfacing device <NUM>, to derive a response to the request signal from a voice agent, and to transmit a derived result to the interfacing device <NUM>. When a voice signal is received from the interfacing device <NUM>, the server <NUM> may perform a function to convert the received voice signal into text.

The server <NUM> may include a dialogue manager <NUM>, which derives a response of a voice agent from a database on the basis of text corresponding to a user input. For example, if text "Google" is recognized, the dialogue manager <NUM> may determine that a user desires to perform a "Google search", and thereby may derive a result "access Google homepage" as a system response. This dialogue result may be transmitted to the interfacing device <NUM>, and then the interfacing device <NUM> may perform access to a homepage of Google.

The server <NUM> may further include an associated Content Provider (CP) determining unit <NUM> that determines an associated content provider on the basis of text corresponding to a user input. According to the claimed invention, the type of the text is determined, and then an associated content provider is derived from the type of the text. The type of the text may be determined, for example, according to a weighted value of each category of words contained in the text. For example, if it is determined that a word contained in the text is an address type, a map service provider may be derived or determined as an associated content provider. Additionally, the server <NUM> may inquire whether to apply an associated content provider. For example, the server <NUM> may request the interfacing device <NUM> to check the applicability of a map service (e.g. check whether use of the map service is desired). If a user accepts the applicability of an associated content provider, the dialogue manager <NUM> may derive and offer a response of a voice agent on the basis of the associated content provider.

<FIG>is a flow diagram illustrating a method for executing a speech dialogue service at a system according to an exemplary embodiment of the present invention.

Referring to <FIG>, at operation <NUM>, the interfacing device <NUM> may select an input mode. A voice input may be set as a default input mode of a speech dialogue service, and thereafter a switch of input modes may be performed, for example through a voice command. For example, if a voice command "open camera" is received, the interfacing device <NUM> may switch an input mode to a mode for receiving an image input through a camera.

At operation <NUM>, the interfacing device <NUM> may receive a user input on the basis of the selected input mode. Then, at operation S203, the interfacing device <NUM> may extract text, for example comprising content of dialogue, from the received user input, as previously discussed above in relation to <FIG>.

Thereafter, at operation <NUM>, the interfacing device <NUM> may transmit the extracted text to the server <NUM>.

At operation S205, the server <NUM> may determine an associated content provider from the received text. At operation S206, the server <NUM> may transmit to the interfacing device <NUM>, a request for checking the applicability of the associated content provider. Then, at operation S207, the interfacing device <NUM> may transmit to the server <NUM> a result of checking the applicability of the associated content provider.

At operation S208, the server <NUM> may derive a response of a voice agent based on the received text, and optionally the received checking result for the applicability of the associated content provider. Then, at operation S209, the server <NUM> may transmit to the interfacing device <NUM> a result of dialogue recognition based on the derived response.

At operation S210, the interfacing device <NUM> may execute a response on the basis of the received dialogue recognition result and offer it to a user.

<FIG>is a flow diagram illustrating a method for executing a speech dialogue service at an interfacing device according to an exemplary embodiment of the present invention.

Referring to <FIG>, the interfacing device <NUM> receives a user input at operation S301 and then determines at operation S302 whether the received user input is a voice input. In the case of a voice input, the interfacing device <NUM> transmits to the server <NUM> a voice signal, or a signal comprising text converted from the voice, at operation S303.

In the case of a non-voice input, the interfacing device <NUM> extracts text from the non-voice input at operation S304 and then transmits a signal comprising the extracted text to the server <NUM> at operation S305.

<FIG> is a flow diagram illustrating a detailed process of text extraction operation illustrated in <FIG> according to an exemplary embodiment of the present invention.

Referring to <FIG>, in detail the operation S304 is illustrated. First, it is determined whether a non-voice input comprises an image input at operation S401. If the non-voice input comprises an image input, the interfacing device <NUM> may recognize text of metadata contained in an image file and then perform a filtering process at operation S402, and/or may recognize text by performing OCR from the image file and then perform a filtering process at operation S403. If only one of the operations S402 and <NUM> is performed, text extracted through the performed operation may be offered. However, if both operations S402 and <NUM> are performed, text extracted through one of the operations may be offered.

Referring back to <FIG>, at operation S306, the interfacing device <NUM> checks the applicability of an associated content provider determined by the server <NUM> and then transmits a checking result to the server <NUM>. At operation S307, the interfacing device <NUM> receives a result of dialogue recognition from the server <NUM> and then, based on the received result, executes a response to offer it to a user. For example, the interfacing device <NUM> may output information related to the result of dialogue recognition to a user, or executes a function using information related to the result of dialogue recognition.

<FIG> is a table illustrating an example of determining an associated content provider at a server according to an exemplary embodiment of the present invention.

Referring to <FIG>, in the case that text extracted by the interfacing device <NUM> is transmitted to the server <NUM>, the associated content provider determining unit <NUM> may determine the type of the extracted text by analyzing words contained in the text. For example, as illustrated in <FIG>, if the extracted text is "<NUM> Rue du Fouarre <NUM> Paris, France" <NUM>, this text contains the name of a country, the name of a city, and the name of a street, and therefore a text type may be determined as a location (address) <NUM>. As another example, if the extracted text is "Samsung, Sennheiser, Ferrari" <NUM>, a text type may be determined as a company name <NUM>. As a further example, if the extracted text is "Dark knight, Spider man" <NUM>, a text type may be determined as a content (movie) name <NUM>.

If a text type is a location (address) <NUM>, a map service <NUM> may be offered or determined as an associated content provider. If a text type is a company name <NUM>, a search web <NUM> may be offered or determined as an associated content provider. If a text type is a content (e.g. movie) name <NUM>, a content (e.g. movie) service application <NUM> may be offered or determined as an associated content provider.

If a user accepts the applicability of an associated content provider offered to the interfacing device <NUM> (e.g. the user confirms the associated content provider that has been offered or determined), the server <NUM> may request the interfacing device <NUM> to display thereon text-related information in the associated content provider. For example, the interfacing device <NUM> may offer, as a response to text extracted from a user input, text-related information in the associated content provider to a user. Text-related information in the associated content provider may comprise information related to the text provided by the associated content provider.

<FIG> and <FIG> are screenshots illustrating a process of executing a speech dialogue service at an interfacing device according to exemplary embodiments of the present invention.

Referring first to <FIG>, the interfacing device <NUM> may display a dialogue between a user and a voice agent on the screen. In an embodiment, some icons, for example an icon (on/off) for controlling an on-off status of a speech dialogue service, an icon (mic. ) for controlling the activation of a microphone, an icon (SEL) for selecting an input mode, etc. may be displayed on the screen.

When a voice agent offers a comment <NUM> notifying a speech dialogue service as illustrated in screenshot <NUM>, a user may offer, as a response, a voice input <NUM> to a voice agent as illustrated in screenshot <NUM>. In this case, a user inputs a voice "Navigate to this place". In reply, a voice agent offers a response <NUM> corresponding to a user's voice input as illustrated in screenshot <NUM>. In this case, a voice agent offers a response "Where is it?".

Referring to <FIG>, a switch of input modes and a response of an interface according to an image input is illustrated.

As illustrated in screenshot <NUM>, a user may desire to offer an image input in response to a request of a voice agent and may therefore require a switch of input modes. The switch of input modes may be performed, for example, using a voice command. In this case, an input mode may be switched through a voice input <NUM> "Open camera and read this image".

Then, as illustrated in screenshot <NUM>, the interfacing device <NUM> may capture an image by executing a camera and extract a text part from the captured image. For example, text contained in the captured image may be recognized through an OCR process, and a desired text part may be selected among the recognized text through a filtering process. As illustrated in screenshot <NUM>, a text part <NUM> extracted from an image input may be displayed as a user response on the screen. Thereafter, a voice agent outputs a response to the extracted text part <NUM>. Since the extracted text part <NUM> indicates a location (address), a voice agent may offer a map service for representing the location (address) as illustrated in screenshot <NUM>.

In the embodiments described above, a server <NUM> is used. However, the skilled person will appreciate that, in certain embodiments which are not covered by the claims, some or all of the functions of the server <NUM> may be performed by another entity. For example, in certain embodiments, some or all of the functions of the server <NUM> may be performed by the interfacing device <NUM>. For example, in the example illustrated in <FIG> and <FIG>, if the interfacing device stores suitable map data, then the functions of the server <NUM> may be performed by the interfacing device <NUM>. In the case that all of the functions of the server <NUM> are performed by another entity, the server <NUM> may be omitted.

As fully discussed hereinbefore, a speech dialogue service may be offered through various input interfaces other than a voice.

It will be appreciated that various embodiments of the present invention can be realized in the form of hardware, software or any combination of hardware and software.

Any such software may be stored in a non-transitory or non-volatile computer readable storage medium. The non-transitory computer readable storage medium stores one or more programs (e.g. software modules), the one or more programs comprising instructions, which when executed by one or more processors in an electronic device, cause the electronic device to perform a method according to the present invention.

Any such software may be stored in the form of volatile or non-volatile storage such as, for example, a storage device like a Read Only Memory (ROM), whether erasable or rewritable or not, or in the form of memory such as, for example, Random Access Memory (RAM), memory chips, device or integrated circuits or on an optically or magnetically readable medium such as, for example, a Compact Disk (CD), Digital Versatile Disc (DVD), magnetic disk or magnetic tape and the like.

It will be appreciated that the storage devices and storage media are various embodiments of non-transitory machine-readable storage that are suitable for storing a program or programs comprising instructions that, when executed, implement various embodiments of the present invention. Accordingly, various embodiments provide a program comprising code for implementing a system, apparatus or method according to any claim, aspect, embodiment or example described herein, and a non-transitory machine-readable storage storing such a program.

Claim 1:
A system comprising an electronic device (<NUM>) for supporting a speech dialogue service and a server (<NUM>), the electronic device comprising:
an input unit (<NUM>) configured to receive a user input including a voice input (<NUM>) and a non-voice input (<NUM>), the non-voice input including image data acquired from a camera or selected from stored images;
a communication unit (<NUM>) configured to communicate with the server;
a control unit (<NUM>) configured to:
execute an interface for the speech dialogue service,
receive (S202, S301) the user input through the executed interface,
when the received user input is the voice input:
transmit (S303), as a request signal to the server, at least one of the voice input or a text extracted from the voice input,
when the received user input is the non-voice input:
extract (S203, S304) a text from the image data of the non-voice input,
transmit (S204, S305), as the request signal to the server, the text extracted from the image data of the non-voice input,
receive (S209) a result of dialogue recognition in response to the request signal from the server, and
provide (S210, S307) a particular operation corresponding to the received result of dialogue recognition; and
an output unit (<NUM>) configured to output a result of the particular operation corresponding to the received result of dialogue recognition,
wherein the control unit is further configured to:
receive (S206), from the server, a request for checking whether to apply an associated content provider determined by the server, wherein the server determines a type (<NUM>, <NUM>, <NUM>) of the text extracted from the user input and determines (S205) the associated content provider which provides the dialogue recognition from a plurality of content providers based on the determined type of the text extracted from the user input, check (S306) whether to apply the associated content provider determined by the server, and
transmit (S207) a result of the checking to the server