Patent ID: 11935543
Assignee: OPENSTREAM INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 0:
1. A multimodal conversational dialogue method comprising:
obtaining, by a system comprising one or more computing devices, one or more multimodal inputs from a user via sensors configured to obtain one or more multimodal inputs including an utterance, the utterance containing at least one of text, speech, audio, visual, body language and postures, gestures, facial expressions, eye gaze, and lip reading,
wherein the sensors include at least a camera, a motion sensor, and a microphone, and
wherein a portion of a multimodal input includes the utterance;

performing, by the system, semantic parsing and multimodal fusion of the one or more multimodal inputs to determine a goal of the user,
wherein the semantic parsing and multimodal fusion includes generating, by the system, a logical expression based on the one or more multimodal inputs, and
wherein the logical expression is at least one of a first-order logic formula or logical modality operators applied to the logical expression;

performing, by the system, an inference over the logical expression by at least recursively decomposing the logical expression into constituent parts and utilizing meta-logic interpreters to interpret the constituent parts,
wherein the meta-logic interpreters include at least a proof meta-logic interpreter and an assertion meta-logic interpreter,
wherein the proof meta-logic interpreter is configured to interpret that a proof of a first side of the constituent parts is contingent upon establishing the proof of a second side of the constituent parts, and
wherein the assertion meta-logic interpreter is configured to interpret that an assertion of the first side of the constituent parts is contingent upon establishing an assertion of the second side of the constituent parts; and

generating, by the system, a dialogue with the user based on a database that includes multimodal dialogue state information,
wherein the multimodal dialogue state information includes goals and at least one of beliefs, preferences, intentions, obligations, expectations, norms, rules, domain knowledge, personal knowledge, actions, and data, and
wherein the dialogue includes one or more system-generated utterances;

outputting, by the system, the one or more system-generated utterances; and
determining, by the system, an emotional state of the user based on the dialogue.