Patent ID: 11935543
Assignee: OPENSTREAM INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 10:
11. A multimodal conversational dialogue system comprising:
one or more sensors configured to obtain, by a system comprising one or more computing devices, one or more multimodal inputs from a user, wherein a portion of a multimodal input includes an utterance, the utterance including at least one of text, speech, audio, visual, body language and postures, gestures, facial expressions, eye gaze, and lip reading, wherein the sensors include at least a camera, a motion sensor, and a microphone;
a multimodal semantic parser configured to perform, by the system, semantic parsing and multimodal fusion of the one or more multimodal inputs to determine a goal of the user, wherein
the semantic parsing and multimodal fusion includes generating, by the system, a logical expression based on the one or more multimodal inputs, and utilizing a meta-logic interpreter to perform an inference over the logical expression, and
the logical expression is at least one of a first-order logic formula or logical modality operators applied to the logical expression;

a dialogue manager configured to:
generate system-generated plans based on actions in a database that include multimodal dialogue state information;
determine user plans based on the actions in the database;
identify a portion that forms obstacles in the system-generated plans and the user plans, respectively;
determine a plan based on the portion that forms obstacles; and
generate, by the system, a dialogue with the user based on the plan,
wherein the dialogue includes one or more system-generated utterances; and

output, by the system, the one or more system-generated utterances.