Patent ID: 8219655

Claim:
A method of associating multiple modalities in a multimodal managing system that sends content that corresponds to multiple, simultaneously connected, different modalities and receives input data sent therefrom, and provides information and services via a network based on a prescribed dialogue scenario that is synchronized among the modalities, comprising: based on a connection request from a first modality of the plurality of modalities, generating a first connection identifier and storing the correspondence between the first connection identifier and the first modality in a session information table; generating association data needed in order to associate a second modality and the first modality of the plurality of modalities, updating the session information table so that the association data and the first connection identifier correspond, generating content that includes association data, wherein the association data is combined with the content to be sent to the first modality, and then sending the generated content that includes association data to the first modality; upon receiving link request data that was sent by the second modality which is not yet synchronized to the first modality using the association data included in the content that includes association data sent to the first modality, comparing the link request data and the association data stored in the session information table, and determining whether to associate the second modality with the first modality; when it is determined to associate the second modality with the first modality, generating a second connection identifier for the connection with the second modality, and updating the session information table with the link information that establishes a correspondence between the second connection identifier and the first connection identifier; acquiring positional information of the plurality of modalities; based on the positional information of the plurality of modalities, selecting only the modality that is at a prescribed position with respect to the first modality and generating association data therefor.