Sharma, R., Pavlovic, V., and Huang, T. 1998. All these previous efforts on multimodal integration have concentrated primarily on semantic representations and incorporation of new input technologies, rather than on the statistical integration process that defines a multimodal system architecture. We found this metric to provide better results than the joint rank criterion expressed as the weighted sum of individual ranks. A distinct feature of the IGDS system is its bimodal human computer interaction. Queries can be formulated and refined and the retrievals can be browsed using speech recognition or graphical input.
Audio feedback is provided by speech synthesis.
The paper is organized as follows. Some modalities like speech and lip movements, are more closely tied than others, such as speech and hand gestures. The fusion of such different modalities can be explored at different levels of integration. We found this metric to provide better results than the joint rank criterion expressed as the weighted sum of individual ranks. A distinct feature of the IGDS system is its bimodal human computer interaction.