Some modalities like speech and lip movements, are more closely tied than others, such as speech and hand gestures. The fusion of such different modalities can be explored at different levels of integration. We found this metric to provide better results than the joint rank criterion expressed as the weighted sum of individual ranks. A distinct feature of the IGDS system is its bimodal human computer interaction. Queries can be formulated and refined and the retrievals can be browsed using speech recognition or graphical input.
Audio feedback is provided by speech synthesis.
The paper is organized as follows. Next we would analyze this task for identifying the gesture primitives and keywords which would encompass possible interaction modes. This will be followed by a study on ways of integrating the multiple modalities. It is our goal to achieve a perceptual user interface which will make it possible to perform complicated tasks with ease in an enhanced multimodal display control environment. Further feedback is an important component of any natural interaction process.