Spatio-temporal browsing of multimedia presentations
Authors: Ramazan Savas Aygun
Venue:  (2003)

Emerging applications like asynchronous distant learning and collaborative engineering require organization of media streams as multimedia presentations. The browsing of presentations enables interactive surfing of the multimedia documents. We propose spatio-temporal browsing of multimedia presentations in the sense that browsing can be performed both in the spatial and temporal domain. The spatial browsing is provided by incorporation of camera controls like panning, tilting, and zooming. Panoramic images enable a kind of browsing by storing the image at high resolutions from various angles. However, the generation of high resolution sprite (mosaic) from digital video is not an easy task. Since the video data may also exist in a compressed format, new features like boundaries have to be extracted from the compressed video. We consider compressed data that is generated by Discrete Cosine Transform (DCT), which has been used in MPEG-1, MPEG-2, MPEG-4, and H263.1. Motion sensors, which are sensitive pixels to motion, are proposed to indicate the existence of motion. The temporal integration of images is performed using the histemporal filter based on the histogram of values within an interval. We also introduce a new sprite called conservative sprite where new pixels are exclusively mapped on the sprite during temporal integration phase. The sprite pyramid is introduced to handle sprite at different resolutions. To measure the quality of the sprite, a new measure called sharpness is used to estimate the blurring in the sprite. The generated sprite is used for spatial browsing. On the other hand, temporal browsing is closely related with the synchronization of different streams. The power of synchronization models is limited to the synchronization specifications and user interactions. The proposed synchronization model is an event-based model that can handle time-based actions while enabling user interactions like backward and skip. The synchronization model processes the synchronization rules based on Event-Condition-Action (ECA) rules. The synchronization model uses Receiver-Controller-Actor (RCA) scheme to execute the rules. In RCA scheme, receivers, controllers, and actors are objects to receive events, to check conditions, and to execute actions, respectively. The synchronization rules can easily be regenerated from SMIL expressions. A middle layer between the specification and the synchronization model assists the synchronization model to provide user interactions while keeping the synchronization specification minimal. We call this middle layer as middle-tier. The middle-tier for multimedia synchronization handles synchronization rules that can be extracted explicitly from the user specification and synchronization rules that can be deduced implicitly from explicit synchronization rules. The verification and correctness of schedules are also important. Model checking is a technique that automatically detects all the states that a model can enter and checks the truthness of well-formed formulas. PROMELA/SPIN tool has been used for model checking to check LTL (Linear Temporal Logic) formulas. These formulas can automatically be generated and verified.