# An Ontology for Spatio-Temporal Media Management and an Interactive Application
## 1 Introduction  
Most of the visual and audio content that we interact with daily, such as movies, TV programs, and music, is communicated using digital media. After being recorded as digital media on a recording device, it is transmitted, distributed, and played back on a playback device to provide visual and auditory experiences. Although many standards and specifications [1; 2; 3; 4; 5] exist for digital media, video and audio media representations are highly restricted because they are created for playback on widely used playback devices. The basic playback style of video is a single rectangular screen, and most of the audio content is stereo audio or channel-based audio with fewer than ten channels [6]. Although, there are various factors in the representation of digital media, such as video resolution, pixel representation, refresh rate, audio code length, sampling rate, and the number of channels, this representation is only a variation of the basic style of the rectangular screen plus channel-based audio. In addition, when playing back video and audio content encoded on digital media, normal playback devices may fine-tune the picture and sound quality, but they do not make any changes to the content. Users can only passively view video and listen to audio content that has been edited according to the creator's intention. Meanwhile, the use of VR images using computer graphics and 360\" images, spatial sound reproduction, and sound reproduction using binaural recordings [7], among other innovations, has been spreading, and such audio-visual content with a high sense ofrealism is becoming increasingly popular [8; 9; 10; 11]. Typical VR applications allow users to view images from any viewpoint in a virtual space while the localization of the sound changes according to the viewpoint to provide a highly realistic viewing experience. To reproduce images from arbitrary viewpoints, it is necessary to use computer graphics synthesis or a combination of many video camera images [12]. At the same time, it is also important to reproduce sound according to the user's position and posture in order to provide a high sense of realism [13]. Since video and audio content is not processed or edited in a general playback environment, the editor produces video and audio content as a finished product optimized to most effectively reproduce the editorial intent in a general playback environment. On the other hand, to reproduce an interactive viewing experience, video and audio must be processed. In this sense, the playback device can be considered responsible for the final stage of the editing process. Therefore, digital media content that is one step behind the finished product and includes the information necessary for this operation is required. In addition to video and audio, displays for various senses such as touch, force, smell, and taste are also being attempted to reproduce a higher sense of presence [14; 15]. For this purpose, it would be convenient to have a system that can handle video and audio media as well as data collected by various sensors in a unified manner. There is still no digital media framework that can comprehensively meet the above requirements. Therefore, we believe that a new digital media management framework is needed that is suitable for providing interactive and highly realistic media playback environments that are expected to become popular in the near future. Hence, we established a Software Defined Media (SDM) consortium ([[https://sdm.wide.ad.jp/](https://sdm.wide.ad.jp/)]([https://sdm.wide.ad.jp/](https://sdm.wide.ad.jp/)) (accessed on 1 June 2023)) in 2014 to promote research on Internet-based video and audio media [16]. The main research objective of the SDM consortium is to develop new ways of representing digital media by managing digital media in object units and controlling them with software, and at the same time, build an architecture that can realize flexible and highly applicable systems and services.