interview

Tiago Soares da Costa Mr Tiago Soares da Costa from the Institute of Systems and Computer Engineering, Technology and Science at the University of Porta, Portugal, talks to Electronics Letters about the paper ‘Predictive multi-view content buffering applied to interactive streaming system’, page 837. My main field of research is adaptive multimedia streaming, focused on immersive experiences provided by dynamic content selection and seamless interaction mechanisms. I have worked on this field at INESC TEC since 2012, specialising on MPEG-DASH and multi-view content streaming. On the other hand, Artificial Intelligence (AI) is another field of research which I have studied throughout my PhD, focusing on the introduction of “intelligent mechanisms” for pre-emptive buffering. Regarding the most interesting aspects, combining these two different fields represents the most challenging task for me, due to their specific requirements which aren't always compatible between them. Despite this, bringing them together will allow us to provide better Quality of Experience (QoE) for end users: systems capable of “understanding” user preferences will be able to pre-select the most suitable point of interest within a scene, optimising network resources beforehand. At INESC TEC, I have worked on several technologies capable of providing immersive experiences to users, such as Microsoft Smooth Streaming and MPEG-DASH. However, even with the latter, streaming of multi-view content has highlighted several problems: standard limitations for representation of multiple viewing angles, network throughput requirements, view switching latencies and presence of graphical artefacts. To further aggravate these issues, the introduction of interactive mechanisms (e.g. head tracking) has presented us with new issues, such as latencies between initial view requests and content playback. Based on previous work, a research line was devised for my PhD, focusing on analysis of historical and real-time data from user viewing preferences, for pre-emptive buffering purposes. In the Electronic Letters paper, the focus was on the introduction of predictive view buffering mechanisms, heavily based on Machine Learning (ML) techniques. By relying on head tracking data seamlessly provided by users, coupled with Recurrent Attention Models (RAM), the proposed mechanism should be able to estimate viewing angle selections and pre-emptively request appropriate views. To achieve this goal, relevant data such as current head positioning, viewing angle, focus of attention and network conditions were used. Through this work, our major goal is to improve QoE by optimising network resources, reduce view switching latencies and address visual artefacts faced during streaming of DASH-based multi-view content. The significance from this work is directly related to its dependence on AI and, particularly, ML techniques. While current approaches make use of real-time user data to deliver new content, no additional effort is established for further analysis, understanding and prediction of any possible user interaction. By introducing AI, we are delivering a new, dynamic approach to adaptive multimedia streaming. Combining historical viewing profiles with real-time user data will allow multimedia solutions to evolve throughout time, improving correlations between previous user selections and future content requests. Regarding the short-term impact of this work, the inclusion of predictive mechanisms within multimedia scenarios should prove the feasibility of using ML techniques, despite their demanding dataset and computational requirements. In the longer term, I hope that this work represents an initial step towards bringing better immersive experiences to end users, relying on multiple fields of research and considerable amounts of data (collected from interaction devices and complementary sensors) to establish credible and coherent user viewing profiles. Being an on-going work at INESC TEC, our approach is already being tested with convolutional neural networks (CNNs) to evaluate the overall efficiency of the proposed algorithm. From the baseline results gathered throughout this initial stage, our next steps will focus on the research of Recurrent Attention Models (RAMs) and their integration within the architecture of our own solution. Direct comparison between these latter models and initial CNNs will provide us with valuable data, from which we will be able to draw substantiated conclusions. As a side note, due to the specific requirements from this work, a new dataset may also be proposed and evaluated for usage and potential public release. Adaptive multimedia streaming is an ever-evolving field of research: its development is directly correlated to technical requirements, formats and forms of service being presented every day to consumers. Research in this field in 2012 focused on Full HD and stereoscopic streaming, both of which have been surpassed by 360° content and 8 K. Likewise, ML has also contributed with numerous improvements on multiple fields of research throughout latter years, fostered by affordable distributed, GPU-based computing. In the next 10 years, a symbiotic relationship between both areas is expected, as AI becomes a common slogan for the everyday user. For example, while on-demand services, such as Netflix and Amazon Prime, have expanded throughout the years with remarkable content, immersive experiences are still lacking on home-based scenarios. As a matter of fact, multiple topics within multimedia streaming are still open for improvement, representing the perfect “excuse” for integration of AI.