Brand-new Sport Viewing Style by Immersive Live Experience(ILE) and Its Standardization Status
Hideo Imanaka

Senior Expert Engineer
Business Intelligenceand AI center,NTT Advanced Technology

Yoshihide Tonomura

Senior Research Engineer
NTT Service Evolution
Laboratories,
Nippon Telegraph andTelephone Corporation

1. Introduction
For large scale sports events like the Olympics andParalympics, and the World Cups of Soccer and Rugby, beyond
being at the venue itself, it has become popular to be able to seelive coverage from many countries around the world. Sports
events can also be viewed remotely, through theatre distribution
services called public viewings or live viewings, giving a feelingof togetherness with other spectators similar to actually beingat the event venue. However, current services such as video
distribution do no more than display the image on a large screen,
and it is difficult to achieve immersiveness or highly-realistic
sensations remotely. There is much anticipation for more realistic
public viewings and live viewings, which will provide immersive
experiences more like actually being at the venue, for the 2020
Tokyo Olympics and Paralympics.
Recently emerging technologies utilizing head mounted
displays (HMD), such as virtual reality (VR), augmented reality(AR), and a combination of these called mixed reality (MR),are starting to be used to provide a variety of highly immersive
entertainment. These technologies could potentially be used toprovide a more immersive sports viewing experience, like actually
being in the crowd and cheering on the players. Although wearing
an HMD to enjoy such a presentation individually would also begood, it may be difficult to achieve the sense of unity of watching
and cheering together with other spectators, as at a public viewing.
Wearing an HMD also makes it difficult to enjoy other aspects
essential to watching and cheering, such as having a beer at a
baseball game.
Immersive Live Experience (ILE) technologies have beenstudied to realize more realistic, immersive public and live
viewings. Beyond just high-definition video and high-quality
multi-channel audio, 5th generation (5G) mobile and high-speed
broadband communication networks could be used to transmit
a variety of data with low latency, such as positions and sizesof players and production data (lighting, vibration, etc.), and toprovide new styles of sports viewing through ILE.
This article introduces examples of new sports viewing styles
using ILE, and the status of international standardization efforts
toward realizing such systems at ITU-T SG16.

Kiyoshi Tanaka

Senior Research Engineer,Supervisor
NTT Service Evolution
Laboratories,
Nippon Telegraph andTelephone Corporation

the game at the same time, and can also provide benefits thatcannot be realized with live viewing, using production effects.
NTT Laboratories’ Kirari! immersive telepresence technology
uses multiple ultra-high resolution video images and high-quality
sound with efficient compression and synchronous transmission toreconstruct a pseudo-3D video representation of the 3D space. It
has demonstrated ability to maintain a highly-realistic sensation ofpresence, even in environments such as public viewings.
2.2 Proof of Concept with table tennis viewing
In a table tennis demonstration, the system was used toreconstruct the space of the actual venue very realistically using
pseudo-3D images of the players mapped onto a real table-tennis
table at the remote site, and positioning the sound images at thelocations of the corresponding objects. Figure 1 is an example withthe table-tennis match projected into the space using ILE. In thisexample, a real table and a real master of ceremonies (MC) are
shown with pseudo-3D images of the players mapped over them.
2.3 Live coverage of Karate
In another example, live coverage of a karate competition
was given by extracting images and sound in real time andtransmitting them synchronously to the remote location. Figure 2
shows an example of remote live coverage distribution of a karate“Enbu” competition using ILE. Images of the performance by
the player at the competition venue are extracted in real time
(separating the image of the player from the background), images
and sound from multiple cameras are sent synchronously to theremote viewing site, and all are presented as live coverage. A
pseudo-3D image of the karate athlete was shown at the viewing
location, together with simultaneous images centered on the face
■ Figure 1: An example with the table-tennis matchprojected into the space using ILE

2. Brand-new sport viewing style through ILE
2.1 Sharing the excitement
The demand for access to global-scale sporting events throughpublic viewings is growing. This style of viewing promises newsynergistic effects due to having like-minded spectators watching

New Breeze Spring 20185■ Figure 2: An example of remote live coverage distribution of a karate “Enbu” competition using ILE

and from behind, providing production effects not possible whenviewing at the real venue.

3. The need for and objectives of ILE
standardization
The goal of the brand-new viewing style using ILE is toprovide a realistic viewing experience, like actually being there,
from anywhere in the world. In addition to distributing theexperience to multiple locations simultaneously, ILE can also
connect viewers with each other at the viewing venue, providing
an even stronger sense of togetherness. When implementing
such a system, event organizers are usually different fromoperators of public viewing venues, so an interface specification forinternational media transmission is needed. Spatial informationin addition to the video data from the event location is needed toreproduce the event using pseudo-3D video at the viewing venue
as shown in Figure 3.
■ Figure 3: Example
technologies for realizing ILE

6New Breeze Spring 2018

MPEG and other standardization bodies have recently
studied standards related to VR, and parts of these standards,
such as image formats, could be applicable for ILE. However,ILE requirements differ from those of VR with an HMD in thatlarge numbers of people are watching. As such, it is essential thatinternational standards for ILE be created, including standards
for international media transmission, so that sports events
can be viewed all over the world. Existing MPEG and otherstandard specifications for media transmission, compression andformats could be used, but international standards on how to use
MPEG specifications (profiles) and interfaces for internationaltransmission of media will need to be created by the ITU-T inorder to realize ILE.

4. Status of ILE standardization at the ITU-T
4.1 Overall ILE standardization trends at SG16
NTT proposed establishing a new Question to study ILE at
the June 2015 meeting of SG16, and it was approved in Octoberat the World Telecommunication Standards Assembly, which
is a general assembly meeting of the ITU-T. Study of ILE
standardization officially began with Question 8 under Study
Group 16 (Q8/16) at the January 2017 meeting of SG16. Notethat Dr. Hideo Imanaka (one of the authors) is the Rapporteur for
Q8/16.
With support from Japan, South Korea, China, and Brazil,
Q8/16 is studying draft Recommendations for ILE service
scenarios, ILE requirements, functional frameworks, and MPEGMedia Transport (MMT) profiles for ILE. To raise awarenessof ILE, three workshops on ILE have also already been heldwith ITU-T members and others, to share information. This
has included the state of VR study at MPEG and DVB (DigitalVideo Broadcast), and prospects for immersive services at NTT,
Fraunhofer (Germany), KT (South Korea), and China Mobile.
4.2 Study of service scenarios
A draft Recommendation summarizing ILE service scenarios(H.ILE-SS) is being studied from a proposal by South Korea.
As shown in Figure 4, several scenarios have been considered,including one which is for viewing live coverage of sporting events.
Other possibilities include entertainment scenarios such as music
concerts and theatre, and telepresence scenarios such as remote
lectures and speeches.
Draft Recommendation H.ILE-SS summarizes ILE service
scenarios from Japan, South Korea, and China, and an appendix
describes concrete use cases and service concepts, including
the Kirari! from NTT. This draft Recommendation is to becompleted before the end of 2018.
4.3 Study of requirements
The definition and requirements for ILE services are
summarized in the ILE requirements draft Recommendation,
H.ILE-Reqs. ILE is defined as follows:
Immersive Live Experience (ILE): the shared viewing
experience which stimulates emotions within audiences atboth the event site and remote sites, as if the ones at remotesites wandered into substantial event site and watched actual
events in front of them, from high-realistic sensations brought
by a combination of multimedia technologies such as sensorial
information acquisition, media processing, media transport, media
synchronization and media presentation.
The current draft Recommendation indicates 13 requirements
for realizing ILE. The main essential requirements are as follows.
• The direction of sound is reproduced at the viewing venue.

■ Figure 4: Examples of ILE service scenarios• The spatial environment is reproduced at the viewing venue.
• Video, sound and spatial information must be synchronized.
• Multiple types of media, such as video, sound and spatial
data, can be processed to reproduce a virtual space at theviewing venue.
The H.ILE-Reqs draft Recommendation is to be completed
before the end of 2018.
The requirements for implementing ILE services will be
studied further in the future.
4.4 Study of functional frameworks
An overall architecture and identification of interfaces
between functional blocks are summarized in the functionalframework draft Recommendation, H.ILE-FW. The overallarchitecture of the ILE system is shown in Figure 5.
The left side of Figure 5 is the Capturing environment, withfunctionality to collect video and sound at the event venue. The
right side is Presentation, with functionality to reconstruct theexperience at the remote viewing site. These two are connectedby the Transport layer, and also placed between them are ILE
Application functions, such as media processing and signalprocessing.
Three interfaces are envisioned: between capturing functions
and the transport network, between transport network andpresentation functions, and between transport network and ILE
applications.
Draft Recommendation H.ILE-FW summarizes definitionsof the various functions in the overall architecture in Figure 5
as well as the technical elements that will be needed. This draft
Recommendation is expected to be completed in 2018.
4.5 Study of MMT profiles
One of the main technologies for ILE will be for synchronoustransmission of video, sound and environmental data. MPEG has
already created a specification for synchronous transport, called
MPEG Media Transport (MMT), and this could potentially beapplied to ILE.
The concept of synchronous transport of video, sound andspatial information using MMT is illustrated in Figure 6. With
conventional digital broadcasting using MPEG2-TS, video andsound must be inserted in the same time frame, but with MMT,
video and sound data each contain their own time information, so
they can be transmitted separately. Spatial information and stage
production information such as lighting can also be sent with time
information, making media easy to multiplex.
The purpose of draft Recommendation H.ILE-MMT
is to identify MMT functions needed for ILE services, andto specify how MMT is to be used (profiles). Work to create
the Recommendation began in September 2017 and it is to becompleted in 2019.

New Breeze Spring 20187

■ Figure 5: Overallarchitecture of the ILE system■ Figure 6: Concept of synchronous transport of video, sound and spatial information using MMT

4.6 Future directions for ILE standardization
Standardization of the basic components for ILE
is currently in progress and there are plans to create detailed
technical standards for the MMT profiles and interfaces needed
to implement international interconnectivity for ILE in thefuture. Use of standards being studied in other standardization
organizations, for technical elements such as VR, is also being
considered in order to advance standardization work efficiently.

5. Conclusion
This article has summarized the status of international
standardization for ILE, which will enable highly realistic public
viewings of large scale sporting events such as the Olympics and8New Breeze Spring 2018

Paralympics from anywhere in the world. Through ILE, newviewing styles were proposed for sports, with the goal of providing
a sense of realism and togetherness with other spectators that will
be like watching at the actual event, from anywhere in the world.
ILE itself is a new type of service, and ILE standardization
has only just begun. In order to expand the immersive service
technologies being studied in Japan globally, effort to create
concrete Recommendations for high priority items is necessary, toquickly create Recommendations for requirements and elemental
technologies needed. Use of VR technologies is also a possibility,
so it is hoped that collaboration will be strengthened with MPEG
and other SDO organizations and contribute to standardization
activities from Japan, to realize ILE as early as possible.