4‑2

4 Innovation Projects Innovation Initiatives

Future‑oriented Sports Viewing Project
KOSHIBA Keiichi

KINOSHITA Shingo

With the Olympic Games new visual technologies such as color broadcasting satellite relay and high‑definition
television have been tested and put to practical use This games were held under the pandemic disaster and the challenge
was whether a live broadcast could provide the sense of presence and togetherness at the actual venues
The “Future‑oriented Sports Watching Project” in collaboration with partner companies NTT and Panasonic used
dome video streaming and Kirari! Technology to provide a live broadcast of the tension of the athletes at that moment thepower and beauty of their bodies the sense of unity between spectators and athletes and the unique Olympic atmosphereat the venues This is the first‑ever challenge to convey a sense of reality as it is in real life was undertaken (Researchedby Tokyo 2020 (cooperation of Prof Okyudo of Wakayama University an advisor to the Committee) and NTT)
Keywords：Dome image High realistic Kirari!

2．Immersive LIVE Viewing(Dome Video Streaming)

1．Introduction
This project aims to deliver experiences as if one werein the competition venue to someone who cannot visit

To achieve the vision of the Tokyo 2020 Games to be

first“the most innovative games in history” we set a target

technology that expresses the sense of presence of themeasure that would remain as a legacy after consideringvenue which cannot be expressed on a flat screen by

innovations in the field of Audio & Videothe venue

The technologies employed were

projecting it onto a dome screen and second NTTʼs

There are approximately 350 planetariums in Japanultra‑realistic communication technology which proj-

and no other country in the world has as many dome

This article

screens for the public as Japan Recently instead of thesummarizes the results of the “Future‑oriented Sports

conventional proprietary system for astronomical repre-

Viewing Project” implemented at the Olympic andsentation

Paralympic Games Tokyo 2020 (hereinafter referred toprojection system with a high‑luminance projector This

as “Tokyo 2020 Games”) with the above objectives insystem was recommended by Prof Okyudo of Wakaya-

mind

ma University for sport viewing (1)ects the players as they really are

there has been a trend to replace the

As a result of our research for various realistic image
expressions we thought that image expressions thatallow spectators to share the excitement and wonder ofthe game with each other are more promising as a new
KOSHIBA Keiichi
G1 company Inc
E‑mail kossi@genki‑1 com
KINOSHITA Shingo
NTT Human Informatics Laboratories NIPPON TELEGRAPH AND TELEPHONE CORPORATION CORPORATION
E‑mail shin kinoshita@hco ntt co jp

style of sport infection in the future than methods of

The Journal of The Institute of Electronics Information and Communication Engineers Vol 105 No 8 Supplement pp 203‑210 August 2022

the “Future‑oriented Sports Viewing Project : Immer-© 2022 The Institute of Electronics Information and Communication Engineers

individual enjoyment using VR glasses and the like
After many tests and negotiations with the support ofProf Okyudo we decided to plan promote and operate
sive LIVE Viewing”

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020

203

Figure 1

Overall Structure of the Distribution System

The concept of this project was “to have spectators
experience a sense of presence and unity each other in asafe” secure and closer location In this project the goal
was to express the speed height and strength of sport
by projecting images of competitions on a 10 m classdome screen and to express the sense of presence andunity at the venue by installing cameras in the
spectatorʼs seats We thought that people can enjoy the(a) SLR camera with a fisheye lensFigure 2games with those close to them in the domes and cheer
together with others in the venue
21

System Configuration(b) Video signal image

Single‑lens Reflex Camera with Fisheye Lens andImage

has minimized
213

Screening System

The overall configuration is shown in Figure 1 The

Originally planetariums projected point images ofsystem is divided into the following sections dependingstars on a hemispherical dome screen by means ofon the location : venue‑dedicated line‑studio‑distribu-

specialized planetarium equipment In recent years

tion platform‑general line‑screening theatre In thishowever projectors have made it possible to projectproject the key point was to ensure that the system

images over the entire dome screen so that pictures

would remain as a legacy in the future i e to combine

representing constellations and simulated images forand utilize existing infrastructure and technologies withastronomy can be projected The resolution is improving

as little operational cost as possible

year by year and facilities that can project dome imagesof 8K in diameter using multiple 4K and 8K projectors

211

Camera Systemhave been launched Currently the systems vary widely

Considering that the screen is a hemispheric dome wedepending on the facility and the manufacturer How-

decided to use a consumer‑use 4K‑resolution mirrorlessever it is possible to project fisheye images (domesingle‑lens camera with a fisheye lens The focal length

master format) with a viewing angle of 180 degreesof the lens was chosen so that the fisheye image isinscribed in a square although the maximum possible

inscribed in the vertical position of the 4K format toresolution differs depending on the systemachieve the highest resolution possible (Figure 2)

Therefore in cooperation with the manufacturers of
the planetarium systems installed in each of the212

Distribution Systemplanetarium facilities participating we proposed the

The video format used for transmission was the 16 : 9

addition of a system that can cut out and project fisheye

4K video format also used in television broadcasting

images inscribed at the top and bottom of a 16 : 9 4K

This format doesnʼt need to construct a special system

image in dome master format

In addition considering the cost ease of operation andstability of the system we proposed using a best‑effort

22

Outline of the Experiment

general optical fiber line instead of a dedicated line

The outline of the experiment is as follows

Furthermore we decided to use an existing Internet TV
platform to distribute live and recorded images Thanks

221

Schedule

to this the screening operator at each planetarium can

The screening methods include live transmission andselect the program menu in STB by remote controller

reception with video‑on‑demand (VOD) of recorded

just as if operating a television then operational problems

images As a result it was decided that the shooting

204

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

dates were from July 23 to August 8 2021 and themunicipality and screening theatre As a result a total ofscreening dates were from July 27 to August 8 2021

11 live screenings and 44 VOD screenings wereconducted All scheduled programs were filmed trans-

222

Competitions

mitted and screened without any major problems even

The selected competitions were ceremonies volley-

though communication methods and procedures forball 3×3 basketball badminton and sport climbing

troubleshooting had been prepared in advance The

Three cameras (four cameras only for the opening andfacilities of the participating planetariums all differed inclosing ceremonies) were installed at each competition

the resolution of the dome master diameter from 2K tovenue to realize the concept of providing a realistic

8K the dome diameter from 9 m to 25 6 m and the dome

feeling of being at the spectatorʼs sheets

screen from horizontal to tilted (Table 1)

Audio equipment was installed in each camera and226

signals were mixed out

Online Survey

An online survey was conducted at the screening
223

theatres under the initiative of Wakayama University to

Camera Locations

It was decided to place the cameras in the spectatorʼsclarify the realism of this experiment All screeningseats rather than on the field to reproduce the concept astheatres distributed promotional flyers to visitors andmuch as possible As a result of negotiations with theasked them to access a web page using a smartphone orinternational federations and the like it was decided toother device to answer questions via a two‑dimensionalfilm the opening ceremony volleyball badminton 3×3

barcode printed on the flyer Iida City Museum of Art

basketball and sport climbing from three to four camera

conducted its independent open‑ended questionnaire

locations in the spectator stands Furthermore for sport

survey and exit interviews and Wakayama University is

climbing in which participants climbed a high wall itplanning to conduct research and analysis in the future

was decided to install one camera close to the23

competition stage since it was confirmed from pre‑test

Results

The original plan was to hold public viewing at 11

filming that the wall height could be expressed withvenues but there was a nationwide movement to refrainclose to the stage

from public viewing due to the coronavirus disease 2019224

(COVID‑19) then resulting in 4 venues that could

Program List

We decided a competition schedule to be filmed based

attract the general public 4 venues that became limitedon considerations from the spectatorsʼ perspective such

events for related parties and 3 venues that wereas the presence of Japanese national team athletes andcanceled As a result 969 people (851 for the general

medal games and operational considerations

public and 118 for related parties) participated in thisdemonstration experiment

225

Although subjective we have analyzed and organized

Screening Theatres

the three realistic sensations obtained through thisThe final screening schedule was left to each

Table 1

List of Theatres

Location
Planetarium

Capacity

Dome

Prefecture

Seats

DiameterType

Katsushika City Museum

TOKYO

172

18 m

TiltedFuchu City Museum

TOKYO

218

23 m

Horizontal

Iida City Museum

NAGANO90

12 m

Horizontal

Fukui City Museum of Astronomy(Seiren Planet)

FUKUI

160

17 m

Horizontal

Tsukuba Expo Center

IBARAKI232

25 6 mTiltedFukuoka City Science Museum

FUKUOKA

220

25 m

Horizontal

Ishigakijima Hoshinoumi Planetarium

OKINAWA

46

9m

TiltedMiraikan (The National Museum of
Emerging Science and Innovation)

TOKYO

112

15 24 mTiltedSpecial Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020

205

experiment as below
231

Realism of SportWe believe that the most significant advantage ofimages projected on a dome screen is its expressive
power that encompasses the entire viewing angle of thespectator One of features is the expressive power for
vertical movement which is different from the ordinary
TV broadcasting
For example the athletes who dash up in sport
climbing with the fastest speed in the world and all of the
trajectory of the badminton shuttle of lob serve above

Figure 3

View of the Screening Venue (Location : The National
Museum of Emerging Science and Innovation (Miraikan) Photo courtesy of Panasonic)(© 2021‑International Olympic Committee‑All Rights Reserved)

the heads of athletes that takes over two seconds can befollowed perfectly This systemʼs ability to experiencesport as they are is a significant feature
On the other hand the ordinary 16 : 9 video can bedramatically directed to a high degree of perfection andtors it was not possible to verify the effect of the sense ofresolution by zooming and switching But the dome

unity between the cheering spectators at the competi-

screen video requires high‑performance cameras andtion venue and the cheering audiences at the screening

lenses and expensive infrastructure to deliver images oftheatrethe entire surface (hemisphere) in high‑resolution
Therefore the camera location that can deliver powerful

3．Transmission of Holographic Images of
Badminton Matches

images without its zoom function is very important It is
important to consider this point when installing cameras

We carried out technology demonstrations aimed atfor similar initiatives in the futuredelivering the experience of being at the badminton
232

venue of the Tokyo 2020 Games to those who could not

Realism of the Venue

The enjoyment of watching sport is not limited to theattend the actual event by using Kirari! ultra‑realistic

sport itself Especially in the Olympic Games competi-

communication technology to transmit holographic

tion venues are beautifully decorated without advertise-images of the event This section presents results ofments and with many flags of participating countries

demonstrations of a next‑generation immersive technol-

under the concept of “Clean Venue”

This time byogy for providing a sense of “being there” namely

setting up cameras near the spectator seats we decided

through the sense of presence and sense of unity by

to provide “memories of being there” by recording thetransmitting holographic images of the badminton

atmosphere of the entire venue the works of the event

matches
We initially planned demonstrations of live viewing bystaff and everything else (Figure 3)
Similarly by recording the sounds from the venue

inviting the general public However to prevent thewith microphones without adding the running commen-

spread of the COVID‑19

tary we could provide a surround sound experience that

demonstration and instead opened “Future‑oriented

we cancelled the public

makes the most of the 5 1 channel even though it picked

Sports Viewing Project” at the National Museum of

up various noises such as reverberations

Emerging Science and Innovation (Miraikan) (Figure
4)233

Realistic Sensation of Entertainment

For the opening and closing ceremonies and sport
presentations between games we could convey a uniquesense of scale by capturing images of the entire

31

System Configuration

Figure 5 shows the overall system configuration for
the technology demonstrations

competition venue from a wide‑angle including fire-We set up 8K cameras at the Musashino Forest Sport

works paper airplanes descending and lighting effects

Plaza the venue for the Tokyo 2020 Games badminton

However since the event was held without specta206

competitions

and transmitted video images taken

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

during the badminton matches via a 1‑Gbit/s network to
the broadcast centre

We then used the real‑time(2)

to the remotely located Miraikan museum
We set up an approximately 100‑seater spectatorastand and a full‑size court equipped with holographic

component technology of Kirari! to selectively extract

projection equipment at the remote‑viewing venue toextraction of objects with arbitrary background

only the images of players and shuttlecocks from therecreate the actual venue The extracted images oftransmitted 8K images at the broadcast centre Using

players and shuttlecocks were holographically displayed

highly realistic media synchronization technology (Ad-by presenting players in their actual positions in front

vanced MMT (MPEG Media Transport)) (3) we thenand behind the net using the birdʼs eye‑view sports

synchronously transmitted the extracted images along

viewing multi‑layer aerial image display technology As

with multiple images that included video and audioa result we were able to create a space where theprovided by the Olympic Broadcasting Services (OBS)

badminton players appeared to be actually present in the
remote‑viewing venue
32

Technical Elements

The following is a list of the technical elements thatcomprise the system described above
321

Real‑time Extraction of Objects with Arbitrary
Background

The real‑time extraction of objects with arbitrary
background is a technology for selectively extracting
only the specific objects in images such as players andshuttlecocks from the videos taken during a competition
Normally extraction of only the objects from videos
Figure 4

Kirari! Technology Demonstration Experiments of theTokyo 2020 Games Badminton Competition (© 2021‑
International Olympic Committee‑All Rights Reserved)

Figure 5requires using a green or blue background and erasing
the background color by chroma keying This technol-

Workflow of Object Extraction and Synchronous Transmission

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020207

ogy however enables extracting only the objects from()

the raw images of the competition venue in real time
without the need for any special background environment

Automatic Correction of Missing Parts in
Player Images

When the player at the back of the court overlapped

To apply this technology to the badmintonwith the net a black gap zone appeared on the player

matches we carried out the following five improve-

image Images were corrected by inferring the missing

ments on our previous system that is for less compli-

parts on the basis of the color information above andcated content

below the gap zone()

Individual Extraction of Each Player in the(

Front and Back Sides of the Court

Reproducing the playersʼ shadows appearing in the

Separately extracting the front and back players is not)

Automatic Generation of Playerʼs Shadow

actual venue and in the remote‑viewing venue enables

possible with the conventional technology To achieve

the creation of more natural player images

this we developed a deep learning model that can

extracted the shadows from the video based on thesimultaneously extract and infer front and back players

results of player‑image extraction

We

by inputting depth information simulating the badminton
court space This enabled the extraction of individual

322

players for a game like badminton where players areHighly Realistic Media Synchronization Technology

located separately in the front and back sides of the

The highly realistic media synchronization technology

court We therefore succeeded in stable and accurate(Advanced MMT) is NTTʼs proprietary technology

extraction of objects

developed by extending the media transmission standard MMT and enables the transmitting of various()

Improvement of Output‑image Resolution andFrame Rate of Playersʼ Images

We carried out multilayering of the image frames to becontinuous data (streams) such as video audio andlighting information

while maintaining their timesynchronization For this project we synchronously

processed and leveling of computing resources totransmitted multiple streams such as video of the

support images from 8K 60‑fps cameras As a resultcompetition taken at the venue audio images of players

we were able to produce smooth high‑definition images

extracted from competition videos extracted shuttle-of players at 930‑pixel resolution and 60‑fps frame rate

cock images

video obtained from OBS

etc

and

displayed the necessary data on the remoteviewing()

Stable Extraction of Subtle and Rapid Shuttle-

venue at the appropriate timing to create a highly

cock Movements

realistic spatial representation

We developed extraction algorithms specific for the
shuttlecock and succeeded in detecting its exact position

323

Birdʼs Eye‑view Sports Viewing Multi‑layer
Aerial Image Display Technology

and in accurately extracting its various contours To
detect shuttlecock position we devised a method for

To deliver the experience of watching as if viewers

learning the shuttlecockʼs position and movement

were at the competition venue we created a space that

information by inputting continuous frames in a convolu-

physically simulated the actual venue (Figure 6)

tional neural network (CNN) to eliminate the effects ofWith our technology a spectator stand at the same

small objects similar to the shuttlecock appearing in theheight as the actual competition venue was set up andvideo (such as seat guide lights) The position of thethe locations of the court net two half‑mirrors light‑

shuttlecock was determined on the basis of the roughemitting diode displays

position (heatmap) and correction value (offset) ob-

optimized to create a realistic appearance of the players

tained from the CNN Extraction of the shuttlecockin front and behind the net This enabled the holographi-

image was carried out by generating the image using acally displaying of the two players at their correct

background‑subtraction method and filtering with thepositions

extracted shuttlecock position and shape and predicted

and the projectors were

The use of two half‑mirrors however results in a

shuttlecock information (position contour and motion

discontinuous appearance of the shuttlecock

blur level) in the next frame

moves back and forth between the two mirrors Since

which

this problem is a structural issue arising from having two
208

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

Figure 6

Image of Equipment and Example of Multi‑layer Aerial Image Display

physically separated layers there is no perfect solution
to it

To produce a more natural appearance

experiment

we

repeatedly conducted trial‑and‑error experiments on

331

the timing for switching the shuttlecockʼs display layers

In terms of object‑extraction performanceand came to the conclusion that the best timing forsucceeded in real‑time processing of 8K 60‑fps videos

switching is when the player hits the shuttlecock To

There are issues that need to be addressed for extraction

determine the best timing for switching we combined

accuracy such as when images of the two players

Evaluation of Object Extraction Performancewemultiple approaches such as determining the shuttle-

overlap or extraction for doubles matches Neverthelesscock position through analysis of lateral court images

we succeeded in processing the images without missing

determining the timing when the shuttlecock is hit

parts or errors that could hinder proper viewing of thethrough detection of hitting sounds from the audio

games even for players wearing different uniforms forduring the match and human operators watching the

all the six live broadcast experiments We also confirmed

game manually switching the display layers

Thethat there were no errors in the correction of the black

combination of multiple approaches enabled the real‑zones caused by the net and in the addition of shadowstime switching of shuttlecock‑image display positions

For the shuttlecock images however more improve-

The highly realistic environment with a full‑size court
and net and life‑size holographic projections of players

ments are necessary due to considerable discontinuity
depending on the viewing position

apparently elicited a sensory illusion that the players
were actually in front of the participants This indicates332

the possibility of achieving an emotional connection at a

In terms of transmission performance the total end‑

Transmission Performance Evaluation

deeper level than when watching on television The

to‑end delay from the competition venue to the remote‑spectators were able to more easily feel the energy andviewing venue was less than 2 800 ms Table 2 lists thevibe during the competitions or the excitement oflag time details The MMT transmission delay included

players before the start of the matches and even the

network transmission delays of approximately 1 ms

emotions associated with winning or losing When Kento

between the competition venue and broadcast centre

Momota fell to his knees when he lost in the secondand approximately 0 1 ms between the broadcast centre

round of his match during the group play stage theand remote‑viewing venue Delays in object‑extraction

realistic sense of his presence in the remote‑viewing

processing included buffering that takes fluctuations invenue was very striking

processing time etc

into consideration but the actual

delay in extraction processing excluding the buffering
33

Results

The following are the technical results of thewas less than 400 ms
Since the demonstrations involved one‑way transmis-

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 2020209

Table 2

Details of Lag Timebetween competition and remote‑viewing venues andCamera system≤17 ms (≤1 frame)

conduct research and development on Kirari! and other

MMT transmission

1 600 ms (4K 59 94p BT 709
rate of encode : 40 Mbit/s FEC : 5%)

technologies for achieving the Remote World andExtraction of objects

1 000 ms(including input/output buffering)world using communication technologies

Presenting system〜120 ms (4 frames at 30 fps)

eventually make future proposals that will surprise theWe implemented this project working with Professor
Okudo and Researcher Yoshizumi of Wakayama University (the base technical research was supported by JSPS

sion of images from the competition venue to the

Grant‑in‑Aid 19K01141) partner companies people

remote‑viewing venue an approximately 3‑s delay did

involved in broadcasting competitions and conventions

not cause major problems However our future goal is tocompanies and organizations related to each system as

connect the two venues bi‑directionally and deliver thewell as local governments and JSCEs nationwide thatcheers from the audience in the remote‑viewing venue

have gathered citizens and related people and projectedback to the competition venue in which case significant

the images to them The projection was supported bylag times would become problematic making it neces-many people including local governments across thesary to further reduce processing time

country and planetarium facilities such as the National

We were able to confirm that the above performance

Museum of Emerging Science and Innovation (Mirai-of the demonstration system was sufficient in deliveringkan) which held projection events for citizens and othera realistic experience of the presence of the athletes ininterested parties We want to express our gratitude to

the remote‑viewing venue

them
References

4．Conclusion

(

)

(

)

(

)

The planned filming distribution and screening of all
programs could be performed without significant
problems with the dome image distribution

The

projection was sufficiently practical in operations sinceconsumer cameras and commercial best‑effort lines
were used to project images to planetarium systems
with entirely different specifications delivered to eachmuseum Clearly this approach was practical enoughand suggests the possibility of becoming a legacy in the

K. Nagai, “Evolution History of Broadcast Technology Boosted by
Olympic Games,” THE JOURNAL OF THE INSTITUTE OF IMAGE
INFORMATION AND TELEVISION ENGINEERS, vol. 70, no. 5,
pp. 363-371, 2016.
H. Kakinuma, J. Nagao, H. Miyashita, Y. Tonomura, H. Osada, and K.
Hidaka, “Real-time Extraction of Objects from Any Background Using
Machine Learning,” NTT Technology Journal, vol. 30, no. 10, pp. 1620, 2018.
Y. Tonomura, H. Imanaka, K. Tanaka, T. Morisumi, and K. Suzuki,“Standardization Activities for Immersive Live Experience (ILE),”
Journal of the ITU Association of Japan, vol. 47, no. 5, pp. 14-17,
2017.
(Received February 28 2022 ; Revised March 24 2022)

future for the live relay of realistic images of various
events
We applied Kirari! to badminton matches and demonstrated that it can be used to deliver a realistic
experience of being at the competition venue Although
there are issues regarding accuracy of extracting player
images and in the method for displaying the shuttlecock
we succeeded in demonstrating the possibility of a new
sports‑viewing experience beyond what television can
provide for the Olympic Games an event that attracts
attention from all over the world
Going forward we will carry out improvements inextraction performance and other technical aspects aswell as deploy the technology to other games and areas
such as for music concerts We will also study methodsof minimizing lag time for bi‑directional connection

210

KOSHIBA Keiichi
He joined Matsushita Electric Industrial Co
Ltd in 1989 and was transferred from Panasonic to Tokyo 2020 Organising Committee in 2014
After implementing this project he left Panasonic then he is verifying and analyzing thisproject as a visiting fellow at Wakayama
University currently working on the projectʼslegacy as CEO of G1 company Inc
KINOSHITA Shingo
He received a B E from Osaka University in1991 and M Sc with Distinction in technology
management from University College London
in 2007 He joined NTT in 1991 Until recently
he had been in charge of NTTʼs Tokyo2020
initiatives including sports‑watching video
technology He has been in his current position
since 2021 where he manages R&D on
information and communication processing of
humans based on human‑centered principles

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS