2‑11

2 Telecommunication Radio Broadcast

Universal Service Utilizing Real‑time Sports
Data Feeds
KUMANO Tadashi

UCHIDA Tsubasa

KANEKO Hiroyuki

We developed a technology to automatically generate play‑by‑play text using competition data broadcast from OBS andprovide them with synthesized speech and subtitles as well as sign language CG for the Olympic and Paralympic GamesTokyo 2020 The technology was implemented as a universal service to add to live streaming video The live sporting
updates were provided on a special Internet site in a format that was easy to understand and tailored to usersʼ
requirements such as for people with visual or hearing impairments people who cannot view a screen while driving a caror people on a train who cannot hear the sound This article describes the automated generation technology used for voice
play‑by‑play with subtitles and sign language CG play‑by‑play
Keywords：Visual impairment Hearing impairment Information accessibility Sign language Automated play‑by‑playSpeech generation Speech synthesis Sign language CG

1．Introduction

abandoned providing speech and subtitles due to humanresource limitations which was a problem from the

A widely used practice in program production is forperspective of allowing more people to enjoy the content

announcers to add audio descriptions such as play‑by‑

In addition even programs that include play‑by‑play

play and narration to video content This is useful forand narration are mainly intended for viewers without

conveying information that cannot be understoodvisual impairment and are often inadequate for visually‑

without viewing the video and enables the same

impaired viewers Broadcasting with live commentary is

information to be conveyed to all viewers irrespective ofan audio‑added service to compensate for this How-any impairments they might have (this is calledever in live broadcast programs in particular the“information accessibility”)

The information is alsoburden on the person providing the commentary tohelpful for many viewers in that it helps them better

judge the information to be commented on and speak atunderstand the content that is difficult to understandthe right time while listening to the broadcast audio iseven by viewing the video In the past non‑broadcast-high and this service has not yet become a continuous

ing services such as live‑streaming services sometimes

service
Although subtitles are available as a means ofproviding information to people with hearing impair-

KUMANO Tadashi
Science & Technology Research Laboratories Japan Broadcasting Corporation
E‑mail kumano t‑eq@nhk or jp
UCHIDA Tsubasa (IEICE Member)
Science & Technology Research Laboratories Japan Broadcasting Corporation
E‑mail uchida t‑fi@nhk or jp
KANEKO Hiroyuki (IEICE Member)
Science & Technology Research Laboratories Japan Broadcasting Corporation
E‑mail kaneko h‑dk@nhk or jp

ments for those who are deaf from birth or have lost

The Journal of The Institute of Electronics Information and Communication Engineers Vol 105 No 8 Supplement pp 82‑88 August 2022

There is a need to develop technologies for solving© 2022 The Institute of Electronics Information and Communication Engineers82

their hearing at an early age sign language is theirnative language and understanding information provided in Japanese and viewing it for long periods can be asignificant burdensuch problems that can automatically generate speech

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

subtitles and sign language CG (Computer Graphics)

Games

that provide appropriate explanations of the situation

distributes real‑time data called ODF (Olympic Data

instead of people especially for live broadcast content

Feed) (1) for all events Based on this data broadcasters

In response to this NHK has been conducting research

produce various contents such as superimposed text andand development on the automatic generation of play‑graphs on the broadcast screen Web content etc

the Olympic Broadcasting Services (OBS)by‑play commentary Sports content generally contains

In ODF “messages” are delivered each time there is aa large amount of video information and play‑by‑playchange in message content which is organized into acommentary is also indispensable for those who can view

single XML (eXtensible Markup Language) document

the video to understand the gameʼs content Therefore

for each type of information such as “participating

we developed a technology to automatically generate

athletes” and “game status” including current score andplay‑by‑play scripts from available real‑time competi-stats and a “list of events” that occurred during thetion data synchronized with the gameʼs progress andgame Although the types of messages delivered and theconvey them vocally using a speech synthesizer or by

specific information in the messages differ depending on

sign language using CG animation technology
This article describes the “robot play‑by‑play service

the competition specifications are defined so that they
can be handled in a unified format

with subtitles” and the “sign language CG play‑by‑playservice” implemented by NHK as universal services at
the Olympic and Paralympic Games Tokyo 2020
(hereinafter referred to as “Tokyo 2020 Games”)

3．Mechanism for Generating Play‑by‑play
Scripts
Figure 1 shows the overview of our system that

2．Real‑time Competition Data

automatically generates voice play‑by‑play from competition data broadcast in real‑time Real‑time data is in a

Data such as team or athlete names scores penalties

predefined format updated at the venue as the gameand each athleteʼs movements have been generated andprogresses and distributed with every update The

distributed in real‑time especially in large‑scale sport-

play‑by‑play generation system receives the data

ing events In the case of the Olympic and Paralympic

sequentially and always maintains the latest status of the

Figure 1

Mechanism of the Robot Play‑by‑play System

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 202083

game In addition the system maintains the gameʼs

special condition such as “when a certain time has

status which the listeners who have heard the previous

elapsed since a certain information has been conveyed

play‑by‑play comments should know to find thewithout any change”

differences between the previous and current gameconvey the same information periodically such as a

status Specifically the following steps are repeatedcompetitorʼs name or current score that has not changed

from the gameʼs start to end (2)for a long time This is useful for those who cannot

which enables the system tocontinuously check the information on the screen or who①

Compare the latest status with the game statushave started watching a game from the middle

already known to the listeners and check whether
there is anything new that needs to be conveyed②

If there is anything new to be conveyed as a result

4．Production and Distribution for Each Serviceof step ① select one from the collection of preparedThis section describes the method of producing andincomplete play‑by‑play scripts according to the

distributing service contents for each robot play‑by‑result of the comparison and complete it by filling inplay and sign language CG play‑by‑play from generated

the latest game status If nothing is to be conveyed

scripts by the method described in Section 3 Figure 2

return to step ①

shows an overview of the entire universal service③

Convert the completed script into voices using asystemspeech synthesizer and send it out together with thevideo and audio from the venue④

Reflect the conveyed information in the gamestatus known to the listeners⑤

Return to step ①

41

Robot Play‑by‑play with Subtitles

The generated play‑by‑play scripts were synthesized
into voice speech for our audio play‑by‑play service
named “robot play‑by‑play” The synthesized voice wassuperimposed on the video and audio content from theWe call a collection of incomplete (some information

venue and provided via the Internet In addition theneeds to be filled) play‑by‑play scripts required in step

textual scripts to be read by the speech synthesizer are② a template The templates must be prepared for eachalso delivered to enable viewing with subtitles on a Web

sort of target event for each voice and sign language CG

browserplay‑by‑playcannot hear Figure 3 shows an image of the delivered

Each script in a template has someconditions to be selected according to the type ofdifference detected in step ① They can also have a

Figure 284

thereby improving service for those who

content of the robot play‑by‑play with subtitles
We developed a dedicated speech synthesizer suited

Overview of Universal Service System (©2021‑International Olympic Committee‑All Rights Reserved)

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

for sports play‑by‑play This is achieved by preparing

Figure 4 (3)many voice samples that include expressions used in

Each sign language script is described as a sequence ofplay‑by‑play by recording the human voice reading andsign language unit IDs Every ID represents a different

having the DNN (Deep Neural Network) learn them In

sign language expression that could be used in play‑by‑addition we achieved a more realistic and understand-playable speech play‑by‑play by implementing a mechanism

expressions competition terms such as “defensive fouls”

for the speech synthesizer which can recognize specialand “traveling” athlete names and other proper nouns

symbols embedded in speech scripts indicating where toincluded in the competition data

place emphasis and pause

including words

common phrases

numerical

The sign language CG generator converts the ID
sequences mentioned above into CG representing a42

Sign Language CG Play‑by‑play

As in the case of robot play‑by‑playseries of actions It consists of a sign language motion
the sign

database that stores the motion data of CG characters

language CG play‑by‑play production system firstcorresponding to each ID CG model data such as CG

generates textual sign language scripts by filling thecharacters and studio sets and a rendering engine that

latest status of a game into an incomplete script selected

generates sign language CG animations using these datafrom a template then converts them into sign language

Optical motion capture was used to record the motion

CG animations The system configuration is shown indata for the sign language motion database Markers
were attached to the head and joints of limbs etc andtheir movements were captured by more than 40
infrared cameras recording and storing the data Figure
5 shows how the recording was done in the motion
capture studio and hand with markers attachedThe generated sign language CG animation was laidout with the score and subtitle information generated
together with the speech text from the competition data
It was output with the competition video in the upper left
corner of the image blank
The video synthesizer combined the competition video
and the animation output from the sign language CG
generator through a delay device to produce a single
video stream The final composite screen consisted of
the competition video in the center with the generated
sign language CG animation on its right the score
display using the distribution data on the bottom andFigure 3

Screenshot of the Robot Play‑by‑play Service with
Subtitles (©2021‑International Olympic Committee‑
All Rights Reserved)

Figure 4

subtitles generated from the distribution data on thelower right On the left side of the competition video a

Structure of the Sign Language CG Play‑by‑play System (©2021‑International Olympic Committee‑All Rights Reserved)

Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 202085

graph visualizing the strength of the audio included intext at the bottom of the composite video When the user

the competition video is displayed enabling viewers toclicks on the text another male character appears on theobtain comprehensive informationsuch as visuallyright side and explains the foul in sign language This

capturing the excitement of the spectators at the venue

feature was developed in response to many requests

The composite video generated by this system wasfrom users with hearing‑impaired in a preliminary

provided as though streaming a video on a website Anquestionnaire survey such as “I cannot understand what

example of the website screen is shown in Figure 6 The

happens when a foul occurs just by looking at the video”

fouls during the game are immediately displayed in theand “I understand that an athlete stops playing and the( a ) Studio recording( a ) Animated character( b ) Natural (photorealistic) character( b ) Attaching markers to the hand
Figure 786Comparison of Animated CG and Photorealistic CG
Representing 3 to 0

Figure 5

Motion Capture Recording

Figure 6

Example of the Sign Language CG Play‑by‑play Service Screen (©2021‑International Olympic
Committee‑All Rights Reserved)

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS

game is suspended but I want to know the details of
why they were suspended”

Sign language CG play‑by‑play was used for basketball at the Olympics and 14 videos of 28 hours wereA new CG character was developed for the sports
play‑by‑play which is more natural (photorealistic)

distributed It was implemented in two sports for the
Paralympics

wheelchair basketball and wheelchairlike a live‑action character (Figure 7) instead of therugby and 13 videos of 26 hours were distributed Thispreviously developed weather sign language CG (4)was NHKʼs first attempt to add sign language CG to live

developed by NHK for weather information which wassports broadcasts and the service received positive

an animated character with large caricature eyes In

comments from a wide range of viewers not just thosesign language not only are the shape and movement ofwith hearing impairments The naturalness of the sign

the hands but also the lifting and lowering of thelanguage expression using high‑quality CG characterseyebrows the widening of the eyes the shape of thewas highly evaluated on social networking services withmouth and the expression and posture of the entire facecomments such as “The sign language is quite easy to

are used as grammar to convey information For theunderstand” and “The fine facial expressions that arephotorealistic character of this project various initia-

important for sign language have been studied and thetives were implemented to express natural facial

soft hand movements are amazing !” In general sign

expressions in response to local changes in the motion

language interpreters usually take turns every 15

data

minutes but in extended programs such as sports there
is a significant burden on the interpreters In additioninterpreters with specialized knowledge must be found

5．Service Achievements

for sign language in sports

which is infrequently

For the Tokyo 2020 Games we provided the “Robot

expressed in daily life making it challenging to provide

Play‑by‑play Service with Subtitles” and “Sign Lan-sports play‑by‑play using sign language However thisguage CG Play‑by‑play Service” via the Internet on

technology has made it possible to provide fully‑NHKʼs special website

automated sign language CG play‑by‑play

The information in the real‑time data differs significantly from competition to competition

making it

6．Conclusion

challenging to provide meaningful play‑by‑play forcompetitions with little helpful information and inputThis article describes the robot play‑by‑play withdelays significantly impact the quality of the actual

subtitles and sign language CG play‑by‑play for thesituation as most of the data is input manually The

Tokyo 2020 Games as an approach to universal service

selection of competitions was made based on a compre-

utilizing real‑time data feeds We developed a technol-

hensive consideration of these factors

ogy to automatically generate play‑by‑play scripts fromthe robot play‑by‑playsports data transmitted in real‑time during competitions

service with subtitles was provided in 10 competitions

and convert them into synthesized speech and sign

(weightlifting canoeing swimming table tennis tennis

language CG animations We also reported implement-

beach volleyball basketball badminton volleyball anding a universal service that adds them to live streaming

rowing) and 175 videos of 518 hours were distributed

video

At the Olympic GamesAt the Paralympics 110 videos of 506 hours wereThis technology will automate the production of audio

distributed in seven competitions : swimming wheel-play‑by‑play

subtitles

and sign language for live

chair tennis wheelchair basketball wheelchair rugby

content which has been challenging to provide due tositting volleyball table tennis and badminton The

human resource issues and will enable the increase ofservice was especially popular for rowing and canoeing

universal content that more people can enjoy Public

and we believe it helped people watch live streaming ofbroadcasting aims to provide all people with therarely broadcasted sports The site was well‑received

necessary information quickly and accurately We willon social media with users commenting that it wascontinue to focus on research and development of“simple and easy to understand” and “follows the flow ofuniversal services in cooperation with the parties

the sport and explains the situation with fairly quick

concerned

responses” and the system was used not just by people
with disabilities but also by a wide range of people
Special Issue : Technology and Innovation in the Olympic and Paralympic Games Tokyo 202087

References()

International Olympic Committee, “Olympic data feed.”
http://odf.olympictech.org/( ) T. Kumano, M. Ichiki, K. Kurihara, H. Kaneko, T. Komori, T. Shimizu,
N. Seiyama, A. Imai, H. Sumiyoshi, and T. Takagi, “Generation of
automated sports commentary from live sports data,” Proc. IEEE Int.
Symp. Broadband Multimedia Systems and Broadcasting, June 2019.
( ) T. Uchida, H. Sumiyoshi, H. Kaneko, and M. Sano, “Delay reduction
and usability improvement of automatic sign language CG generation
system for sports program,” ITE Winter Annual Convention, 15B-2,
Dec. 2019.
( ) M. Azuma, N. Hiruma, H. Sumiyoshi, T. Uchida, T. Miyazaki, S.
Umeda, N. Kato, and Y. Yamanouchi, “Development and evaluation ofa system for automatically generating sign-language CG animation
using the meteorological information,” Proc. 16th Int. Conf. Computers
Helping People with Special Needs, July 2018.
(Received February 28 2022 ; Revised March 18 2022)

KUMANO Tadashi
He graduated from the Tokyo Institute ofTechnology in 1993 with a degree in Computer
Science He completed his Masterʼs in Computer Science from the Graduate School of
Science and Technology Tokyo Institute ofTechnology in 1995 He joined NHK in thesame year He has been engaged in researchand development of natural language processing machine translation sports play‑by‑playgeneration and speech synthesis at the Scienceand Technology Research Laboratories since
the same year From 2000 to 2004 he wasseconded to ATR Spoken Language Communication Research Laboratories Currently he isa senior researcher in the Smart Production
Research Department

UCHIDA Tsubasa (IEICE Member)
He graduated from Ibaraki University in 2008
with a Bachelorʼs degree in Media and Telecommunication Engineering He completed his
Masterʼs course in Media and Telecommunication Engineering at the Graduate School of
Science and Technology Ibaraki University in2011 He joined NHK in the same year After
working at Sendai Broadcasting Station he has
been engaged in the research and developmentof sign language CG at the Science andTechnology Research Laboratories since 2014
Currently belongs to the Smart Production
Research Department
KANEKO Hiroyuki (IEICE Member)
He graduated from Sophia University in 2000
with a B E in Electrical and Electronic
Engineering He completed his M E in Electrical and Electronic Engineering at the Graduate
School of Science and Technology Sophia
University in 2002 He joined NHK in the same
year After working at Sendai BroadcastingStation he has been engaged in research on
content description production and sign
language CG generation technology at the
Science and Technology Research Laboratories
since 2006㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇㍇

88

THE JOURNAL OF THE INSTITUTE OF ELECTRONICS INFORMATION AND COMMUNICATION ENGINEERS