NTT Technical Review

December 2021 Vol. 19 No. 12
View from the Top
 Toru Takama, President & CEO, NTT Anode Energy Corporation
Front-line Researchers Yasushi Takatori, Senior Distinguished Researcher, NTT Access Network Service Systems
Laboratories
Rising Researchers
 Atsutoshi Kumagai, Distinguished Researcher, NTT Computer and Data Science Laboratories
and NTT Social Informatics Laboratories
Feature Articles: Toward a New Form of Coexistence between the Real World andCyberworld
 Research and Development of Digital Twin Computing for Creating a Digitalized World Thought-processing Technology for Understanding, Reproducing, and Extending Human
Thinking Ability
 Research on Symbiotic Intelligence for Achieving Symbiosis among People and between
People and Machines Cybernetics Technologies for Symbiosis between People and Machines Elemental Technologies for Supporting the Integration of Real Space and Cyberspace
Feature Articles: Olympic and Paralympic Games Tokyo 2020 and NTT R&D—Technologies
for Viewing Tokyo 2020 Games Introduction
 Initiatives for a New Sports-viewing Experience Sailing × Ultra-realistic Communication Technology Kirari!
 Badminton × Ultra-realistic Communication Technology Kirari!
 Marathon × Ultra-low-latency Communication Technology
Global Standardization Activities
 High-accuracy Time-synchronization Technology
Communications in the 5G and Beyond 5G erasfor

Low-latency,

High-capacity

Practical Field Information about Telecommunication Technologies
 Case Studies of Communication Problems with Video Transmission Systems Using Wireless
LANs
External Awards/Papers Published in Technical Journals and Conference Proceedings

View from the Top

Treasure Every Encounter to ExpandYour World
Toru Takama
President & CEO, NTT Anode Energy
Corporation
Overview

NTT Anode Energy is promoting the smart energy solution business,
which leverages the resources and information and communication technologies possessed by the NTT Group. Two years after its establishment,
the company is striving to contribute to resolving social issues, such asenvironmental and population problems, revitalizing industry, and building
a sustainable society. We interviewed Toru Takama, president and chief
executive officer of NTT Anode Energy, about the progress of business topromote and expand renewable energy and the mindset of top management.
Keywords: renewable energy, carbon neutrality, power purchase agreement

Expand sales by developing renewable energy
sources and promote smart grids—NTT’s new environment and energy vision “NTTGreen Innovation toward 2040” is exactly where NTT
Anode Energy will play an active role.
In September 2021, NTT announced a new environment and energy vision called “NTT Green
Innovation toward 2040” to simultaneously achieve
the contradictory objectives of “zero environmentalload” and “economic growth” by reducing environmental load due to business activities and creating
breakthrough innovation. This vision states twogoals: the first is an 80% reduction in greenhouse-gas
emissions by the NTT Group (compared to FY2013)and carbon neutrality for the mobile (NTT DOCOMO)
and datacenter sectors by FY2030, and the second iscarbon neutrality for the NTT Group by FY2040.
NTT Anode Energy is playing a role in achieving
these goals through business development on thebasis of two pillars: (i) developing renewable energy
sources and expanding sales and (ii) promoting local
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

generation for local consumption of energy, centeringon “Battery Farms.”
Since the establishment of the company, we havebeen focusing on developing green power as a renewable energy source. In addition to constructing andinstalling green power plants, we provide green
power to companies that recognize the value of
renewable energy. We will supply green power to
DOCOMO Denki, a retail electricity service scheduled to be launched in March 2022.
The supply of green power requires storage batteries to store surplus power from green power plants
and to supply power when the amount of power being
generated drops. Accordingly, we are establishing
Battery Farms, an environment in which green power
plants can be easily connected to power-transmission
and distribution networks by installing storage batteries in approximately 7300 of NTT Group’s telecommunication buildings scattered across Japan.
We began providing green power to stores such as
Seven & i Holdings Co., Ltd.’s convenience storesthrough Japan’s first off-site power purchase agreements (PPAs). Under off-site PPAs, power is supplied1View from the Top

to the customer’s facilities from power-generation
facilities installed in remote locations through powertransmission and distribution networks. We also provide green power to Daiichi Sankyo Chemical
Pharma Co., Ltd. and Furukawa Electric Co., Ltd.
through on-site PPAs under which power is supplied
from solar-power generation equipment, etc. installed
next to the customer’s facilities.
—What specific action will be taken within the NTT
Group to achieve this vision?
The NTT Group currently uses just over 1% of theelectricity consumed in Japan. I believe that ourefforts to transform this 1% into green power are ofgreat social significance. If NTT Group’s electricity
consumption follows its current trend, it is projected
to double by FY2040 compared to FY2013. Therefore,“NTT Green Innovation toward 2040” presents avision of achieving decarbonization and carbon neutrality of the NTT Group by FY2040 by cutting
power consumption in half and converting it into
renewable energy through implementing theInnovative Optical and Wireless Network (IOWN)
and energy saving.
Conventional energy systems have typically been
centralized systems; that is, power generated at large
power plants is delivered to points of consumption
via power-transmission and distribution networks. In
contrast, energy systems from now on will be distributed systems. NTT Anode Energy plans to construct
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

a distributed energy system by using NTT Group’s
telecommunication buildings across Japan as Battery
Farms to provide locally generated renewable energy
for local consumption. This system will reduce electricity loss due to long-distance power transmission
and distribution and enable more efficient use ofenergy by, for example, controlling the power supply
in accordance with supply-and-demand conditions.
The construction and rapid development of Battery
Farms and power plants is our highest priority.
How to seize opportunities during
challenging situations—Have you had anything to do with energy in yourpast work?
My position at NTT Anode Energy is my firstenergy-related job. I joined Nippon Telegraph andTelephone Public Corporation (now NTT) in 1981,
and for 26 years starting my second year there, I haveled a challenging corporate life during which I havetaken on work without any briefing or handover; inother words, I have always been assigned to projects
new to NTT. After completing my training period, I
was assigned to a technology development department related to transmission and took charge of digital-transmission technology. At that time, telecommunication networks were being digitized, and a very
accurate digital clock was needed to ensure that entire
networks operated in synchronization. To meet this2

View from the Top

need, we developed the “D2 standard-clock generator,” which became Japan’s standard clock. (Theactual device is on display at the NTT History Centerof Technologies in Tokyo.) Since then, I have worked
in fields such as product development, global affairs,
technology development, and software development,
but NTT Anode Energy is my first job in the energy
field. However, most of the work I have experienced
thus far has been characterized by taking new steps,
such as starting new projects or joining new organizations, and the energy field is the same in the sense ofbeing new to NTT.
Among my experiences, one that left a particularly
strong impression on me was working at the WorldBank in Washington, D.C. beginning in 1987 forthree and a half years, where I served as a professional in charge of loan appraisal and supervision of
telecommunication projects in Indonesia and
Bangladesh. The World Bank has nothing to do withtechnological development. I arrived in Washington,
D.C. on a Sunday, and since I had just arrived, I hadneither found a house to live in nor other living
arrangements. The next day, Monday, on greeting myboss for the first time, I was given a document of
about 150 pages and told, “Tomorrow, you will attend
a meeting on behalf of our section and write a reporton it.”
I had hoped to be able to get my life together before
I started work, but that hope was dashed. Although I

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

had studied in France before, this was my first time in
an English speaking country. On top of that, my bossat the time was Indian, and I had difficulty in understanding his heavy Indian accent. The theme of themeeting that I attended was, naturally, finance, andthe conversation was filled with banking terminology
in English. As an engineer, I didn’t know such banking jargon, so I couldn’t write the expected report. As
a result, I spent nine months in a private room without
being given any work.
—That’s a tough situation. How did you overcome
that?
First, I made an effort to brush up on my English by
studying at a business school and earned an MBA(master of business administration) at night school.
At the World Bank, professionals have their own
offices without having meetings or document
exchanges, so I had no way of knowing what wasgoing on in my own section. Therefore, I invited colleagues of my section to lunch to find out what wasgoing on and ask if there was any work I could do. As
the World Bank was a competitive workplace, theywouldn’t talk about good jobs with me.
Occasionally, some people told me they had a job
for me, but it was either very challenging or unpleasant. However, it was better than no work for me, so I
volunteered for such a challenging job. Since I didn’t

3

View from the Top

have the capability to carry it out, I invited other colleagues to lunch and asked them how to do the job. As
I repeated this process over and over, I was able toreceive help from different people and finally andfortunately was assigned to telecommunication projects in Bangladesh and Indonesia after nine months.
The World Bank is an organization with full ofsenior professionals. One of the chief economists atthat time was Lawrence Summers. Later he became
the U.S. Secretary of the Treasury in 1999 and thepresident of Harvard University in 2001. I worked
with former Managing Director of Sri Lankan
Telecom, and my boss with the heavy accent was theformer Managing Director of Indian Development
Bank. Looking back, I was fortunate to have opportunities to meet these people.
It was also during that time that I met one of thefounders of Apple, Mike Markkula. Later, in 1995, I
happened to be a member of the joint project between
NTT and Apple. During the project, we developed
and commercialized a system called Desktop
Communication Conference System FM-A71, which
had almost identical functionalities to today’s web
conferencing systems, allowing users to share images
during a conference call. Web conferencing is now
used for remote work as a matter of course; however,back then, the Internet was not yet widespread, andour system required an ISDN connection (integrated
services digital network). Even though it did not sellwell, in retrospect, I think our efforts were pioneering.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Results will always emerge over time
—Are there any moments when you feel that the work
you have done thus far has made a difference in yourcurrent position?
NTT’s electric-power business has a shorter history
than its telecommunication business. As I mentioned
earlier, I have many experiences of starting new businesses from scratch, so I think I am used to challenging situations. Even if you face a big decision or afork in the road in regard to the job at hand, it may not
be such a big issue if you look at it from a wider perspective. I believe that we should work on one thing
at a time and not rush things, because results will
surely come as time goes by. However, some things
must be addressed immediately. For example, investment in Battery Farms and power plants that enablelocal generation and local consumption of energy isneeded right now. I think that my past experiences ininvesting in overseas companies has come in handy inthat regard.
I have learned from past experiences two things thatare important to me in top management. One is communication, and the other is “ichi-go ichi-e,” whichmeans treasure every encounter, for it will neverrecur. I have thus far managed to overcome difficulties thanks to the help of many people and cherish themany encounters I have had. Thanks to them, I am
now able to do the job I do today.
In addition to these two points, in top management4View from the Top

you must have the ability to execute plans, make decisions, and make judgments. It is important to listen
carefully to the opinions of others, including employees, and once you have made a decision, try your best
to carry it out or call it off if you find that the decision
is not working. For that reason, I listen to people a lot.
This is because many clues can be found by listening
to people. It is also obvious that there are limits towhat one person can think and do alone. Therefore, it
is crucial to create an environment in which people
support each other. This environment cannot be created without good communication.
—Communication is also important for engineers
and researchers to flourish. What is your advice tothem?
I have always valued communication—both internally and externally. I invite employees to lunch tocreate a place for communication, and I have been
holding quarterly group discussions with employees
for the past 20 years. Setting a theme and discussing
it creates a variety of ideas, and in a number of cases,
those ideas have become reality.
Outside the company, a study group that I launched
38 years ago is still going, and a total of 800 people
are registered. It is held once a month, and each session has 20 to 30 participants. I try to make the mostof every moment I get to meet each of the members
who attend the study sessions. This accumulation of
encounters leads to work and connections that expand
my world, and I am grateful that it does.
When I was head of the Advanced IP Architecture
Center at NTT Communications, a technology-devel-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

opment organization, I set up an open lab for co-creating with venture companies by offering NTT
Communications’ cloud services and mentoring forfree for six months. We started this project to support
as many entrepreneurs as possible, and although it did
not directly benefit the NTT Group, some of the organizations that used it are now successful companies.
Also, when I was working at Verio, a US Internet andweb hosting company, I learned about the benefits of
the open-source model from their business.
I have been creating settings that lead to such open
innovation in Japan and abroad as well as promoting
initiatives to energize engineers. I hope that engineers
will appreciate the importance of communication
with various people and keep challenging themselves. I also hope that researchers in the software
field will communicate with their outstanding engineering colleagues and work on many interesting
projects together.

Interviewee profile
Career highlights
Toru Takama joined Nippon Telegraph andTelephone Public Corporation (now NTT) in
1981. After being seconded to the World Bankand Verio Inc. he had served as President of theAdvanced IP Architecture Center of NTT
Communications, member of the Board of NTT
COMWARE, and Executive Vice President of
NTT TechnoCross. He has been in his current
position since June 2020.

5

Front-line Researchers

Venture Out of the Laboratory and
Test Your Research Results in theReal World
Yasushi Takatori
Senior Distinguished Researcher,
NTT Access Network Service Systems
Laboratories
Overview
Future networks are expected to provide high-speed, largecapacity communications and enormous amount of computing
resources that exceed the limits of the conventional network infrastructure. To meet these expectations, NTT is conducting researchand development on the Innovative Optical and Wireless Network(IOWN), a future information-processing infrastructure that
includes user’s terminals. Yasushi Takatori, a senior distinguishedresearcher at NTT Access Network Service Systems Laboratories,
is taking on the challenge of innovating wireless access toward the 6G (six-generation mobile communication system) era, which will be an essential component of IOWN. We asked him about his research
activities and his attitude as a researcher.
Keywords: wireless access, Cradio®, extreme NaaS

Provide a natural and uninterrupted
communication environment that does not force
users to be aware of it—Please tell us about your current research projects.
My current research is on using multiple wirelessaccess systems to provide wireless access in thesixth-generation mobile communication system (6G)
era, which will be an essential component of theInnovative Optical and Wireless Network (IOWN). In
the field of wireless communications, the amount ofcommunications via smartphones is increasing, andaccompanying the development of the Internet ofThings, a multitude of objects have become connectNTT Technical Review

Vol. 19 No. 12 Dec. 2021

ed to the Internet. The types of wireless terminals andusage scenarios are also diversifying, as illustrated byremote control of autonomous vehicles and drones
and exchange of ultra-high-definition images. Under
these circumstances, the role played by wireless communications should increase dramatically in every
aspect of our lives and that the amount of such communication will continue to grow. Such diversification of terminals and usage scenarios will require
wireless networks to cover not only urban areas,
where the population is concentrated, but also suburban areas, rural areas, and many other locations. It isalso necessary to satisfy various requirements concerning wireless communication quality, which varies according to various usage scenarios.
6Front-line Researchers

Prediction

Wireless-quality
prediction/estimation
technologies

Cradio®

Assessment

Wireless sensing
and visualization
technologies

Cooperation

Wireless-network
dynamic design/
control technologies Control

Fig. 1. Multi-radio Proactive Control Technologies (Cradio®) with various applications and social systems.

Against that backdrop, I am taking on the challenge
of developing a wireless access network that offers
new value with the following key phrases in mind:“achieving continuous connection,” “sharing experiences,” and “expanding into unexplored areas.” 5Gfeatures high speed, large capacity, low latency, andsimultaneous connection of many terminals. Looking
toward IOWN and 6G, however, it will be necessary
not only to improve the performance of each feature
but also skillfully combine these features. Therefore,
we are developing elemental technologies for
extreme network as a service (NaaS) to deliver wireless access that satisfies individual extreme service
requirements in a flexible manner whenever andwherever needed. One group of such elemental technol‑ogies for proactively controlling wireless access byassessing various wireless conditions is called Multiradio Proactive Control Technologies or Cradio ®
(Fig. 1).
—You are attempting to create a communication environment in which people do not have to be aware of
the wireless network. What exactly are you workingon?
As one of the components of IOWN, Cradio is a
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

group of wireless technologies that will provide users
with a natural and uninterrupted communication
environment without making them aware of the wireless network. It consists of the following three areasof technologies: (i) wireless sensing and visualization
technologies are tasked with assessment, namely, collecting and visualizing wireless status and visualizing
real-world status through means such as wirelesssensing; (ii) wireless-quality prediction/estimation
technologies are tasked with prediction, namely, predicting and estimating wireless communication quality that changes moment by moment; (iii) wirelessnetwork dynamic design/control technologies aretasked with control, namely, designing physical locations of movable base stations, deriving optimal wireless parameters, and dynamically controlling network
parameters, resources, etc. according to the environment and requirements. By developing the three areasof technologies and coordinating them in real time,
we aim to satisfy user requests and control the
momentarily changing wireless conditions to provide
a natural communication environment that does notforce the user to be aware of the wireless network.
To maximize the potential of wireless communications, we are also researching intelligent radio environment formation technology for controlling the7Front-line Researchers

Analysis/forecast
AI

EffectCapacity
expansion

AI/ML
Control oftransmission
environmentsCollection/
visualization
RIS

Frequency
sharing

Virtual antenna

Big data

Improvementof quality
Improvementof accuracy

Sensing

Conventional concept of control oftransmission environments

Formation of intelligent radio environments

Performance of a wireless system depends
on propagation channel
User
Base
equipment
station
Digitalprocessing

Channel(Given)Digitalprocessing

Control the propagation-channel components to further
improve performance of wireless systems

Digitalprocessing

Channel(Formed)Digitalprocessing

Control
AI: artificial intelligence
ML: machine learning
RIS: reconfigurable intelligent surface

Fig. 2. Formation of intelligent radio environments.

environment in which radio waves propagate (Fig. 2),
aiming at a paradigm shift from a given propagation
channel to a formed propagation channel by controlling the propagation-channel components to achievea fast and stable connection.
Up until now, we have considered that the environment is given to us and we had to build a communication system optimized for that given environment;
however, we changed such perspective and started
researching and developing various technologies forcreating the environment ourselves. For example,
when a typical reflector receives light or radio waves
from a certain direction, it reflects the light or radio
waves in a certain direction. However, using a recently developed reflector that can arbitrarily change thedirection of reflection, we conducted an experimentto change the reflection environment.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

We have used Cradio to drive autonomous farmmachinery on a farm road in Iwamizawa City, Hokkaido. As the autonomous farm machinery runs
across multiple wireless networks such as 5G, artificial intelligence predicts fluctuations in communication quality and automatically switches to an appropriate network before the communication quality
degrades. Thus, we demonstrated stable and automatic driving of the machinery without interrupting
the remote monitoring of the machinery. In coordination with NTT’s Cooperative Infrastructure Platform,
Cradio overcomes the differences in Internet Protocol
addresses and communication schemes, thus provides connectivity independent of the particularaccess network. In this manner, users can use applications as usual without having to be aware of the network.
8

Front-line Researchers

Accumulate experiments, measurements, andevaluations while confirming those results in thereal world—You were able to achieve results by changing theconventional perspective and approach, right?
Yes, we got that idea from a chat within our team.
A technology can be environment-dependent; that is,
one technology may produce good results in a certain
environment but not the same results in another environment. We wondered if we had been bound by that
notion and willfully set limits based on it. Therefore,
we decided to overcome those limits by applying different ideas more freely, controlling the environment
for ourselves, and changing our perspectives.
This shift in thinking has also been influenced bytechnological advances. In the past, only one type of
wireless system existed in one area; however, various
types now co-exist. As technology progresses, we’ll
likely end up with an environment in which many
wireless systems overlap each other. When I pondered this situation, I thought that rather than looking
at optimizing each system on a one-by-one basis, weshould conduct research and development (R&D)
assuming an environment in which multiple wirelesssystems overlap.
I’m thankful that great expectations are being
placed on our R&D concerning wireless technology.
This heightened expectation from society is echoed in
the motivation of our researchers. People from various industries are interested in exchanging opinions
on this topic. This type of stimulation can trigger new
ideas and lead our R&D in the right direction. However, the more active the discussion, the more issues
will arise. Although we are sequentially addressing
the issues that have arisen, new issues may come up,
and it will take some time to address them all. Nevertheless, we are determined to patiently repeat experiments and accumulate knowledge bit by bit. Theseexperiments are low profile and physically demanding; however, they will provide us with unique datathat no one else in the world has seen. Being able tocreate new technologies based on those data isrewarding.
—You conduct experiments not only in the laboratory
but also in the actual field, as in the case of
Iwamizawa City.
One of the interesting aspects of research activities
is to actually go out into the field and conduct experiNTT Technical Review

Vol. 19 No. 12 Dec. 2021

ments and measurements. It is important to conduct
experiments, measurements, and evaluation while
confirming those results in the real world. When we
consider what type of wireless environment is needed
for use of better services in real life, the place forresearch activities is not the laboratory. Merely saying, “We have achieved various results in the laboratory.” does not mean that the results are real. To be
useful in the real world and meet the expectations ofusers, it is important to get out of the laboratory andverify our research results in the real world.
These experiments would not be possible without
the collaboration of people who are facing problems,
so it is very important to have an opportunity to conduct experiments in the field. In addition to conducting the experiment in Iwamizawa City, we haverecently collaborated with people who are having
trouble detecting damage caused by birds and otheranimals. The cost of the damage to crops caused bywild birds and animals is about 20 billion yen peryear, of which about 80% is accounted for by wildboars and deer. Such damage also leads to secondary
damage, such as loss of motivation to farm and abandonment of cultivation, which is becoming a serious
social problem in Japan. As a countermeasure to suchdamage caused by birds and animals, we have been
researching sensing technology using radio waves for
detecting intrusions by animals.
The request from our collaborators was for a technology to remotely check what the sensors installedin the cages detected. When a sensor detects something, an administrator or other person is sent tocheck the cage; however, the person sent to checkshould depend on whether a wild boar, deer, or otheranimal is captured in the cage. Since the current system does not detect what is in the cage, the person
sent to the site to check the content of the cage often
finds captured birds and other animals that are nottheir responsibility, so another person—who is
responsible for those animals—is sent to the site.
Sending images wirelessly from the mountains where
the cages are installed is a very difficult task becauseof many obstacles. However, we were able to carryout an experiment because that task matched a technology we were researching and developing.
Responding to requests from the fishing industry,
we also experimented with transmitting images taken
by underwater drones to the coast to obtain information on the types of fish caught in offshore nets andthe condition of the nets. It is valuable experience tobe able to see how well our technologies can performin such real environments. When we asked people
9Front-line Researchers

Don’t set limits, aim to break through themtise. I believe that since the sphere has only one center, the closer we get to the center by pursuing our
specialties, the more our sense of commonality
grows, no matter which field we pursue, and the more
we can understand each other at essential points. I
want to pursue R&D that pushes toward the center ofthe sphere.

—What do you keep in mind when you are looking fora research theme?

—Do you have any words of encouragement foryounger researchers?

It is important not to set limits. When I hear someone say, “This is the most significant characteristic,”
I would think, “Something is wrong,” and seek anapproach to break the deadlock. This way of thinking
came from my experience at Aalborg University in
Denmark, where I studied for a year (2004 to 2005).
After quoting Guglielmo Marconi’s words, “It is dangerous to put limits on wireless,” my supervisor asked
me what limits I was thinking of overcoming inregard to future wireless technology. This question
made me more determined to push the envelope, andI began to examine whether a goal is high enough or
how unconventional it is when setting my research
themes.
When IOWN becomes a reality, a world in which
everything is interlinked will come. In this era of
R&D, it is necessary to incorporate not only wirelesstechnology but also a variety of other elements into
the R&D process. This idea of incorporating R&D
and perspectives from other fields has been important
for a long time; however, it will become even more
important in the future, and I think we should aim to
adopt that idea.
If we consider only wireless technology, we tend to
pursue R&D for evaluating communication quality
and speed and determining how to improve it.
Instead, it is important to verify those parameters in
the actual environment, while keeping in mind how
the user’s experience and services can be improved,and create technology that works in conjunction witha real society. I also believe that we need to think ofproducts and services that are not only easy to use
from the viewpoint of the users but also easy todevelop from that of the people who develop them.
Imagining the R&D world as a sphere, I think there
are two approaches to R&D: one is to look at the surface of the sphere as a whole, and the other is to pushtoward the center of the sphere and hone one’s exper-

I believe that age does not determine one’s seniors
and juniors in regard to research activities and hopethat researchers will work hard together and improveeach other’s skills. When I talk to young researchers,
I sometimes find it a bit of shame that although they
all have very good sense and come up with the best
idea very quickly, they tend to dismiss ideas that area bit strange or didn’t work out. I think it is also
important to try out ideas that “might not work” withthe mindset that what didn’t work out this time willlead to a success down the road. The time you spend
thinking about something is unique to you, and thatexperience in itself is an asset that you can use foryour next R&D topic.
Also, don’t lock your ideas away; instead, create
new technologies by knocking those ideas against
other ideas. I want to work with young researchers onthese activities. Of course, sometimes we get stuck,and it can be difficult to make decisions about what todiscard from research that has not been successful
and whether to continue a research topic. In such acase, if you have the sense of “I like it” or “It’s interesting” even though your judgment with logic tells
you otherwise, you don’t have to throw it away.
I have been with NTT for more than 26 years, andwhen I joined the company, there were times when
our R&D on wireless technology did not bear fruit.
However, when many researchers envisioned thefuture of devices and networks, the last-mile accessnetwork was always wireless, and I have been pursuing research believing that the time of wireless technology would come. In light of the growing interestin wireless technology, I am glad that I have been
researching it for all these years.
R&D does not produce results in one or two years,
so keep a long-term perspective and be persistent. I
am sure that one day you will be glad you did.

with actual problems to collaborate in experiments,
the fact that they said, “Let’s give it a try.” indicatesthat they have high expectations for our research andtechnology. Being recognized in the real world in thisway is rewarding for us.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

10

Front-line Researchers

Interviewee profile
Yasushi Takatori
Senior Distinguished Researcher, Executive
Manager, NTT Access Network Service Systems
Laboratories.
He received a B.E. and M.E. in electrical andcommunication engineering from Tohoku
University, Miyagi, in 1993 and 1995. Hereceived a Ph.D. in wireless communication engineering from Aalborg University, Denmark, in2005. He joined NTT in 1995 and has served as asecretary of Institute of Electrical and Electronics
Engineers (IEEE) Japan Council Awards
Committee and a vice chairman of the Wireless
LAN system development project in the
Association of Radio Industries and Businesses(ARIB). He was a visiting researcher at theCenter for TeleInFrastructure (CTIF), Aalborg
University from 2004 to 2005. He served as a cochair of COEX Adhoc in IEEE 802.11ac from2009 to 2010. His current research interests
include future wireless access systems for IOWN.
He received the Best Paper Awards from theInstitute of Electronics, Information andCommunication Engineers (IEICE) in 2011,2016, and 2020. He was honored with the IEICEKIYASU Award in 2016. He received the Radio
Achievement Award from ARIB in 2020 and theIEEE Standards Association’s Outstanding
Contribution Appreciation Award for the development of IEEE 802.11ac-2013 in 2014. He is asenior member of IEICE and a member of IEEE.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

11

Rising Researchers

Research on Transfer Learning toGive AI the Same Versatility and Skillas Humans
Atsutoshi Kumagai
Distinguished Researcher,
NTT Computer and Data Science
Laboratories and NTT Social
Informatics Laboratories
Overview
Machine learning is needed to build artificial intelligence (AI),and this requires a large amount of training data. Sometimes,
however, you cannot get enough high-quality training data.
What’s more, to prevent an AI from becoming out of date, youneed to re-train it regularly to keep pace with data that is constantly changing. In this interview, Distinguished Researcher Atsu‑
toshi Kumagai talked to us about the technology of transfer
learning, which can be used to improve AI performance evenwhen ideal data cannot be obtained.
Keywords: transfer learning, AI, anomaly detector

Transfer learning improves AI performance even
without sufficient data
—What sort of technology is transfer learning?
When an artificial intelligence (AI) is given an
unknown task or a task without sufficient training
data, transfer learning technology helps improve performance by leveraging training data from related
tasks to compensate for data shortages. We are studying the possibility of using transfer learning technology to give AI the same versatility and skill as ahuman. For example, we humans can do a variety ofdifferent tasks like cooking, doing simple calculations, reading and writing, talking, running, throwing
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

a ball, and more. And when we’re presented with a
new task we’ve never seen before, we can use our pastknowledge and experience to adapt to and get thehang of it in relatively few attempts.
On the other hand, recent AIs have been able toperform as well as, and in some senses exceed,
humans in specific tasks for which large amounts ofdata can be prepared, such as image recognition. One
example of this is the news that an AI that plays go,called AlphaGo, beat the top player in the field. However, AIs do not have the versatility or skill thathumans have. This is common knowledge among
researchers like myself.
That’s why transfer learning technology is key
when trying to build a versatile and skilled AI. The
12

Decisionboundary fromanomaly detector

Normal data

NN modelTarget dataset

Related dataset 2

Related dataset 1

Rising Researchers

It is possible to generate appropriate decisionboundaries by utilizing information from relevant
data sets, even if there is no abnormal data.

Fig. 1. High-speed creation of an anomaly detector based only on normal data.

simplest way to build a versatile AI is to list all thetasks and provide large amounts of training data foreach task. However, actually listing every task is no
simple task, and even if you manage to do so, gathering the large amounts of training data required foreach task is not practical in terms of cost or resources.
Transfer learning technology can help us build versatile and skilled AI through performance improvements using related training data to compensate fordata shortages, even for unknown tasks.
—What specific research is being conducted into thistechnology?
The first piece of research being carried out is ontechnology for rapidly generating appropriate anomaly detectors in situations where only normal data isavailable. Anomaly detection is the task of finding
abnormal data, which is data with different properties
from normal data. To create a highly accurate anomaly detector, both normal and abnormal data sets are
generally required as training data. However, sinceabnormal data is very rare in nature, and so is hard to
obtain in practice, it is often not practical for teaching
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

anomaly detectors.
The approach I’m researching uses similar data setsthat contain both normal and abnormal data to generate an anomaly detector, even when only normal data
is available. Figure 1 is a diagram showing an appropriate anomaly detector being generated by inputting
a target data set with no abnormal data into a trainedneural network (NN) model. The NN model is trainedin advance using several related data sets (1, 2...) that
include normal and abnormal data to map the normal
data group onto the anomaly detector.
Normally, AI requires re-training each time a new
problem is presented, which is a process with a very
high computational cost. However, with this approach,anomaly detectors can be generated by simply entering a new data set into an NN model, making it a good
fit in cases where a real-time response is required, orin cases where computational resources are limited,
but you still want to perform anomaly detection.
The second piece of research is on technology fortransfer learning for tasks that change over time.
Machine learning learns something called a “classifier” that classifies data. But as the data changes overtime, so does the classifier. These changes are very
13

Rising Researchers

Decisionboundary

Class A sample
Class B sample

Fig. 2. Change in the decision boundary of the classifier due to a change over time.

common in real-world situations such as malware
detectors, where attackers are continually developing
new attack methods to slip past detectors. As a result,continuing to use malware detectors that have been
trained once without updating them results in theclassification accuracy continually deteriorating.
An effective method for preventing the accuracy ofthe classifier from degrading is to update the classifier in a timely manner using new additional data.
This method is simple and very effective, but collecting additional data costs money. Such image data
must be labeled, for example “this image is a cat” or“this image is a dog,” but this is work that must be
carried out by human beings, making it somewhat
costly. In addition, there may be other factors thatmake it difficult to obtain labeled data, such as theneed to protect personal data.
The approach I’m researching involves predicting
future decision boundaries from decision boundaries
learned using labeled data, without the need for any
additional labeled data. Figure 2 shows a decisionboundary with labeled Class A and Class B samples
that goes through three stages over time, allowing thenext change to the decision boundary to be predicted.
Transfer learning usually assumes there will be data
available from where it will be applied, but what’sinteresting about this study is that it does not use
additional labeled data.
—What are the current challenges you’re facing?
The main challenge is how to ensure that the model
is correct after training. In general machine learning,
separate validation data is prepared to check thetrained model. The accuracy of the learning can beassessed by how correctly it discerns between thevalidation data.
However, in research like this, it can sometimes bedifficult to create data for validation in the first place,or it may not exist at all. In particular, there is no datathat can be used for validating future forecasts, so it
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

is difficult to determine their accuracy. This is a major
challenge because safety must also be ensured when
considering practical applications in the future, especially in mission-critical areas.
Aiming to build versatile and skilled AI in
the future—What fields can this research be applied to?
For example, in the field of anomaly detection, suppose there’s a manufacturer that owns several factories. They build a new factory, and they want to makean AI that will automate the monitoring of equipment
on site. This is equivalent to creating an anomaly
detector for identifying abnormal data that differsfrom the normal data for the equipment. As the newfactory has no operational results, it may not be possible to create a high-performance anomaly detector
until enough abnormal data is collected for a rareequipment. However, if there are other factories thathave been running for a long period of time, sets ofnormal and abnormal data from those factories can beused to create highly accurate anomaly detectors assoon as the data from the new factory is entered. In
addition, to use a transmission service to centrally
manage the security of your network for multiple
customers, you can automatically create conditions
that are tailored to your customers by simply entering
the correct data.
Speaking of transfer learning technology for tasks
that change over time, another example of the security issues I mentioned earlier is applying transfer
learning to automatically updating anti-virus software. In addition, some companies may use a “blacklist” of sites that must not be connected to in order toprotect the company from malware, and the technology we have developed can be used to create theseblacklists.
E-commerce is another potential area of application. Transfer learning can be used to accurately
14

Rising Researchers

ocean areas where the potential of transfer learning isbeing overlooked and not being effectively utilized.
As a medium-term goal, I would like to focus on suchfields, and conduct research into broadening thescope of transfer learning. The lack of training data
can be a problem in many fields, so I hope that transfer learning will be a way of solving this practicalproblem.
In the long term, I would like to build an AI that has
the versatility and skill we discussed earlier. It will beinteresting if AI become able to learn independently
and do anything with just a single model, rather thanhumans having to teach and tune the AI for every new
problem or task.
predict customers’ hobbies and preferences as they
get older, and recommend the right products. It canalso be used to estimate the needs of new and infrequent users. It may be difficult to keep making predictions over a long period of time, but I think it may
be possible to reduce costs by slightly increasing thetime between updates. Research into increasing accuracy in situations where there are multiple data sets is
applicable in many areas, so it should have a wide
scope of application.
—What are the plans for future research?
At present, there are areas in which transfer learning can be used to ensure a practical level of accuracy,such as in image recognition and language processing. However, we believe that there are also blue-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Interviewee profile
Atsutoshi Kumagai
Distinguished Researcher, NTT Computer andData Science Laboratories and NTT Social
Informatics Laboratories.
He joined Nippon Telegraph and Telephone
Corporation (NTT) in 2012, where he researchesmachine learning and cybersecurity. He worked
as a member of NTT Secure Platform Laboratories(2012 to June 2021) and NTT Software Innovation
Center (2018 to June 2021). He has been withNTT Computer and Data Science Laboratories
and NTT Social Informatics Laboratories since
July 2021.

15

Feature Articles: Toward a New Form of Coexistence
between the Real World and Cyberworld

Research and Development of Digital
Twin Computing for Creating a
Digitalized World
Ryo Kitahara, Takao Kurahashi, Toru Nishimura,
Ichibe Naito, Daisuke Tokunaga, and Koya Mori
Abstract

The Digital Twin Computing Research Project aims to make Digital Twin Computing, a key component of the Innovative Optical and Wireless Network (IOWN) initiative promoted by NTT, a reality. We
established four grand challenges to achieve this objective and have been conducting research and development to meet each one. This article outlines each grand challenge and associated technologies anddescribes horizontal activities across these challenges.
Keywords: Digital Twin Computing, grand challenges, IOWN

1. Grand challenges for achieving Digital
Twin Computing

tives. We also describe activities designed to accelerate the meeting of these four challenges.

Digital Twin Computing (DTC) is a key componentof the Innovative Optical and Wireless Network(IOWN) initiative promoted by NTT [1]. It aims to
make large-scale and high-accuracy predictions andtrials of the future beyond the limits of conventional
information and communication technologies. It also
aims to achieve advanced forms of communication
having new value by combining various types ofhigh-accuracy information related to things, people,
and society in the real world. The idea is to create asmart society by solving a variety of social issues
affecting the world and creating innovative services.
We established “Mind-to-Mind Communications,”“Another Me,” “Exploring Engine for Future Society,” and “Inducing Inclusive Equilibrium Solutions
for the Earth and Its Social and Economic Systems”
as four grand challenges for achieving DTC as
announced in an NTT press release in November
2020 [2]. We have since begun specific studies oneach of these grand challenges. In this article, weoutline these challenges and the technologies being
developed to meet them and introduce future objec-

2. Mind-to-Mind Communications

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

“Mind-to-Mind Communications” is a research anddevelopment (R&D) objective to achieve a new wayof communicating that will enable people to directly
understand how other people perceive and feel things.
This new form of communication will transcend not
only language and cultural differences but also individual characteristics such as one’s experiences andsensitivities.
To achieve this objective, we are undertaking the
R&D centered about three key technologies: mindto-mind modeling, mind-to-mind conversion andpresentation, and state transformation and environment creation. First, mind-to-mind modeling technology uses sensor information consisting of biologicalsignals, images, speech, etc. to infer expressions inmind-to-mind communication, that is, in ways ofperceiving and feeling things. Next, mind-to-mind
conversion and presentation technology converts andpresents expressions in mind-to-mind communication that differ from one individual to another in a16

Feature Articles

I’m not staring at you but justlistening closely to understandwhat you are saying.

Change facial
expression

Improve psychological safety
and allow you to say what
you thinkResearcher

Actor

I was very nervous about
speaking at my firstinternational conference.

Mind-to-Mind Communications(It’s a feeling like your first
appearance on stage…)

Directly understand how another person mentally perceives and feels
things beyond differences not only in language and culture but also in
individual characteristics such as experiences and sensitivities.

Fig. 1. Mind-to-Mind Communications.

form suitable to one’s communication partner. Finally, state-transformation and environment-creation
technology promotes a change in how one perceives
and feels things by intervening not only in terms of
the information exchanged in communication but
also in regard to a person’s inner world, such as
receptivity, and the communication site.
Achieving this R&D objective will lead to a newstage of communication that will solve the problemsof discommunication (failure to communicate one’s
intentions) and miscommunication (mutual misunderstanding) prevalent in past ways of communicating. This will enable a person to realistically anddirectly understand how another person mentallyperceives and feels things beyond differences in language, culture, experiences, values, sense perceptions, and sensitivities. By improving psychologicalsafety and promoting mutual understanding, Mindto-Mind Communications creates an inclusive society in which people having diverse characteristics
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

engage in activities together while exploiting thosecharacteristics, thereby inspiring and helping eachother to grow (Fig. 1).
3. Another Me
The loss of various opportunities in one’s life due,for example, to the difficulty of balancing childcare
and caregiving with work or the inability to participate in society despite an interest or desire to do so, isbecoming a social problem. To dramatically increase
the opportunities for a person to become more active
and grow while expanding and merging the range ofactivities from the real world to the virtual world, weare taking up the challenge of achieving “AnotherMe” as another version of oneself in digital form who
can participate actively as oneself in a way that transcends the limitations of the real world and who canshare the results of such activities as experiences(Fig. 2). Our main initiatives in meeting this17

Feature Articles

Grow together while supporting
inspiration and decision-making ineveryday life
Life stage
Oneself

Create andupdateShareexperiences

Grow together

Shareexperiences

Another Me

Experiences

Fulfill one’s role throughopportunities such as performing
one’s duties at work

Fulfill one’s role through opportunities
such as community volunteer work
Experiences

10 times theopportunities

Another Me at work

Another Me in the community

Fig. 2. Another Me.

challenge are summarized below.
(1)	Transcend the limitations of the real world: Totranscend the limitations of time, space, andpersonal ability that reduce one’s opportunities in life, we seek to establish autonomous
agent technology that enables Another Me toautomatically perform activities in place of
the user and telepresence/body-augmentation
technology to accommodate remotely locatedusers or solve ability-related issues of thephysically challenged, elderly, etc.
(2)	Foster the user’s sense of self-agency: To
make the user feel as if the activities of Another Me are those done by oneself and experience a feeling of accomplishment and growth,
we seek to establish experience-transfer technology to achieve psychological teleportationand feedback. So that the actions of AnotherMe can be recognized by the user and others
as those of the user himself/herself, we will
also establish technology for creating and nurNTT Technical Review

Vol. 19 No. 12 Dec. 2021

turing a human digital twin who can develop apersonal image that changes from day to day
and exhibit individual characteristics such as asense of values and personality.
(3)	Social acceptance research: We will investigate philosophical, ethical, and legal issues on
the basis of how we view ourselves toward
widespread adoption of Another Me in society.
This approach will accelerate self-realization by
providing a variety of opportunities in one’s life andimproving one’s state of mind, health, and motivation
while enabling the experiences of Another Me togrow as those of one’s own through many growth
opportunities.
4. Exploring Engine for Future Society
Due to the increasing complexity of the social
structure and an increase in unpredictable events suchas natural disasters, it is becoming difficult to observe18

Feature Articles

Exploring Engine for Future Society
LivingcultureMathematicalmodel ofpopulation
Crowd

Multimodal and counterfactual simulation

Crops

EconomyResources
andsociety Multimodal

Society

Industrial
societyP1modeling

EnergyArtificial
society

Urban Naturalenviron- environments
ment

P2

Artificial society(Digital twins of society)
Real-time
data
assimilation

Unchosenworld

Real world (current)
Behaviors

P3

AA

P4

AB

P5

BA

P6

BB

Results A

Results B

Futureexploration
Real world

Future real-worldgroup

Behaviors
Real world

Future world

Future world

Fig. 3. Exploring Engine for Future Society.

the effects of individual behavior and cooperation onsociety and the natural environment and how thoseeffects come back to affect the individual. Under
these circumstances, it would be desirable to search
for ways that society and lifestyles can coexist withnature. To this end, we are researching and developing an “Exploring Engine for Future Society.” This
engine will divide the society that people are active in
into the three layers of environment, city, and people
and express these layers as digital twins. It will thenuse these digital twins to simulate the transformationof this multilayered society.
To make this engine a reality, we are researchingand developing technology for creating in real time
digital twins of the real world obtainable throughsensing and for modeling abstract information suchas public opinion and individual consciousness. We
intend to use basic technologies such as previously
developed temporal-spatial data management technology. We are also researching and developing basic
simulation techniques to combine the above with cityand environment simulations. These technologies
will make it possible to drive society’s digital twins
and present a variety of future scenarios and candidate measures that should be taken now by calculating backward from an ideal image of society.
Visualizing various forms that future society cantake in this manner should make it possible to bolster
various types of social solidarity by having not only
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

government and corporate management but also local
residents and individual citizens view the same future
images. Additionally, we aim for a world in which
certain types of behavior and cooperation can be proactively searched for and spontaneous actions can be
encouraged so that a society desired by the individual
can be achieved (Fig. 3).
5. Inducing Inclusive Equilibrium Solutions for
the Earth and Its Social and Economic Systems
In view of climate change and social crises now
occurring on a global scale, discussions have begun
on reconstructing social and economic systems thatcan maintain an autonomous equilibrium on the
Earth. This is a problem with a high degree of uncertainty, so it is important to prepare multiple options
and scenarios and study and execute comprehensive
approaches. With this in mind, we have taken up thechallenge of presenting options with regard to thetransformation of energy, the economic system, andhuman society. The aim is to determine changes andmutual effects (chain reactions) among the many different systems making up the Earth, such as the environment, ecosystem, energy, economic systems, andhuman society, and preserve an autonomous equilibrium that can maintain a balanced state in which nosystem will exceed a critical point and collapse.
First, to meet this challenge using as themes the19

Feature Articles

Energy production and use
Climate change

Prosperity

Water-Energy-Food Nexus

Mutual chaining of complex systems(making observations difficult,increasing uncertainty, delaying theimpact of changes, etc.)

Economy and society

Water circulation

Autonomy of global
environment+ Economy and Society

Food
production/disposal

Autonomy of humancontrollable systems

Essential theme
What are the candidates for a transformation of the social system that canmaintain long-term compatibility between these two forms of autonomy?

Fig. 4. Inducing Inclusive Equilibrium Solutions for the Earth and its Social and Economic Systems.

energy supply corresponding to changes in climate
and water circulation along with economic systems
and human society, we will undertake the R&D of (1)large-scale multimodal technology for massive coupling of model groups of many and varied phenomena and simulation of inter-model chain reactions, (2)modeling technology for designing on a global scale
while taking into account inter-system coupling
between the environment and economic systems, and(3) techniques for determining key indices of autonomous equilibrium and local solutions of those indicesand techniques for evaluating index-value groups foreach scenario.
In this way, we will be able to link many and varied
large-scale models and conduct inclusive circulation
simulations on a global scale of diverse and complex
phenomena occurring in the real world throughout
the Earth while presenting multiple options for asocial transformation (Fig. 4).
6. Horizontal activities for the grand challenges
Technology development for meeting the above
grand challenges covers a wide range of fields, so weare engaged in unified development activities acrossthese grand challenges. We introduce three of these
activities: (1) creation of a DTC reference model, (2)NTT Technical Review

Vol. 19 No. 12 Dec. 2021

collaboration with partners, and (3) standardization
activities.
(1) Creation of a DTC reference modelIn creating a DTC reference model that compiles
technical achievements across multiple projects, wehave documented digital-twin concepts common to
DTC while taking a comprehensive view across thefour grand challenges and released the document
both inside and outside NTT laboratories [3]. This
document includes conceptual structures now being
studied at the NTT Digital Twin Computing ResearchCenter with respect to the definition of digital twinsand configuration of human digital twins. The aim of
releasing this information is to promote discussion
with partner companies having a common vision andmake joint explorations of even more essential DTC
concepts. Going forward, we will add more research
results as they become available from the NTT Digital Twin Computing Research Center to provide aneven more complete reference document.
(2) Collaboration with partners
To accelerate the R&D of the above grand challenges through collaboration with partners, we will
search for outside partners to accelerate DTC R&D
and work on creating a collaboration scheme.
Although necessary technology areas differ from one
grand challenge to another, we will pursue a broad
20

Feature Articles

range of partnering, including tie-ups with universities, other research institutions, and startup companies targeting a wide range of fields from artificial
intelligence technologies for understanding humanemotions and behavior to digital-twin technologies
for smart cities. The field of digital twins is relatively
new, but at the same time, it is connected to a wide
range of technology areas such as virtual reality(extended reality, etc.) and digital humans; therefore,
many startup companies with deeply interesting technologies are appearing and growing. It is our belief
that establishing collaborative partnerships with suchcompanies will accelerate R&D and enable the early
implementation of novel technologies.
(3) Standardization activities
Finally, we will engage in standardization activities
to standardize and spread DTC on a global scale. Aspart of this effort, we will promote activities for visu-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

alizing DTC and research results at global forums.
Specifically, we are participating in the Digital Twin
Consortium and the IOWN Global Forum while making contributions and presentations in relation to
DTC and grand challenges. Through these activities,
we hope to increase NTT’s presence in the digitaltwin field and promote standardization activities for
digital-twin technologies together with global partners we can collaborate with in making DTC a reality.
References
[1]
[2]
[3]

IOWN, https://www.rd.ntt/e/iown/
Press release issued by NTT, “NTT Announces New R&D Projects ofDigital Twin Computing,” Nov. 13, 2020.
https://group.ntt/en/newsrelease/2020/11/13/201113c.html
NTT Digital Twin Computing Research Center, “The Digital Twin
Computing Reference Model,” Mar. 2021.
https://www.rd.ntt/_assets/pdf/iown/reference-model_en_2_0.pdf

21

Feature Articles

Ryo Kitahara

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a Master’s degree in information
science from Tohoku University, Miyagi, in 2003
and joined NTT the same year. His researchinterests include image-processing technologies.
Since 2020, he has been in charge of research on
Digital Twin Computing at NTT Digital Twin
Computing Research Center.

Takao Kurahashi

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received an M.E. in industrial engineeringfrom Hosei University, Tokyo, in 2002 and joined
NTT the same year. His research interests
include human-computer interaction and robotics.

Ichibe Naito

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received Bachelor’s and Master’s degrees in
information science from Waseda University,
Tokyo, in 2006. Since joining NTT in 2006, hehas been engaged in R&D of distributed data
stream management systems and personal information management systems. In 2010, he moved
to NTT Communications, where he designed the
database of operation support systems and developed an IaaS (infrastructure as a service) service.
In 2013 he joined NTT Software Innovation
Center and developed operation support systems
for distributed object storage. His current
research interests include efficient managementof huge geospatial data.

Daisuke Tokunaga

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a Master’s degree in information
science from Kyushu Institute of Technology,
Fukuoka, in 1999 and joined NTT the same year.
His research interests include large-scale application system technologies. Since 2020, he has
been in charge of research on Digital Twin Computing at NTT Digital Twin Computing ResearchCenter.

Toru Nishimura

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a Master’s degree in informationengineering from Nagoya University, Aichi, in
1998 and joined NTT the same year. He has been
involved in the R&D of an operation support
system for information technology systems
including networks. Since 2020, he has been in
charge of research on Digital Twin Computing at
NTT Digital Twin Computing Research Center.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Koya Mori

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
Since joining NTT in 2004, he has been
involved in the R&D of an Internet-of-Things
application based on software technologies, suchas OSGi, OpenStack, and edge computing. Since
2020, he has been in charge of research on Digital Twin Computing at NTT Digital Twin Computing Research Center.

22

Feature Articles: Toward a New Form of Coexistence
between the Real World and Cyberworld

Thought-processing Technology for
Understanding, Reproducing, andExtending Human Thinking Ability
Kyosuke Nishida, Takeshi Kurashima, Noboru Miyazaki,
Hiroyuki Toda, and Shuichi Nishioka
Abstract

The Human Insight Laboratory at NTT Human Informatics Laboratories aims to deepen our understanding of human thinking ability, which consists of human information understanding, human information processing, and response and behavior, and to create technologies that can reproduce theseabilities on computers and expand human thinking ability. In this article, the technologies of visualmachine reading comprehension, behavior modeling, speech recognition, and thought-enhancing stimulus design are introduced.
Keywords: human information understanding, human information processing, human insight

1. Introduction
The goal of Human Insight Laboratory of NTT
Human Informatics Laboratories is to deepen our
understanding of (1) human information understanding (from perception to cognition), (2) human information processing (for interpreting acquired information and taking the next action), and (3) reaction andaction (for acting on the outside world). We createtechnologies to reproduce these processes on computers for expanding human thinking ability. We
introduce four technologies that will lead to this goal,
i.e., (i) visual-machine-reading-comprehension technology for understanding documents visually, (ii)behavior-modeling technology for considering themechanism of human decision-making and behavior,(iii) speech-recognition technology for understanding a human’s internal state, and (iv) thought-enhancing stimulus design for drawing out and expanding ahuman thinking ability.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

2. Visual-machine-reading-comprehension
technology for understanding documents visually
To further develop information retrieval and dialogue/question-answering services, the Human
Insight Laboratory has been researching machinereading comprehension, by which artificial intelligence (AI) reads text written in natural language (i.e.,
the language used in our daily lives) and understands
its meaning [1, 2].
Research on machine reading comprehension has
made great progress, and according to evaluation
data, AI has exceeded human reading comprehension; however, machine reading comprehension islimited in that it can only understand textual information. The PDF (portable document format) documents and presentation slides that we use on a daily
basis contain not only linguistic information but alsovisual elements such as font size and color, figures,tables, graphs, and layout information. An integrated
understanding of vision and language is therefore
essential in regard to developing AI to support office
work and daily life.
To achieve visual machine reading comprehension
23

Feature Articles

I watch at least one hour of video every day.
How much is the monthly fee for the “Lite plan”?

Lite plan

Illustration of usage fee
Maximum charge(tax included)
5800 yen/month

Monthly
usage feeTwo-stepflat-rate
plan

The maximum monthly charge is
5800 yen including tax.

Basic charge(tax included)
2500 yen/monthUsage fee increases
as line is used.

200 MB

1000 MB

200-MB rough estimate
The main use is Internet browsing andemail. Use the Internet once every twodays for about 30 minutes.

If you watch videos for one
hour every day, line usage
will exceed 1200 MB.
1200 MB Line usage

1000-MB rough estimate
In addition to browsing and email,watch videos for about one hour per
month. Use the Internet once every
two days for about an hour.

Line usage is about 1000
MB at about one houra month.

Fig. 1. Visual machine reading comprehension.

of document images (Fig. 1), we at the Human InsightLaboratory have constructed a dataset called VisualMRC [3] and are pushing ahead with research using
it. This dataset consists of free-text question-answer
data concerning document images of webpage
screenshots, and regions in documents are annotatedwith nine classes, including title, paragraph, list,
image, and caption. We conducted research using thisdataset and proposed a visual-machine-readingcomprehension model that can take into account—asadditional input—areas in the document (extractedusing object-recognition technology) as well as position and appearance information of “tokens” (extracted using optical character recognition technology)[3]. Although this model is not yet able to match theaccuracy of human question answering, we confirmed that understanding the visual information of
documents improves the performance of question
answering compared with that of text-only models.
We will continue to work on various research projects
for integrated understanding of vision and language.
3. Behavior-modeling technology forconsidering the mechanism of humandecision-making and behavior
We are studying the mechanisms of human deciNTT Technical Review

Vol. 19 No. 12 Dec. 2021

sion-making and behavioral judgments (including
irrational judgments) by using both humanistic andsociological knowledge (such as behavioral economics) and a variety of human data that have become
available with the recent spread of the Internet ofThings. We are also constructing a behavioral modelthat reproduces human decision-making and behavior by using the obtained knowledge as a framework
and aim to apply this model to enable future predictions and simulations about people. When an undesirable future is predicted, we will be able to use thesimulation results to search for ways that will lead toa brighter future.
As part of our efforts to explain the mechanism of
human decision-making, we have been studying theeffects of factors, such as personality, values, physical and psychological states, and social environment,
on a person’s decision-making and behavior. Ecological momentary assessment (EMA), also known as
the experience sampling method, is a survey methodwith which participants are sent simple questions
about their situation, thoughts, emotions, and behavior via a portable device (such as a smartphone ortablet) and asked to spontaneously respond to thequestions. Using the results of a 10-week EMA questionnaire administered to university students [4], we
analyzed in detail the differences in the students’24Feature Articles

Temporal change in response volume

Stress level
High

Low

Decrease in
response volume

Amount of answers

Personal characteristics

Time
Regular test period(a) Analysis of temporal changes in the relationship between personal characteristics (stresslevel), social environment (regular examination), and response volume of EMA

Granting
incentives

High

Low

Increase in
response volume

Amount of answers

Stress level

Time
Regular test period(b) Give an incentive when the amount of responses decreases and encourage an increasein the amount of responses

Fig. 2. Search for strategies on the basis of results of behavioral-data analysis.

tendency to respond to the questionnaire on individual characteristics from the perspective of temporalchanges [5]. For example, as shown in Fig. 2(a),participants who were judged to have high stresslevels on a daily basis (according to a preliminary
survey taken before the start of the EMA questionnaire) tended to actively self-disclose information totheir device throughout the survey period. However,
our analysis revealed that immediately after a stressful event (e.g., a regular exam), the students tended to
be uncooperative toward answering, and their volumeof responses dropped sharply. We also found that theparticipants who were judged to have high integrity inthe preliminary survey were cooperative and gave alarge amount of responses immediately after the startof the EMA questionnaire; however, the amount of
their responses tended to decrease significantly overtime. By deepening our understanding of the tempoNTT Technical Review

Vol. 19 No. 12 Dec. 2021

ral changes in such human behavior (in this case, theact of voluntarily answering a questionnaire), it will
be possible to predict what will happen regarding
people’s behavior and outcomes, and it will be easier
to make decisions on how to change the future for the
better. From the viewpoint of constantly understanding a person’s condition, the EMA questionnaireshould secure a uniform number of responses withoutbias in time or participant. For example, when theamount of responses is expected to drop, as shown in
Fig. 2(b), it is possible to give special incentives toparticipants with high stress levels to encourage themto respond after the test.
The above example is just an analysis that captures
one aspect of complex human behavior. In the future,
we plan to make our human decision-making andbehavioral models [6] more sophisticated by taking into
account behavioral economic human characteristics
25

Feature Articles

Speech-recognition
technology

Target voice

Assembly/businessmeeting voice

Rough spoken language
between people(Voice recognition)

Inquiry-response
voice

Read aloudCreating meeting
minutes and memos

2019~

Mobile- and smartdevice speech
recognition

2013

Presentation (monolog)(Voice recognition)
Free speech of machinesand humans(Voice recognition)

Parliamentary recordmaking system
Voice-dialogue agent

Subtitle-creation systemfor broadcasting stations

Read aloud like announcer(Voice recognition)

Telephone-voice-answering service
Word voice

Word-speech
recognition

2010

Contact-centervoice-mining system

Parliamentary
speechCommand voice

Related services

2008
2003

2003

2002

Watch-type PHS 1998
Financial CTI system

1982

CTI: computer telephony integration
PHS: personal handy-phone system

Fig. 3. Efforts concerning voice-recognition technology at NTT.

such as attitude toward uncertainty (risk-aversion
tendency) and toward waiting (patience). By “sophisticated,” we mean that such a model will be able to
make more “human-like” decisions. We believe that
because a model is “human-like,” it will be possible
to predict and depict a future of society, even an undesirable future.
4. Speech-recognition technology forunderstanding a human’s internal stateWith the advent of voice assistants, which allow
users to operate devices by speaking into a smartphone or AI speaker, speech-recognition technology
has spread rapidly around the world. The practicalapplication of speech-recognition technology for
operating computers by means of spoken dialogues
using short-word exchanges began in the 1980s with
the introduction of interactive voice-response systems. The introductions of car (satellite) navigation
systems in the 1990s and deep-learning technology inrecent years have greatly improved speech-recognition accuracy and spurred the rapid spread of voiceassistants.
Natural communication between people involves
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

longer sentences and more complex linguistic expressions, which are more difficult for speech recognition
to handle, than the short sentences uttered to a voiceassistant. Moreover, when the conversational partner
is familiar, the tone of voice becomes more casualand utterances fragmented, which makes it more difficult to predict both sounds and linguistic expressions. Regardless of those difficulties, recognizing
such conversational speech can be used for applications with significant business value, such as analyzing the content of conversations in call logs stored inlarge quantity at call centers and supporting conversations in real time; accordingly, various speech-recognition technologies based on deep learning are being
actively studied. As shown in Fig. 3, speech-recognition technology has evolved through repeated
improvements in the accuracy at which speech can beconverted to text, expansion of its applications, andincreasing complexity of the speech to be handled.
Information communicated through speech includes
not only verbal information (text information) but
also non-verbal information such as gender, age,emotions, intentions, and attitudes. As well as working on recognizing text information from speech withhigh accuracy, we are investigating technologies for26

Feature Articles

recognizing and using non-verbal information anddeveloping technologies that can extract speaker
attributes (adult male, adult female, and child),
emotions (joy, anger, sadness, and calmness), andquestions and/or non-questions. We are also researching, developing, and practically applying technologies for estimating the degree of a customer’s anger
and/or satisfaction as well as the impression that the
telephone operator’s response gives to that customer
during conversational speech consisting of two
speakers (such as when a customer calls a call center).
The recognition technology for verbal and nonverbal information we are currently investigating is
the first step toward reading the internal state ofhumans in a manner that allows us to offer moreadvanced services. In everyday conversation, we use
various clues, such as voice tone, line of sight, facial
expression, pause length during interaction, andchanges in wording, to estimate information inside a
person’s mind, such as their emotions and interests orfriendliness and indifference, and we use that information to facilitate smooth communication. The recognition of such internal information using a machineis expected to lead to new services that target thesubtleties of the human mind that cannot be expressedin words. Such services might conceivably include avoice-dialogue agent that is attuned to the emotionsof the user, education that responds to the understanding and interests of each student, and detection of apatient’s physical condition and mental stress frommedical-interview dialogue data. It is also expected tocontribute to creating a new type of communication—as targeted by the grand challenge “Mind-toMind Communications” set for Digital Twin Computing (DTC), which is one of the elements of IOWN
(the Innovative Optical and Wireless Network) promoted by the NTT Group, that can directly understand the way people perceive and feel in a mannerthat transcends differences in individual characteristics such as experience and sensitivity.
5. Thought-enhancing stimulus design fordrawing out and expanding a humanthinking ability
Human thoughts and behaviors are not only influenced by information that is interpreted consciously,
such as language and numerical values, but also by
sensations and perceptions that are perceived unconsciously, such as colors, scents, and the atmospherecreated by the manner of speaking. For example,
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

smelling soap makes us want to wash [7] and changing the color of a product package makes us feel thatthe product is different even though it contains thesame thing [8].
We are accumulating knowledge on the relationshipbetween perceptual stimuli and human thought andbehavior, and on the basis of that relationship, we areresearching generation and control of perceptual
stimuli with the aim of drawing out and expanding
human thinking ability. In one of our previous studies, we investigated the effect of the speaker’s speaking style (voice pitch, speaking speed, and inflection)
on a human’s psychological state and behavior [9]. In
that study, we conducted a large-scale subjectiveevaluation experiment on the relationship between
speech that promotes products and people’s consumption behavior and analyzed the relationshipbetween speech and purchase intention by using theconsumer-behavior model developed by Mehrabianand Russell [10] (Fig. 4) that expresses emotion as amediator.
5.1 Experimental procedure
Via crowdsourcing, we asked 202 native speakers(male and female) of Japanese to listen to an advertisement for an electrical appliance spoken in different styles of speech. After listening, we asked them torespond to evaluation items about the emotion andpurchase motivation they felt when listening to the
spoken ad. We then analyzed the relationship amongspeech characteristics, emotions, and purchase intention on the basis of the obtained parameters (voice
pitch, speaking speed, and intonation) related to thead’s speech (voice characteristics) and the responses
to the evaluation items. The promotional text of the ad
was based on speech uttered by a professional female
speaker in a read-aloud tone, where the speechparameters were set as shown in Table 1. The evaluation items are listed in Table 2.
5.2 Results
The results of analyzing the relationship amongspeech characteristics, emotions, and purchase intention—by using a three-layer model based on Mehrabian and Russell’s consumer-behavior model—are
shown in Fig. 5. In line with previous studies [11], theresults indicate that among the emotions, “pleasure”
and “arousal” positively affect purchase intention.
They also indicate that the effect of pleasure is thestrongest, higher voice, faster speech speed, andhigher intonation lead to higher emotion rating, andamong those parameters, speech speed had the greatest
27

Feature Articles

A model that expresses consumer behavior in three steps:
external stimulus  emotion  behavior

External
stimulus

Emotions• Store atmosphere• Background music• Color• Pleasure• Arousal• Dominance

Purchasing
behavior• Purchase• Presence in store• Increased purchaseamount• Favorability toward
stores and products

Fig. 4. Consumer-behavior model.

Table 1. Voice parameters.
Parameters of voice
Voice pitch

Setting value

Average F0 (Hz)Speaking speed

0.9439x, 1x, 1.059x

Speaking speed (mora/s) 0.890x, 1x, 1.1225x

Magnitude of intonation

F0 variance (Hz)0.6667x, 1x, 1.5x

Table 2. Evaluation items.
Evaluation item
EmotionPleasure

Setting value
Displeasure  Pleasure (7 stages)

ArousalSleep  Arousal (7 stages)

Dominance

Obedient  Dominant (7 levels)

Purchase intention“I want to buy it very much.”  “I don’t want to buy it at all.” (7 levels)

effect on the evaluation items.
To summarize these results, we found that (i) the
consumer-behavior model is valid for a speech stimulus based on read-out advertisements and (ii) the
speed of speech has the strongest effect on pleasure,which has the strongest effect on purchase intention.
The above-described efforts represent the analysisof only a small part of the complex and diverse thinking processes of humans. We plan to study a wider
range of factors, such as the relationship among theattributes of listeners with an auditory stimulus, andinvestigate non-auditory stimuli such as sight andsmell. We also plan to study the effects of stimuli on
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

human thinking phenomena—ranging from intuitive
judgments to deeper stages such as effects on logicalthinking and human values. Through these studies,
we want to clarify what is necessary to draw out
human thinking ability, enhance human potential, andcontribute to the creation of a better society.
6. Concluding remarks
By refining the four technologies described in thisarticle, we plan to understand, reproduce, and extend
human thinking ability, which consists of human
information understanding, human information28

Feature Articles

Arrow indicates the
direction of influence.
Number is its magnitude.

External
stimulus

Emotions

Average F0

Pleasure

.08*
.32*

.03*

.59*
Behavior.08*

Speaking
speed

Arousal

.39*

.17*

F0 variance.16*

Purchase
intention

−.04*

.20*

.08*

.36*

Dominance*p < .001

Fig. 5. Results of analysis of relationships among voice characteristics, emotions, and purchase intention.

processing, and response and behavior.
References
[1]

[2]

[3]

[4]

K. Nishida, I. Saito, K. Nishida, K. Shinoda, A. Otsuka, H. Asano, andJ. Tomita, “Multi-style Generative Reading Comprehension,” Proc. of
the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), pp. 2273–2284, Florence, Italy, July 2019.
K. Nishida, K. Nishida, M. Nagata, A. Otsuka, I. Saito, H. Asano, andJ. Tomita, “Answering while Summarizing: Multi-task Learning forMulti-hop QA with Evidence Extraction,” Proc. of ACL 2019, pp.
2335–2345, Florence, Italy, July 2019.
R. Tanaka, K. Nishida, and S. Yoshida, “VisualMRC: Machine Reading Comprehension on Document Images,” Proc. of the 35th AAAI
Conference on Artificial Intelligence (AAAI 2021), pp. 13878–13888,
Feb. 2021.
R. Wang, F. Chen, Z. Chen, T. Li, G. Harari, S. Tignor, X. Zhou, D.
Ben-Zeev, and A. T. Campbell, “StudentLife: Assessing Mental
Health, Academic Performance and Behavioral Trends of College
Students Using Smartphones,” Proc. of the 2014 ACM International
Joint Conference on Pervasive and Ubiquitous Computing (UbiComp

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

2014), pp. 3–14, Seattle, USA, Sept. 2014.
T. Tominaga, S. Yamamoto, T. Kurashima, and H. Toda, “Effects of
Personal Characteristics on Temporal Response Patterns in EcologicalMomentary Assessments,” Proc. of the 18th IFIP TC13 International
Conference on Human-Computer Interaction (INTERACT 2021),
Bari, Italy, Aug./Sept. 2021.
[6] T. Kurashima, T. Althoff, and J. Leskovec, “Modeling Interdependent
and Periodic Real-world Action Sequences,” Proc. of the 2018 World
Wide Web Conference, Lyon, France, pp. 803–812, Apr. 2018.
[7] K. Liljenquist, C. B. Zhong, and A. D. Galinsky, “The Smell of Virtue:
Clean Scents Promote Reciprocity and Charity,” Psychological Science, Vol. 21, No. 3, pp. 381–383, Mar. 2010.
[8] K. Maki, “Color Studies for Color Design,” Ohmsha, 2006 (in Japanese).
[9] M. Nagano, Y. Ijima, and S. Hiroya, “Impact of Emotional State on
Estimation of Purchase Intention from Advertising Speech,” Proc. of
INTERSPEECH 2021, Brno, Czech Republic, Aug./Sept. 2021.
[10] A. Mehrabian and J. A. Russell, “Approach to Environmental Psychology,” The MIT Press, 1974.
[11] R. Donovan and J. Rossiter, “Store Atmosphere: An Environmental
Psychology Approach,” Journal of Retailing, Vol. 58, No. 1, pp.
34–57, 1982.
[5]29

Feature Articles

Kyosuke Nishida

Distinguished Researcher, Human InsightLaboratory, NTT Human Informatics Laboratories.
He received a B.E., M.I.S., and Ph.D. in information science and technology from Hokkaido
University in 2004, 2006, and 2008 and joined
NTT in 2009. He received the Yamashita SIG
Research Award from the Information Processing Society of Japan (IPSJ) in 2015 and theKambayashi Young Researcher Award from the
Database Society of Japan (DBSJ) in 2017. His
current interests include natural language processing and artificial intelligence. He is a member of the Association for Computing Machinery(ACM), IPSJ ,the Association for Natural Language Processing (NLP), the Institute of Electronics, Information and Communication Engineers (IEICE), and DBSJ.

Takeshi Kurashima

Distinguished Researcher, Human InsightLaboratory, NTT Human Informatics Laboratories.
He received a B.S. from Doshisha University,
Kyoto, in 2004 and M.S. and Ph.D. in informatics from Kyoto University in 2006 and 2014. He
joined NTT in 2006. He was a visiting scholar at
Stanford University, USA from 2016 to 2017.
His current research interests include data mining, machine learning, and recommender systems. He is a member of IPSJ, IEICE, DBSJ, andACM.

Hiroyuki Toda

Senior Research Engineer, Supervisor, Human
Insight Laboratory, NTT Human Informatics
Laboratories.
He received a B.E and M.E. in materials science from Nagoya University in 1997 and 1999,
and Ph.D. in computer science from Universityof Tsukuba in 2007. He joined NTT in 1999. His
current research interests include information
retrieval and data mining. He is a member of
IPSJ, IEICE, JSAI, DBSJ, and ACM.

Shuichi Nishioka

Executive Reserch Engineer, Supervisor,
Human Insight Laboratory, NTT Human Informatics Laboratories.
He received a B.E. and Ph.D. in engineeringfrom Yokohama National University, Kanagawa,in 1995 and 2005. He joined NTT in 1995. His
current interests include data engineering andartificial intelligence. He is a member of IPSJ
and DBSJ.

Noboru Miyazaki

Senior Research Engineer, Supervisor, Human
Insight Laboratory, NTT Human Informatics
Laboratories.
He received a B.E. and M.E. from Tokyo Institute of Technology in 1995 and 1997. Sincejoining NTT in 1997, he has been engaged inresearch on spoken dialogue systems, speechsynthesis, and speech recognition technologies.
He received the IEICE Inose Award in 2001. He
is a member of the Acoustical Society of Japan,
IEICE, and the Japanese Society for Artificial
Intelligence (JSAI).

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

30

Feature Articles: Toward a New Form of Coexistence
between the Real World and Cyberworld

Research on Symbiotic Intelligence
for Achieving Symbiosis among
People and between People andMachines
Tetsuya Kinebuchi, Taichi Asami, Sen Yoshida,
and Ryuji Yamamoto
Abstract

With the rapidly progressing fusion of the real and cyber worlds, the manner of symbiosis amongpeople and between people and machines is changing significantly. Information and communication
technologies will make it possible to gently guide the behavior of people and groups toward a better
future and shared emotions and to enable people and machines to work together through more-naturaland complex interactions. An overview of the research being conducted by the Symbiotic Intelligence
Research Project toward this future vision is given in this article.
Keywords: behavioral modeling, knowledge processing, emotional perception control

1. Introduction
As the real and cyber worlds rapidly converge, themanner in which people work and live is changing
dramatically. Meetings and collaborative work thatused to be done face-to-face in real spaces, such asclasses at schools and social gatherings, are now done
remotely, and social networking services (SNSs) arebecoming the mainstream for chatting and gossip.
Sporting events and theatrical performances, whichused to require a direct trip to a stadium or theater,
can now be richly experienced in cyberspace. The
novel-coronavirus (COVID-19) pandemic has spedup the merging of the two (real and cyberspace)worlds.
We believe that by using the power of informationand communication technologies, all people, andeven people and machines, will be able to coexist
more prosperously than ever before, and that thiscoexistence will enable people to experience evengreater well-being than before. For example, we will
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

be able to gently guide people and groups toward abetter future and shared emotions or work together
with machines that have a similar inner self to that ofpeople through more natural and complex interactions. With this vision of the future in mind, we areresearching and developing technologies that willcontribute to (i) modeling and simulating humanbehavior, especially the behavior (thoughts, actions,etc.) of groups, and (ii) predicting and optimizing
future social activities in which people as well as
people and machines collaborate and live together.
Specifically, this research and development (R&D) isbased on the following three themes.
1)	Group psychological and behavioral modeling:
Modeling of psychological, judgmental, andsocial behaviors of people and their groups andsimulation and optimization of those behaviors
on the basis of the models.
2)	Emergence and social brain: Knowledge processing that (i) accurately understands andconverts the textual content into knowledge and31

Feature Articles

Behavior of people(individuals and groups)

Intervention• Health promotion• Purchase• Movement• Comments on SNSs etc.
• Disaster prevention

Future prediction and measuresfor behavioral change• Message• Incentive• NudgeAnalysisand insight

Knowledge of
behavioral
economics andsocial psychology

Behavior model

Simulation andoptimization

Fig. 1. Research initiatives on group psychology and behavior modeling.

(ii) provides the information required for collaboration with people in a natural exchange.
3)	Emotional perception control: Explanation andsensing of an internal model of human emotional psychology (enthusiasm, sense of unity,etc.), and collaborative-work support to resolve
communication discrepancies by empathizing
with individual interpersonal characteristics.
Hereafter, our efforts concerning these three themes
are introduced.
2. Group psychological and behavioral modeling
We are conducting R&D on the modeling of humandecision-making and behavior. People make various
daily decisions, such as what to eat for dinner, what
to buy, whether to exercise, where to go for pleasure,and what to post on SNSs, that dictate their social
activities. To predict the future of social activities andengender ever greater well-being, modeling and computer simulation of human decision-making andbehavior are indispensable technologies. For example, if we could model behavior and simulate howthat behavior changes with interventions such as messages and incentives, we could create apps and services that enable users to naturally make healthier,safer, or more enjoyable choices. We are analyzing
human behavior, modeling and simulating thatbehavior, and deriving measures to change behavior
on the basis of the models (Fig. 1).
Application of various mathematical-modeling andbig-data-based optimization techniques, including
deep learning, is a promising approach to model andsimulate behavior. However, it is not easy to collect alarge amount of data on the behavior of a wide variety
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

of people. We believe that to elucidate the mechanisms of human decision-making and behavior, amore in-depth approach than simply applying current
mathematical models is necessary.
Human behavior has been studied in the fields of
behavioral economics and social psychology, and agreat deal of knowledge has been accumulated. At
NTT Human Informatics Laboratories, by constructing new mathematical models that incorporate thisknowledge and verifying hypotheses, we aim toclarify the mechanism of human decision-making
and behavior and implement efficient and highly
accurate computer simulation.
3. Emergence and social brain
We are researching and developing knowledge processing that (i) accurately understands and converts
the textual content into knowledge and (ii) provides
the information required for collaboration with people in a natural exchange. Language modeling is afundamental technology for accurately understanding and converting textual content into knowledge.
Regarding Japanese, a language model is a representation of “the essence of the Japanese language” in aform that is easy for a computer to handle. Language
modeling has been dominated by methods for creating deep-learning models through large-scale pretraining from huge amounts of text data.
We are also constructing large-scale pre-training
language models for Japanese and conducting R&D
to further improve their performance. Through these
efforts, we aim to build a foundation for linguistic
communication to enable collaboration among people as well as between people and machines while32

Feature Articles

Investigate theproperties ofsubstance A

ReportFig. 2. Collaboration between people and machines (automatic generation of reports).

contributing to various language-processing-related
services of the NTT Group.
One of the most-promising use cases for collaboration between people and machines in regard to textual content is automatic document summarization.
We are developing a technology that enables highly
accurate summarization using large-scale pre-training language models. This technology can, for example, generate a summary that includes keywords
specified by the user and allows the length of thesummary to be adjusted in accordance with the application. We believe that if we can further develop thistechnology to automatically generate reports concerning tasks such as examining a large amount ofliterature (medical care and intellectual property), it
would generate great value (Fig. 2).
Collaboration would be more natural if people
could talk to machines in the same language as theytalk to other people. Technologies for interacting withusers in natural language are called dialogue systems
and can be broadly classified as task-oriented (which
perform specific tasks) and systems that conduct socalled “chat.” We are researching and developing
chat-oriented dialogue systems; in particular, we aredeveloping technology to add character to such systems so that users feel familiar with them.
4. Emotional perception control
We are researching and developing technology for
improving the quality of symbiosis as a group by (i)understanding the characteristics of people from perception to emotion and (ii) intervening in communication between and among people in accordance withthese characteristics. Specifically, we are developing
the following two technologies.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

1)	Emotional-perception-control technology:
Technology that expands communication (e.g.,fostering a sense of unity and enthusiasm
through sharing of emotions) by understanding
the characteristics of human emotional expressions, estimating emotional characteristics
through sensing and data analysis, and controlling emotions with perceptual stimuli tailoredto emotional characteristics.
2)	Collaborative-work-support technology: Technology for improving the quality of collaborative work within a group by understanding thecommunication characteristics of people andeliminating communication discrepancies by
converting information tailored to those characteristics in a manner that induces behaviors thatfacilitate communication.
Regarding emotional perception control, we aredeveloping emotion-estimation technology for estimating the emotions of individuals and groups(groups, crowds, etc.) by using biometric data, images, sounds, content, etc. and emotion-control technology for changing such emotions to a desirable state inaccordance with the estimation results (Fig. 3).
Regarding collaborative-work support, which
focuses on psychological safety as an important factor in improving the quality of collaborative work ingroups, the aim is to make it easier for people withdiverse characteristics to work together by enhancing
their psychological safety.
For this purpose, we are developing a technologythat can improve psychological safety, namely,
reduce communication discrepancies, by understanding the communication characteristics of individuals
and converting verbal and non-verbal information inaccordance with those characteristics (Fig. 4).

33

Feature Articles

We are developing an emotion-control
theory that uses psychological findings
from studies on emotional transformation
as an emergence from the self andcontagion from groups and crowds andexamining interaction techniques.

Interaction

Emotion-control
technology

Individuals, groups, and crowds

Results of emotion
estimation

Emotion-estimation
technology
Develop machine-learning technology
to estimate the emotions of individuals
or groups from data such as biometric
information, images and sounds,
and content.

Sensing

Fig. 3. Overview of technology for controlling emotional perception.

Understanding
personalcommunication
characteristics

Example: Serious look

Conversion of information in accordancewith communication characteristics

Model
Adjust facial
expression“It looks like I’mglaring, but I’mactually listening
seriously.”

Process facial expression
to soften it“I understand.”

Listener

Psychological safety isensured and you can say
what you think.

SpeakerFig. 4. Overview of technology for supporting collaborative work.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

34

Feature Articles

Tetsuya Kinebuchi

Senior Manager, Symbiotic Intelligence Laboratory, NTT Human Informatics Laboratories.
He received an M.S. in physics from Tohoku
University, Miyagi, in 1997 and joined NTT
Corporation the same year. His research interests
include media processing and artificial intelligence (AI). He is a member of the Institute ofElectronics, Information and Communication
Engineers (IEICE) of Japan.

Taichi Asami

Senior Research Engineer, Symbiotic Intelligence Laboratory, NTT Human Informatics
Laboratories.
He received a B.E., M.E., and Ph.D. in engineering from Tokyo Institute of Technology in2004, 2006, and 2019. He joined NTT Corporation in 2006. His research interests include media
processing, AI, and behavioral modeling. He is amember of the Institute of Electrical and Electronics Engineers (IEEE), IEICE, the Information Processing Society of Japan (IPSJ), and the
Acoustical Society of Japan (ASJ).

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Sen Yoshida

Senior Research Engineer, Supervisor of Symbiotic Intelligence Laboratory, NTT Human
Informatics Laboratories.
He received a B.E. and M. Info. Sci. fromTohoku University, Miyagi, in 1993 and 1995.
He joined NTT Corporation in 1995. His
research interests include natural language processing and AI. He is a member of IPSJ and theJapanese Society for Artificial Intelligence(JSAI).

Ryuji Yamamoto

Senior Research Engineer, Supervisor of Symbiotic Intelligence Laboratory, NTT Human
Informatics Laboratories.
He received an M.E. from Kyushu Institute ofTechnology, Fukuoka, in 1998. He joined NTT
Corporation in 1998. His research interests
include media processing and human-computerinteraction.

35

Feature Articles: Toward a New Form of Coexistence
between the Real World and Cyberworld

Cybernetics Technologies for
Symbiosis between People andMachines
Yushi Aono, Hitoshi Seshimo, Narimune Matsumura,
Yukio Koike, and Masaaki Matsumura
Abstract

With the development of computing technology and artificial intelligence, an environment in which
we can use advanced information and communication services in our daily lives is rapidly being established. By installing mechanisms that recognize and understand in a more human-oriented manner thanconventional technologies, we aim to create a society in which people and machines can coexist in harmony. In this article, we introduce our research themes for achieving this aim.
Keywords: symbiosis between people and machines, motion support, embodied knowledge

1. Toward symbiosis between people
and machines
As the computational efficiency of processers has
increased significantly, the accuracy of image processing and recognition has improved; technologythat handles cross-modal information, such as various
sensors, has been refined; and automated driving ofcars, which until now has only been depicted in science fiction, is gradually being achieved. The
improvement in voice recognition, character recognition, and context understanding makes it possible for
automatic-response engines, such as chatbots, to support counter services (contact persons), and people
now carry around mobile devices (such as smartphones) with high-performance processors. These
factors have led to rapid establishment of an environment in which we can use advanced information andcommunication technology (ICT) services in ourdaily lives. One of the technologies that contributes
significantly to improving the accuracy of recognition and understanding is artificial intelligence (AI),which is constructed by machine learning using large
amounts of real data.
Current mainstream AI technology is akin to modNTT Technical Review

Vol. 19 No. 12 Dec. 2021

eling logical and objective recognition and understanding that works like the human left brain based on
our experience and memory, and it has enriched ourlives mainly by targeting things that can be expressedin words and those that can be symbolized such asimages and sensors data.
We aim to create a society that enables people andmachines to coexist in harmony (Fig. 1) by updating
conventional AI technology with the following two
steps:(1)	Elucidating the processing and action processes that work like the human right brain forthings that cannot be expressed in symbols(such as sensations and actions) and for thosethat are subjective and cannot be uniquely
expressed (such as judgments that differ fromperson to person)(2)	Installing the elucidated mechanisms that provide more human-oriented recognition andunderstanding into conventional AI technology
To achieve this aim, we introduce the following
three technologies from our research themes:•	
Motion support that supports motion by sensing
human biological signals with high accuracy
36

Feature Articles

Current mainstream AI technologies

The technologies we are researching

Voice
Sound

Sensation
Dog

Cat

Inspiration

Image

Motion• Things that can be put into words(including numbers)• Things that can be symbolized andconceptualized

Left
brain

Right
brain

• Things that cannot be put into symbols• Things that cannot be uniquelydetermined (subjectivity)
Modeling the mechanism of
human-oriented recognition
and understanding

Modeling of logical and objective
recognition and understanding basedon our experience and memory
Use of advanced ICT
service in daily life

Provide services, especially for thedomains that require flexible and fuzzy
human-oriented support

Fig. 1. Technologies we are researching.

• Embodied knowledge that supports the acquisition and demonstration of non-verbal and empirical skills•	
Zero latency that controls the perceived time
when a person operates a remote-controlled
robot
2. Motion support
Cybernetics, proposed by Norbert Wiener in 1948,
is a unified approach to communication and control
concerning living organisms and machines. Paying
renewed attention to this concept, we are developing
technologies to support the motion of people.
Human motion is executed when motor instructions
from the brain are transmitted to the muscles, whichthen contract on receiving the instructions. The brain
then perceives and recognizes the result as a stimulus
through sensory organs and repeats planning a new
motion and sending its appropriate instruction. We
are attempting to support this communication andcontrol in the human body by sensing and feedingback biological signals such as electroencephalography (EEG) and electromyography (EMG).
More specifically, we aim to create a society inwhich people can move (live) better by reproducing(transcribing) their current motor skills temporarily
and permanently on the basis of the motions of others(skilled people) and their former selves. For example,
we are considering (i) training that uses electricalmuscle stimulation (EMS) to reproduce the muscle
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

motions of skilled people executing various skills andsports and (ii) rehabilitation based on records of one’s
motions in daily activities.
We are working on enabling the following fourfunctions, which we consider necessary for reproducing (transcribing) human motor skills by using biological signals (see Fig. 2).
•	Measuring biological signals and constructing
kinematic models: Bio-signals (EEG, EMG,
etc.) of the transcription source are sensed and akinematic model representing the relationshipbetween the bio-signals and the motion is constructed.
•	Generating and providing sensory stimulation:
Sensory stimuli (EMS, vibration, exoskeleton,etc.) that control the motion of the transferee are
generated and expressed on the basis of the generated kinematic model.
•	Estimating dynamics: The motion results andbiological signals of the transferee (which wasstimulated and moved) are sensed, and statessuch as posture and muscle contraction are estimated.
•	Adjusting sensory stimulation parameters: The
parameters of sensory stimuli are adjusted toattain better motor results and natural motor
experiences on the basis of the estimation results
concerning the motion state and kinematic
model. New feedback based on the results is thenprovided to the transferee.
We had been working on bio-sensing technology
37

Feature Articles

Reproduction of motor skills(transcription)
Biologicalsignals

Transcription
source(expert)Measuring
biological signals
and constructing
kinematic models

Adjustingsensory
stimulation
parameters

Generating
and providing
sensory
stimulation

Estimating
dynamics

Providingstimulation

Measuringoutcomes

Transfer destination(beginner)

Fig. 2. Efforts to reproduce (transcribe) human-motion ability by using biological signals.

(using, for example, EMG) and feedback technology
using tactile and EMS to the human body. Mechanisms of human motion, however, are complex, andvarious efforts are underway in the fields of physiology, brain science, and biomedical engineering tosupport and enhance motion, so an interdisciplinary
approach is necessary. Looking to the future andusing our accumulated technological strengths, weare planning to acquire new knowledge about themechanisms of motion through creation of technology and interdisciplinary collaboration with internaland external experts. For a society in which people
and machines coexist at a higher level, we will elucidate human processing and action processes related
to motion and expand human capabilities.
3. Embodied knowledge
In human life and social activities, knowledge is thebasis for improving and refining the quality and efficiency of people’s behavior and thinking. It is therefore important for people living in the present toacquire knowledge that has been accumulated over along period and apply it in various situations in their
daily lives.
The first thing that comes to mind when people
think of acquiring knowledge is studying at school orlearning a subject. Such already systematized anddescribed knowledge is called explicit knowledge,
and everyone has experienced the widely practiced
methods of acquiring and demonstrating such knowledge. However, knowledge does not mean only the
explicit type. Another type is the knowledge thatpeople learn and express that is difficult to systemNTT Technical Review

Vol. 19 No. 12 Dec. 2021

atize or describe, and it can only be acquired throughexperience rather than through mental understanding.
This type is what we call embodied knowledge.
Regarding learning to play a musical instrument,
people generally refer to instructional books that systematically describe how to handle the instrument, its
functions, and how to move the fingers. Therefore, if
you read all of this instructional material and understand it, will you be able to play an instrument? Of
course, the answer is “no.” In other words, only bymoving your body (fingers, in this case) and practicing with that understanding in mind will you be able
to play. This reality indicates the existence of embodied knowledge, which is the knowledge rooted in aphysical activity, as opposed to explicit knowledge
expressed in textbooks (Fig. 3).
Embodied knowledge is largely based on the senses
and subjectivity of people, and it is far from being
supported by ICT for acquiring and demonstrating in
the manner of explicit knowledge, which is objective
and easily verbalized. However, with the recent evolution of technology, it has become possible to sense
and process information at high speed in a mannercloser to the sensations received by the people
involved. Accordingly, we are conducting research on
embodied knowledge from the perspective of neocybernetics, which captures the mechanism of actions
from the starting point of the person’s cognition. Taking windsurfing as a subject, we are clarifying themechanism of acquiring and demonstrating embodied knowledge in sports and making it possible tosupport this mechanism. With this in mind, we are
studying an analysis method to find the latent pattern
between the physical activity of the athlete, behavior
38

Feature Articles

Knowledge acquired through learning
Explicit knowledge

Embodied knowledge• Linguistic knowledge• Understanding of systematic and sequential teaching• Non-verbal skills• Sensory and experiential (training) acquisition

Linguistic teaching(text books)

Knowledge, logic(understanding
via the brain 
physical motion)

Understanding

ExperienceNon-verbal teaching(practice)“I can do it” (mybody moves)
Emotion, surprise,interest

Fig. 3. What is embodied knowledge?

[Tools]Behavior ofwindsurfing[Nature]Motion of wind[Nature]Motion of waves[Embodiedknowledge]
Relationship between
nature/tools andphysical behavior
and patterns

Value of embodied knowledge (1):Acquisition and demonstration ofembodied knowledge
“Can do,” “Get better,” etc.

Value of embodied knowledge (2):New experience of competitions
and watching sports events
Amplification of enjoyment/
arousal of interest
Change in thinking and senseof values[Embodiedknowledge]
Communication
through non-verbal
expression/affordancesFig. 4. Embodied knowledge concerning windsurfing.

of the natural environment and tools, and performance of the athlete. Moreover, using the dataacquired from actual competitors, we are also studying how to communicate the discovered pattern toother people and how to improve the value (acquisition and demonstration of embodied knowledge orchanges in interest in the sport and the natural environment) provided to the recipients of the communication (Fig. 4).
Although sports is typical in the study of embodied
knowledge, the aforementioned skills, which areknowledge acquired and demonstrated in strong connection with physical actions, such as playing a musical instrument, drawing a picture, or making a goodpresentation, exist in a variety of everyday situations.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Accordingly, we are aiming to create a society inwhich anyone can acquire and apply embodied
knowledge, which has been confined within certain
people, through ICT based on the perspective of theperson who wants to acquire that knowledge while
simultaneously studying areas other than sports.
4. Zero latency
Physical work by remotely operated robots is promising for solving the labor shortage in the service
industry, such as nursing and caregiving, and for
ensuring safety in dangerous work, such as at medical
institutions and disaster sites, during a pandemic.
However, such remote operation always involves a39

Feature Articles(1) Predictionof operator’s
motion(2) Motion
correction by
prediction(4) Cognition
correction by
prediction(3) Predictionof changes in
the workplace

Fig. 5. Four technologies for achieving zero latency.

delay between the operator and robot, and that delay
is known to have adverse effects such as work errors,
reduced work efficiency, and operator fatigue. It canthus be said that while the sophistication of remotely
operated robots has increased their practicality, theeffects of delays are becoming impossible to ignore.
The delay is caused by not only the communication
but also the operation of the camera, display servo,
etc., and the control of these systems. Even if a lowlatency system is used and remotely controlled over a
very short distance, a round-trip delay (namely, thesum of the delay between the incidence of light at thecamera and screen display to the operator and thedelay between the operator’s operation and actuator’soperation) of 100 ms or more occurs.
We therefore aim to develop a remote-operation
system that can fully demonstrate the capabilities ofthe operator by eliminating the negative effects ofdelays through four technologies: prediction of theoperator’s motions, prediction of physical changes in
the work area, correction of the robot’s motions, andcorrection of the operator’s perception (Fig. 5).
One example of predicting the operator’s motion
and correcting the robot’s motion is predicting the
motion of the operator several hundred milliseconds
in the future and moving the robot before the operator
performs the motion in accordance with the prediction. Therefore, the effect of delay is reduced (Fig. 6).
However, it is possible that the robot’s motion may be
wrong (namely, unintended by the operator), so weare currently working on improving prediction accuracy by estimating the operator’s intentions on thebasis of the content of the work and motion of
objects.
As an example of predicting physical changes in
the work area and correcting the robot’s motion, we
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Fig. 6. E xperimental system for predicting the operator’s
motion and correcting the robot’s motion.

simulated a robot’s physical structure to determine
how it moves according to the operator’s motions.
Presenting the simulation results to the operator
through augmented reality (AR) and other means
allows the operator to recognize the results of therobot’s motions before the robot actually moves,which reduces the effects of delays (Fig. 7). Since theoperator will be working on the basis of AR-based
predictive information, in contrast to direct work, weare currently studying how to present information to
support the operator more effectively during various
tasks.
5. Future developments
We are currently refining these technologies on an
individual basis. In the future, we will unify thesetechnologies to provide services especially for40

Feature Articles

Display the arrival point of
the robot arm on the basisof the prediction result
Pen nib

AR guideFig. 7. Experimental system for cognitive correction using AR.

domains that require flexible and fuzzy human-oriented support.

Yushi Aono

Yukio Koike

Executive Research Engineer, Supervisor,
Laboratory Manager, Cybernetics Laboratory,
NTT Human Informatics Laboratories.
He received a B.E. in biotechnology and M.E.
and Ph.D. in control engineering from Osaka
University in 1994, 1996, and 1999. Since joining NTT in 1999, he has been engaged inresearch on speech synthesis, speech recognition,and spoken dialogue systems. He is a member of
the Institute of Electronics, Information andCommunication Engineers (IEICE) and theInformation Processing Society of Japan.

Senior Research Engineer, Cybernetics Laboratory, NTT Human Informatics Laboratories.
He received an M.S. in physics from Nagoya
University in 2004 and joined NTT the same
year. His research interests include network technology for Internet-of-Things services and biological signal processing. He is a member of
IEICE.

Hitoshi Seshimo

Masaaki Matsumura

Senior Research Engineer, Supervisor, Cybernetics Laboratory, NTT Human Informatics
Laboratories.
He received a B.E. and M.E. in mechanicalengineering from Waseda University, Tokyo, in
1995 and 1997 and joined NTT in 1997. His
research interests include computer aided
instruction, web-based learning, content distribution and navigation systems, and geographicalinformation services.

Senior Research Engineer, Cybernetics Laboratory, NTT Human Informatics Laboratories.
He received a B.E in design from Kyushu
Institute of Design, Fukuoka, and M.E. in designfrom Kyushu University, Fukuoka, in 2005 and2007 and joined NTT in 2007. His research interests include computer graphics, computer science, image/video coding, image processing,computer vision, data science, and machinelearning.

Narimune Matsumura

Senior Research Engineer, Cybernetics Laboratory, NTT Human Informatics Laboratories.
He received a B.E. and M.E. in information
science from Kyushu Institute of Technology,
Fukuoka, in 2002 and 2004 and joined NTT in
2004. His research interests include humanmachine interaction, robotics, and digital transformation of nursing and care.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

41

Feature Articles: Toward a New Form of Coexistence
between the Real World and Cyberworld

Elemental Technologies for
Supporting the Integration of RealSpace and Cyberspace
Jun Shimamura, Shingo Ando, and Ryuichi Tanida
Abstract

To create a human-centered society in which everyone can lead a comfortable, vibrant, and highquality life, we hope that a system that integrates real space and cyberspace will become a reality. In thisarticle, a mobile mapping system (MMS) point cloud data processing technology called geoNebulaTM is
introduced, which analyzes data measured in real space, recognizes space and objects, and compresses
the data into a compact form suitable for constructing cyberspace. This technology is being researchedand developed to establish a mechanism for creating value through the sophisticated blending of real
space and cyberspace.
Keywords: cyberspace, real space, MMS point cloud data processing technology

1. Building cyberspaceWith the development of information and communication technologies, it is becoming possible to collect and analyze vast amounts of data from the Internet of Things. In response, the Japanese government
and various companies are pursuing research anddevelopment to create a system that integrates real
space and cyberspace in the manner advocated inSociety 5.0. NTT has started research and development of the 4D digital platformTM, which can integrate four-dimensional (4D) information (latitude,
longitude, altitude, and time) as precisely as possible
in real time and make future predictions from thosedata to implement the above system in society.
The 4D digital platform integrates sensing data
about people and objects on the Advanced Geospatial
Information Database with high accuracy and containing rich semantic information in a manner thatenables advanced prediction, analysis, simulation,and optimization in various industrial fields (Fig. 1).
We believe that this platform has many possible uses
in areas such as alleviation of road traffic, optimal useof urban assets, and maintenance and management ofsocial infrastructure. To develop the Advanced GeoNTT Technical Review

Vol. 19 No. 12 Dec. 2021

spatial Information Database, it is necessary to convert real space into data by using technology such as
laser-imaging detection and ranging (LiDAR) andconstruct cyberspace with those data. This process isgenerally composed of the following three steps.
Step 1 (data collection): Real space measured using
LiDAR, etc. is converted into data.
Step 2 (space and object recognition): Recognize
space and objects from the measurement data andobtain their exact positions.
Step 3 (data storage): Store the finished data andoriginal data for updating or other uses.
The latest research results concerning geoNebulaTM—a mobile mapping system (MMS) point clouddata processing technology we are developing to savelabor and improve the efficiency of each step—are
introduced hereafter.
2. Real-space structuring technology thatenables labor saving and efficiency improvementof data collection and space/object recognition
With our research and development, we aim toestablish the following two technologies for structuring real space from 3D point clouds measured using
42

Feature Articles

4D digital platformTM
Collection of highprecision sensing
data
Sensing data

High-precision realtime correction
technology forvehicle position

Cloud-based
GNSS positioning
technology

Integration and high-speed
processing of huge amount of data

Existing map data

High-precision 3D
spatial information

Development of Advanced
Geospatial Information Database

Value provision forvarious industries

Smoothing road-traffic flow

Use of urban assets

Optimization simulation andfuture prediction basedon integrated data
Cooperative maintenance ofsocial infrastructures

geoNebulaTM: MMS
point clouddata processing
technology

Sensing changes in environment

GNSS: global navigation satellite system

Grass
3D data

High-resolution 3D data

Grass

GroundWood

Wall

ColumnarobjectFig. 1. Overview of 4D digital platformTM.

Result of scene analysis

3D data resolution enhancement

3D scene understanding

Create high-resolution sparse 3D data(measured using LiDAR) by using
camera images and videos as clues

Analyze and understand the entire scene(including complex shapes) from dataobtained from LiDAR and images

Fig. 2. Real-space structuring technology.

LiDAR, etc. (Fig. 2).
(1) 3D data resolution enhancement(2) 3D scene understanding
3D data resolution enhancement is positioned as
step 1 (data collection) in the three steps for constructing cyberspace listed at the end of the previous
section. Due to the structure of a 3D point cloud,vacant spaces (gaps) of a certain width are createdbetween points that are spatially adjacent to eachother. This technology (3D data resolution enhancement) can acquire higher-definition 3D point clouds
by generating multiple new points to fill these gaps
and interpolating them [1]. 3D scene understanding is
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

positioned as step 2 (space and object recognition). It
is capable of automatically assigning semantic labels(road surface, building, etc.) to all points in a dense
point cloud on a large scale and with high accuracy ona point-by-point basis [2]. These two technologies are
explained in detail as follows.
2.1 3D data resolution enhancement
Less expensive LiDAR equipment tends to produce
larger gaps between points. In such a case, much of
the 3D-shape information of an object is lost, and theobject’s detailed features are obscured. The resolution of camera images has significantly increased, so
43

Feature Articles

it is now possible to acquire relatively dense 2D data
even with inexpensive cameras. Therefore, if theinformation from the camera images taken separately
is used appropriately, it would be possible to compensate for a sparse 3D point cloud. Although several
technologies for this purpose are on the market, it isnecessary to strictly align (calibrate) the LiDAR point
cloud with the camera image as a prerequisite. Aligning the two sensors (LiDAR and camera) is generally
difficult and requires a high level of skill. We therefore proposed a method of generating a stable anddense point cloud even with a relatively simple roughcalibration. What sets our method apart from othermethods is not only installation cost (low cost ofequipment) but also operation cost (simple calibration) are accounted for.
2.2 3D scene understanding
Several deep neural networks (DNNs) that can beapplied to non-grid data such as 3D point clouds havebeen proposed. 3D scene understanding is one such
DNN for non-grid data, and we proposed using it tosolve a task called semantic segmentation [2].
Semantic segmentation of a 3D point cloud is theprocess of separating all points in the point cloud into
multiple segments by assigning meaningful labels toeach point. By pre-training with a DNN, it is possible
to estimate the label that should be assigned to eachpoint in an unknown 3D point cloud. However, mostconventional methods target point cloud data within alimited space such as indoors. Accordingly, to handle
a wide area such as an entire city with a high densityof point clouds, a high-spec computer is required.
With that issue in mind, we devised a method for
efficient semantic segmentation of large point clouddata even on a relatively inexpensive computer [2].
Combining this method with the aforementioned 3D
data resolution enhancement technology makes it
possible to acquire high-density point clouds inexpensively and understand an entire 3D scene in detail.
3. 4D point cloud coding technology that enables
highly compressed storage of 3D point
cloud data
4D point cloud coding is a technology that highly
compresses and stores 3D point cloud data obtained
from LiDAR and stereo cameras [3]. This technology
targets 3D point cloud data concerning, for example,
roads and cities collected using the Advanced Geospatial Information Database, and we are researchingand developing technology for highly efficient comNTT Technical Review

Vol. 19 No. 12 Dec. 2021

pression of such data and their convenient utilization.
The first problem regarding accumulating andusing 3D point cloud data of a wide area of a city is
the size of the data. Efficiently storing a large amountof point cloud data necessitates a highly efficientcompression technique. Moreover, 3D point clouddata collected moment by moment by using in-vehicle LiDAR, etc. are generated in fragments along the
trajectory of a moving vehicle. Therefore, to accumulate and manage those data as a 3D point cloud database covering a wide area of a city, it is necessary tointegrate the fragmented 3D point cloud data anddetect temporal changes in the latest data from past
data. To use such integrated “urban” data, it is necessary to implement a random-access function to
retrieve 3D point cloud data at arbitrary locations. In
addition to developing the compression function, weare researching and developing three main functions:data integration, temporal-change detection, and random access.
The compression function is based on the point
cloud coding technology called geometry-based
point cloud compression (G-PCC), which is being
standardized by MPEG (Motion Picture Experts
Group) and enables highly efficient compression of
3D point cloud data with a large data size. By efficiently representing the spatial location of points as
an octree structure while dividing the space hierarchically, G-PCC can compress the data size up to about
a tenth of the original size (Fig. 3).
Integration of 3D point cloud data and detection of
temporal changes are explained as follows. In addition to developing the function for compressing thedata of individual files, we are investigating functions
for integrating the data in the compressed files andeasily comparing the differences between the data. As
mentioned above, 3D point cloud data compressed by
G-PCC maintain the spatial location of points hierarchically in an octree structure. This hierarchicalstructure makes it easy to extract a rough outline ofthe objects in the data from the encoded data. By
extracting this outline, it is possible to, for example,
detect changes in 3D point cloud data at different
acquisition times and combine fragmented data while
taking object boundaries into account (Figs. 4 and 5).
Regarding the random-access function, by using
the hierarchical structure of G-PCC, we are also
investigating (i) a scalable function for extracting
sparse summary data from dense point cloud data and(ii) a function for extracting only a part of the space.
With these functions, it becomes easy to not only
extract partial data but also obtain a general idea of44

Feature Articles

Hierarchical partitioning of thespace where the point cloud exists

Represents the spatial location ofa point as an octree structure

Fig. 3. Layering of point cloud data using an octree structure.

High-speed integration by detecting the boundary
between both types of data while gradually refining
the accuracy by using a hierarchical structure

Fragment data of 3D
point cloud (1)

1

1

1

1

0
1 1
0 1 1 1
1 1
0 1

001
011
0011
0 11 1
00 11
111
11
01

Fragment data of 3D
point cloud (2)Fig. 4. Combined processing of 3D point cloud data.

the data over a wider area.
We are currently researching and developing a software engine that can handle these three functions in a
unified manner. Using this engine will make it possible to store 3D point cloud data of a vast city in acompact size and easily retrieve the data for any arbitrary position and time. We plan to continue ourresearch and development to enable faster and more
efficient encoding while taking into account the factthat the volume and distribution of 3D point clouddata will further increase in the future.
4. Future developments
Using the technologies described above, we will
establish technologies for constructing cyberspacethat can precisely reproduce real space and embody amechanism for value creation through the advanced
integration of real space and cyberspace. Therefore,
we will contribute to the creation of a human-centered society in which everyone can lead a comfortable, energetic, and high-quality life.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Fig. 5. Result of integrating 3D point cloud data.

45

Feature Articles

References
[1]

[2]

Y. Yao, R. Ishikawa, S. Ando, K. Kurata, N. Ito, J. Shimamura, and T.
Oishi, “Non-learning Stereo-aided Depth Completion Under Misprojection via Selective Stereo Matching,” IEEE Access, Vol. 9, pp.
136674–136686, 2021.
K. Kurata, Y. Yao, S. Ando, N. Ito, and J. Shimamura, “Aggregative

Jun Shimamura

Senior Research Engineer, Supervisor, CyberPhysical System Group Leader, NTT Human
Informatics Laboratories.
He received a B.E. in engineering science from
Osaka University in 1998 and M.E. and Ph.D.
from Nara Institute of Science and Technology in2000 and 2006. He joined NTT Cyber Space
Laboratories in 2000. His research interests
include computer vision and mixed reality. He isa member of the Institute of Electronics, Information and Communication Engineers (IEICE).

[3]

Input Convolution for Large-scale Point Cloud Semantic Segmentation,” The 7th IIEEJ International Conference on Image Electronics
and Visual Computing (IEVC 2021), Sept. 2021.
M. Watanabe, I. Igarashi, R. Tanida, and H. Kimata, “To That PlaceThat Day: An Efficient Method for Merging Compressed Point
Clouds to Create Point Cloud Data at Any Time and Place,” Proc. of
DICOMO 2021, pp. 357–362, June 2021 (in Japanese).

Ryuichi Tanida

Senior Research Engineer, NTT Human Informatics Laboratories.
He received a B.E. and M.E. in electrical engineering from Kyoto University in 2001 and 2003.
Since joining NTT in 2003, he has been engaged
in research on video coding and its application
technology. His research interests include image
processing, video coding, and sensing data coding technology. He is a member of IEICE.

Shingo Ando

Senior Research Engineer, Supervisor, NTT
Human Informatics Laboratories.
He received a B.E. in electrical engineeringfrom Keio University, Kanagawa, in 1998 andPh.D. in engineering from Keio University in
2003. He joined NTT in 2003, where he has been
engaged in research and practical application
development in the fields of image processing,pattern recognition, and digital watermarks. He
is a member of IEICE, the Institute of Image
Electronics Engineers of Japan (IIEEJ), and theInstitute of Image Information and Television
Engineers (ITE).

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

46

Feature Articles: Olympic and Paralympic Games Tokyo 2020
and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Introduction
Shingo Kinoshita
Keywords: Olympic and Paralympic Games Tokyo 2020, sports viewing, ultra-realistic communication
technology, Kirari!

1. The beginning: Toward future proposals
On September 7, 2013, I was on a business trip to
New York with Hiromichi Shinohara, then senior
executive vice president and head of the Research andDevelopment Planning Department, when the decision was made to bid for the Olympic and Paralympic
Games Tokyo 2020. It seems like only yesterday that
we were in a taxi on our way to the airport and I said,“I want to propose a future that will surprise the world
at Tokyo 2020.” That utterance was the beginning of
the Tokyo 2020 Project at NTT’s research and development (R&D) laboratories.
At past Olympic Games and other world events,
NTT has proposed various visions for the future ofcommunications technology. For instance, at the
Olympic Games Tokyo 1964, we provided communication technology for the world’s first live broadcast
using geostationary satellites and proposed a visionof the future of television broadcasting. At the 1970
World Exposition in Osaka, we experimentally demonstrated the world’s first wireless telephone system(Fig. 1) and proposed a vision of the future of personal communications. At the Olympic WinterGames Nagano 1998, Japan, we provided a personal
handy-phone system (PHS) multimedia communication system and proposed a vision of the future ofwearable communications by experimentally demonstrating the world’s first wristwatch-type PHS(Fig. 2). At the Olympic Games London 2012 and theWorld Soccer Tournament in Brazil in 2014, we provided technical support for live viewing via 8K Internet Protocol transmission and envisioned the futureof public viewing. For the Olympic and Paralympic
Games Tokyo 2020, we wanted to propose a vision of
the future that goes beyond this legacy, and that
proposal became our mission.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Fig. 1. T he world’s first wireless telephone system tested
at the 1970 World Exposition in Osaka.

Fig. 2. W ristwatch-type PHS tested at the Olympic WinterGames Nagano 1998.

47

Feature Articles

2. Challenges
Immediately after returning to Japan from that trip
to New York in 2013, we gathered a total of 13
researchers from various NTT laboratories andformed a study team. The ideas we had at that time
were dreamy, diverse, and exciting. Now that theTokyo 2020 Games are over, looking back at the
study materials gathered by that team, I’m amazed at
how many ideas have been implemented.
The 2020 Epoch-Making Project was established at
NTT Service Evolution Laboratories and tasked withthe mission of applying these ideas to the Tokyo 2020
Games and proposing a vision of the future. I’ve been
the project leader since 2015. In that year, an ultrarealistic communication technology called “Kirari!”,
which was envisioned by Katsuhiko Kawazoe, thenhead of NTT Service Evolution Laboratories, andpromoted its research by Akihito Akutsu, was introduced at the NTT R&D Forum. While conducting
various demonstrations at events such as kabuki (thetraditional Japanese performing art) performances athome and abroad, a global technology event in Austin, USA, synchronous performances in three cities
around the world by a three-female-member technopop unit, the opening of the new National Stadium inTokyo, and live viewings of the Japanese professionalsoccer league and U.S. professional baseball league,
we continued to make proposals for the Olympic andParalympic Games Tokyo 2020.
3. Significance of working on the Olympic andParalympic Games Tokyo 2020
For NTT, Tokyo 2020 Games were not only a celebration of sports and world peace but also a celebration of technology and other innovations. I believethat the involvement in the Games has resulted in thefollowing three benefits to NTT laboratories.
(1)	As the largest event of its kind in the world,
the Olympic and Paralympic Games attract
worldwide attention.
(2)	Strict demands and severe constraints are
imposed at the Games, leading to improvement and demonstration of not only technicalcapabilities but also planning and promotion
capabilities.
(3)	We can take on bold challenges.
Regarding (1), the Olympic and Paralympic Gamesare watched by as many as four-billion people aroundthe world, and that about six-hundred million people
watched the opening ceremony of the Olympic
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Games Tokyo 2020 alone. Moreover, the latest technologies used at various Olympic and Paralympic
Games attract attention from the public and are often
used at subsequent sporting events as a legacy.
Benefit (2) has not been pointed out much so far,but it is what we felt most through actual experience.
Although it is the flip side of the first reason, the more
attention it gets, the stricter the requirements must be
satisfied. In addition to performance, quality andsafety of the technology, the world’s highest levels of
expression and design are required. Technology,
planning, proposal, and coordination skills are also
required. In particular, the range of coordination isvery wide, for example, planning proposals required
coordination with the Tokyo Organising Committeeof the Olympic and Paralympic Games (Tokyo 2020
Organising Committee), the International Olympic
Committee (IOC), and international sports federations. When providing live viewing of competition
videos via technologies such as Kirari!, it is necessary to coordinate with the broadcasting-rights holders. In regard to installing filming equipment, it wasessential to coordinate with Olympic Broadcasting
Services (OBS), and in regard to system construction
and operation, it was necessary to coordinate withthose in charge of the technology, venues, and security staff in the Tokyo 2020 Organising Committee aswell as with the Tokyo Metropolitan Government,
local governments, and police. Each of them has specific roles and responsibilities and thus finding solutions for a breakthrough is necessary. On top of allthat, when introducing such activities on the company’s website or issuing news releases, we must ensurethat we are within our rights as a sponsor, and weneed to be creative in devising ways to express ourselves. Although our sponsor category is telecommunication services, the technologies of our researchlaboratories are not limited to telecommunication
services in a narrow sense, such as so-called communication lines, but often cover communication services in a broader sense.
Regarding (3), being able to take on bold challenges is the most important significance for laboratories. Normally, when selecting a theme at our laboratories, we select reasonably convincing viewpoints
such as technological advancement, business potential, or public acceptance. The more viewpoints you
have, the more likely you are to settle on a mediocre
research topic with a high probability of success anda low level of challenge. However, in the case of the
Olympic and Paralympic Games, both public andcompanies become more positive and open-minded,48

Feature Articles

and that thinking generates a better atmosphere forencouraging challenges. As a result, initiatives thatwould normally be unchallenged are keenly undertaken, and unexpected results and value may be created. With these three reasons in mind, we worked on
various proposals for more than two years and finallyobtained more than ten projects of our technology,
including live viewing via Kirari!.
4. Postponement, no spectators, and nevergiving up until the very end: the show went on!
On March 24, 2020, just before the start of theTokyo 2020 Olympic Torch Relay, our relief, however, was short-lived when the novel coronavirus(COVID-19)—which had been spreading since thebeginning of 2020—caused the Olympic and Paralympic Games Tokyo 2020 to be postponed for one
year. The postponement led to a major rethinking of
the Olympic and Paralympic Games Tokyo 2020
vision, from “the most innovative Games in history,
bringing positive change to the world,” to “simplification.” All the projects, including those we had proposed, became candidates for review. Fortunately, theimportance of innovation through technology was
understood, and all our proposed projects could becontinued.
Our next concern became the presence of spectators
because many of our projects were designed to beattended. On June 21, 2021, about a month before theopening of the Olympic Games, the policy wasannounced that the number of spectators should bewithin 50% of venue capacity and no more than
10,000. At that stage, we were relieved that our projects would be able to proceed without any major
effects; however, about two weeks later, on July 8, thepolicy was changed to not allowing any spectators,
except for a few venues. Throughout June, opposition
to the public viewing scheduled for Yoyogi Park in
Tokyo grew, and eventually most public-viewing sites
across Japan decided to cancel their events. We
decided to go ahead with our projects without spectators in the hope that our preparations would be complete and that we would be able to communicate withthe athletes, officials, and the world through various
mass media even if there were no spectators.
The Torch Relay, which started on March 25, wasinitially often held on public roads; however, in manycases, the relay on public roads was gradually cancelled, and only the ignition ceremony was held. NTT
had also planned extended celebration events, inOsaka on April 13 and Yokohama on June 30, to
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

which 5000 spectators were originally invited. In theend, the invitations to the general public were cancelled, and the events were streamed online only.
5. Impact of the COVID-19 pandemic
The aforementioned three benefits of working on
the Olympic and Paralympic Games Tokyo 2020
were greatly changed by the COVID-19 pandemic.
Therefore, I’d like to reflect on how the pandemic
affected the activities of our laboratories in relation toeach of the three above-stated benefits.
(1)	As the largest event of its kind in the world, theOlympic and Paralympic Games attract worldwide attention
As the world’s attention grew, so did the negative
attention, such as the opposition to hosting the Olympic and Paralympic Games during the pandemic. As aresult, the focus of people’s attention became whether
the Olympic and Paralympic Games should be held
and whether there could be spectators. In contrast,
people’s expectations for future proposals based ontechnology and sports entertainment plummeted.
(2)	Strict demands and severe constraints are
imposed at the Games, leading to improvementand demonstration of not only technical capabilities but also planning and promotion capabilities
The requirements and constraints that are normallystringent were made even more stringent by the pandemic. For example, when organizing events, it wasnecessary to take thorough measures to prevent
COVID-19 infection among spectators and otherpeople involved as well as to prepare for all possible
variations of requirements and constraints. For example, if 100% audience capacity, 50% audience capacity, or no attendance is stipulated, the content of the
projects had to be significantly changed. How andwhen to determine this change was a major issue. In
addition, with public opinion changing rapidly,
among the stakeholders involved in our projects,
there were many differences of opinion regarding thecontent and timing of external announcements suchas news releases, and it was difficult to adjust thecontent accordingly.
(3) We can take on bold challenges
Owing to the growing criticism of the Tokyo 2020
Games and the busy schedule caused by switching to
the no-spectators policy just before the event, themomentum in the stakeholders, such as the Tokyo
2020 Organising Committee, to take on new challenges greatly declined. It would have been satisfying
49

Feature Articles

Table 1. List of initiatives of NTT R&D.
Category

Theme

1

NTT R&D technologies forviewing Tokyo 2020 Games

Sailing × Kirari!

NTT R&D initiatives

NTT R&D technologies thatcolored Tokyo 2020 GamesTorch Relay photo session × Kirari!

Badminton × Kirari!
Marathon × ultra-low-latency communication technology

2

Torch Relay runner support × communication control technology “Swarm”
Torch Relay celebration stage × Kirari!
Torch Relay regional events × speech-recognition communication technologies

3

NTT R&D technologies that
supported Tokyo 2020 Games

High-efficiency Wi-Fi
Network security
Technologies for supporting athletesTechnologies for supporting staff
Technologies for supporting spectators

to see more challenging approaches such as virtualreality viewing and remote cheering, which are basedon the premise of no spectators. Despite this atmosphere, we stressed the importance of real-time, interactive remote cheering via telecommunications andproposed to implement it at the marathon and sailing
events. Even though the people involved did not haveany spare energy, they accepted that proposal andmanaged to implement it. When I saw the moment
when the cheering of remote spectators reached theathletes, along with the people involved, I realizedthat it was worthwhile taking on the challenge.
6. Future proposals from NTT laboratories andthe structure of feature articles
After the above-mentioned twists and turns, theinitiatives of NTT laboratories that were ultimately
implemented at the Tokyo 2020 Games fall into thefollowing three categories (as listed in Table 1):

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

 ategory 1, “NTT R&D technologies for viewing
C
Tokyo 2020 Games,” proposes a future of watching
sports through “Kirari!” and other technologies,
which were applied to the sailing, badminton, andmarathons events.
Category 2, “NTT R&D technologies that coloredTokyo 2020 Games,” proposes a future of staging
events by using Kirari! and the communication
control technology called “Swarm,” which were
implemented at events to celebrate the Olympic
Torch Relay.
Category 3, “NTT R&D technologies that supported Tokyo 2020 Games,” introduces high-efficiency
Wi-Fi and network security as well as technologies
that support athletes, staff, and spectators.
These initiatives will be introduced in NTT Technical Review in the coming months.
NTT is an Olympic and Paralympic Games Tokyo
2020 Gold Partner (Telecommunication Services).

50

Feature Articles

Shingo Kinoshita

Vice President, Head of NTT Human Informatics Laboratories.
He received a B.E. from Osaka University in1991 and M.Sc. with Distinction in technology
management from University College London,
UK, in 2007. He joined NTT in 1991 and was asenior manager of the R&D planning section of
the NTT holding company from 2012 to 2015.
He is currently a visiting professor at the Art Science Department, Osaka University of Arts, andvisiting executive researcher at Dentsu Lab
Tokyo. He has served as a member of the Japan
Science and Technology Agency (JST) JSTMirai Program Steering Committee, member ofthe All Japan Confederation of Creativity (ACC)TOKYO CREATIVITY AWARDS 2021 Judging
Committee, and member of the Broadband Wireless Forum Steering Committee. He has been
engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machinelearning, service design, and technology management. Until recently, he had been in charge ofNTT’s Tokyo2020 initiatives, including sportswatching video technology, inclusive design forsocial issues, and promoting the use of ICT in
kabuki, entertainment, and media arts such as live
music.
He has been in his current position since 2021,
where he manages R&D on information andcommunication processing of humans based onhuman-centered principles.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

51

Feature Articles: Olympic and Paralympic Games Tokyo 2020
and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Initiatives for a New Sports-viewing
Experience
Shingo Kinoshita
Keywords: Olympic and Paralympic Games Tokyo 2020, sports viewing, ultra-realistic communication
technology, Kirari!

1. NTT R&D aims to create a new way towatch sports
The first Olympic Games to be broadcast were theOlympic Games Berlin 1936. Even though the receiver was a television (TV), it only had 180 scanninglines and a screen size equivalent to 19 inches. Images of the Games were shown via such screens locatedat 28 venues around Berlin and watched by 100 spectators. At the Olympic Games scheduled for Tokyo in
1940, it was planned to broadcast the events via
screens with 441 scanning lines, frame rate of 25
frames per second, and an aspect ratio of 4:5, which
would be the basis for today’s TVs. However, theOlympic Games at Tokyo was cancelled due to WorldWar II, so it became a phantom TV broadcast. In
Rome in 1960, the Olympic Games were broadcast
live, for the first time, to 18 European countries and,albeit one hour later, to the United States, Canada,
and Japan. At the Olympic Games Tokyo 1964, NTT
developed the wireless relay technology used for the
first satellite relay of the Olympic Games. Since then,NTT has provided technical support for the color
broadcast of all events at the Olympic Winter Games
Sapporo 1972, the first 2K high-definition broadcastof the Olympic Games Seoul 1988, the 2K high-definition broadcast of all events at the Olympic WinterGames Nagano 1998, and the 8K super-high-definition public viewing at the Olympic Games London
2012.
Although video-relay technology has evolved over
80 years in the manner described above, that evolution has been the same story about devices, namely,
TVs, with screens surrounded by a square frame. In
other words, it has not fundamentally changed the
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

experience of watching sports. In preparation for the
Olympic and Paralympic Games Tokyo 2020, in
2015, NTT began research and development (R&D)
of an ultra-realistic communication technology called“Kirari!” for creating a new sports-viewing experience. To deliver the experience of being at the venue
to those who cannot attend the actual event, Kirari! is
an ambitious project that removes the square frame of
conventional TV and public viewing and transmits
the event space itself. Since 2015, we have experimentally demonstrated Kirari! for live viewing of avariety of events in addition to sports such as theJapanese professional soccer league and U.S. professional baseball league. Example live-viewing demonstrations include kabuki events in Japan and abroad, aglobal technology event in Austin, Texas, USA, a
synchronized performance by a three-female-member techno-pop unit in three cities around the world,
and the opening event at the new National Stadium inTokyo (Fig. 1).
Initially, Kirari! was intended to provide viewing
for people who lived far away from the event venue,
people who could not afford tickets, and those unable
to go out due to illness. However, with the spread of
the novel coronavirus (COVID-19) in 2020, people’s
mobility became ever more restricted, and the visionof Kirari! grew ever more important. The feeling thatyou could not go to an event venue even if you wanted to go or that something was missing even if you
watched the event on TV or smartphone became evenmore pronounced in 2020 when events, such as concerts and sports, were canceled due to the COVID-19
pandemic and online distribution suddenly expanded.
In the beginning, many people tried online streaming
to relieve the stress of being unable to get to the52Feature Articles

Live viewing of the U.S. professional baseball league

Demonstration of holographic table tennis

Kabuki performance of a virtual singer and real actors

Synchronous performance in three cities around theworld by a three-female-member techno-pop unit

Fig. 1. Previous showcases of Kirari!.

venue, out of curiosity, or because of its ease of use.
However, in accordance with the results of a survey
called “Awareness Survey on Live Music Distribution” carried out by SKIYAKI Inc. in September
2020, many people who have experienced real live
performances answered that real live performances
are better than online-streaming ones because they
convey senses of presence, unity, and specialness. For
example, 77% of the survey’s respondents answered,“Real live is better” because it gives “a sense of presence” (92.9%), “sense of unity” (93.6%), and “senseof specialness” (69.6%). Given those results, we considered the question, “How can we recreate thosesensations?” For that reason, recognizing once again
the importance of the vision of Kirari!, we decided totake on that challenge at the Olympic and Paralympic
Games Tokyo 2020.
2. Sense of presence
The “sense of presence” concerning a sports eventmeans that you feel as if you are actually there watching the event. In the actual venue, the distance
between a spectator to the action is so great that you
cannot see more than a speck in the distance.
Although it is easier to see the action online, the feelNTT Technical Review

Vol. 19 No. 12 Dec. 2021

ing of “being there” is what makes the real thing so
appealing. This sense of being there is based on two
main factors: the feeling that “you are there” (at thevenue) and the feeling that the “athletes are there” (infront of you). TVs and smartphones fit into a small
square frame, so we perceive them as a separate
space; in other words, we do not feel that we are sharing the event space with the athletes or feel the spacethat expands when we enter the venue and sit down.
To create a sense that “you are there” (at the venue),as if you were sitting in the stadium, we attempted toreproduce an expanse of space that covers the fullviewing angle. To achieve this, we use the ultra-wide
image-synthesis technology of Kirari!. Multiple 4K
cameras are installed and multiple videos from them
are seamlessly stitched together in real time to produce a single ultra-high-resolution video image. As aresult, video images with a resolution of over 20K canbe displayed on a display several tens of meters wide.
Although it is possible to enlarge the video image
from a 4K or 8K camera and project it on a huge display that covers the entire viewing angle, the image
resolution will inevitably be rough, and it will bedifficult to reproduce the sense of presence of thescene. It is also difficult to freely change the size ofthe image (in terms of height and width) in accordance
53

Feature Articles©2021 – IOC- All rights reserved Tokyo 2020

Fig. 2. Sailing × Kirari!.

with the event to be presented. NTT has conducted
experiments with the U.S. professional baseballleague, professional Japanese soccer league, tennis
tournament in France, windsurfing world championships, and a large-scale fashion show for women inTokyo.
For the Olympic and Paralympic Games Tokyo
2020, we used this technology at the sailing, at which
the spectators were seated far from the sailing course.
By using Kirari!, we transmitted the sailing course
directly to an offshore display near the spectators’seats to reproduce the sensation as if the races werebeing held right in front of them (Fig. 2). The details
are introduced in the article “Sailing × Ultra-realistic
Communication Technology Kirari!” in this issue [1].
To create a sense of the “athletes are there” (in frontof you), we attempted to create three-dimensional
images of the athletes at the Tokyo 2020 Games. To
achieve this, we used the “real-time extraction of
objects with arbitrary backgrounds,” a component
technology of Kirari!. Even if there is no green
screen, the athlete’s image is extracted in real timeand displayed in a holographic manner. Even if it is
the same image, displaying it as a holograph gives theaudience a stronger sense of the athlete’s presence.
The sense of presence was further enhanced by setting up realistic objects other than the images of theathletes, such as the badminton court or table tennis
table. We conducted experiments with this technology at karate, judo, and badminton tournaments aswell as at kabuki performances and a large-scalemusic event in Texas, USA.
For the Olympic Games Tokyo 2020, we used thistechnology at the badminton competition. Only video
images of the athletes were extracted from 8K camera
images of the badminton matches held at Musashino

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Forest Sport Plaza, and those images were transmitted to the National Museum of Emerging Science andInnovation (Miraikan), the remote-viewing venue. At
this remote-viewing venue, courts, nets, and spectators’ seats were set up in a manner just like that at themain venue. The holographic images of the athletes
were then displayed in conjunction with those objects
in a way that created a sense of the athletes’ presence
just like that at the main venue (Fig. 3). The details
are introduced in the article “Badminton × Ultrarealistic Communication Technology Kirari!” in thisissue [2].
3. Sense of unity
A “sense of unity” means a connection between theathletes and spectators as well as among spectators.
In sports, cheering from the spectators is the mostpowerful force in regard to the athletes. Spectators
can connect with each other through cheering, thereby creating even more excitement as they become
one. For online streaming, which was expanded in
2020, efforts were made to enhance the sense of unity
by, for example, displaying images of the remoteaudience on the stage and initiating call-andresponse. In reality, however, the timing of the shouts
and cheers was too disjointed, preventing a sense ofunity.
The reason for that disjointedness is mainly thelatency and variability of the communication. Delaysof more than a few dozen milliseconds generally
make music sessions difficult, and delays of morethan a few hundred milliseconds make call-andresponse uncomfortable. Such a delay has the following three components: the propagation delay of light
is about 0.5 ms for 100 km, transmission-processing

54

Feature Articles

Fig. 3. Badminton × Kirari!.

Fig. 4. Marathon × ultra-low-latency communication technology.

delay is several milliseconds to several dozen
milliseconds, and video-coding process takes several
hundred milliseconds. With the addition of video
editing and other factors, it is not uncommon for the
final delay to be up to ten seconds in the case of digital terrestrial broadcasting and live streaming over theInternet.
For the Olympic Games Tokyo 2020, ultra-lowlatency communication technology, which significantly reduces latency, was used at the marathon.
Spectators’ cheers from Tokyo were delivered without delay to the athletes in Sapporo, who were run-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

ning at 5 m/s, and the challenge was to create a senseof unity that transcended distance (Fig. 4). The
details are introduced in the article “Marathon ×
Ultra-low-latency Communication Technology” inthis issue [3].
NTT is an Olympic and Paralympic Games Tokyo
2020 Gold Partner (Telecommunication Services).
References[1]

T. Miyatake, K. Shimizu, T. Hayasaka, M. Nakamura, M. Aoki, H.
Chigira, J. Nakajima, M. Urata, K. Ogo, Y. Yoshida, and S. Kinoshita,

55

Feature Articles[2]“Sailing × Ultra-realistic Communication Technology Kirari!”, NTT
Technical Review, Vol. 19, No. 12, pp. 57–68, Dec. 2021.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202112fa8.html
M. Muto, K. Hasegawa, D. Namikawa, S. Konya, N. Hirachi, T.
Nakamura, K. Suzuki, and S. Kinoshita, “Badminton × Ultra-realistic
Communication Technology Kirari!”, NTT Technical Review, Vol.
19, No. 12, pp. 69–77, Dec. 2021.

[3]

h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202112fa9.html
S. Usui, S. Fukatsu, E. Matsumoto, M. Imoto, D. Shirai, and S.
Kinoshita, “Marathon × Ultra-low-latency Communication Technology,” NTT Technical Review, Vol. 19, No. 12, pp. 78–84, Dec. 2021.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202112fa10.html

Shingo Kinoshita

Vice President, Head of NTT Human Informatics Laboratories.
He received a B.E. from Osaka University in1991 and M.Sc. with Distinction in technology
management from University College London,
UK, in 2007. He joined NTT in 1991 and was asenior manager of the R&D planning section of
the NTT holding company from 2012 to 2015.
He is currently a visiting professor at the Art Science Department, Osaka University of Arts, andvisiting executive researcher at Dentsu Lab
Tokyo. He has served as a member of the Japan
Science and Technology Agency (JST) JSTMirai Program Steering Committee, member ofthe All Japan Confederation of Creativity (ACC)TOKYO CREATIVITY AWARDS 2021 Judging
Committee, and member of the Broadband Wireless Forum Steering Committee. He has been
engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machinelearning, service design, and technology management. Until recently, he had been in charge ofNTT’s Tokyo2020 initiatives, including sportswatching video technology, inclusive design forsocial issues, and promoting the use of ICT in
kabuki, entertainment, and media arts such as live
music.
He has been in his current position since 2021,
where he manages R&D on information andcommunication processing of humans based onhuman-centered principles.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

56

Feature Articles: Olympic and Paralympic Games Tokyo 2020
and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Sailing × Ultra-realistic
Communication Technology Kirari!
Takashi Miyatake, Kentaro Shimizu, Tomoyuki Hayasaka,Mushin Nakamura, Masakatsu Aoki, Hiroshi Chigira,Junichi Nakajima, Masakazu Urata, Kenta Ogo,
Yuki Yoshida, and Shingo Kinoshita
Abstract

NTT provided its ultra-realistic communication technology called Kirari! to the TOKYO 2020 5G
PROJECT, which was implemented by the Tokyo Organising Committee of the Olympic and Paralympic
Games to offer a new experience of sports viewing by using 5G (5th-generation mobile communication
system). A new experience of watching sports at the sailing regatta, held at the Enoshima Yacht Harbor,
is introduced in this article. In short, a sense of realism—as if the spectators were watching the racesfrom close up—was provided by transmitting ultra-wide video images taken close to the race course to
the spectators’ seats set up at the harbor.
Keywords: 5G, Kirari!, live viewing

1. Overview of our initiatives at thesailing regatta
In sailing races, spectators usually have to use binoculars to watch the races (from breakwaters, etc.)
because the races are held quite far from land. At thesailing regatta of the Olympic and Paralympic GamesTokyo 2020, we took up the challenge of solving thisproblem by using communication technology to create an innovative style of spectating that goes beyond
the real world; namely, the spectators feel as if they
were watching the races from a special seat on a
cruise ship. Ultra-wide video images of the races(with a horizontal resolution of 12K) were transmitted live to a 55-m-wide “offshore wide-vision” display near the spectators’ seats (Fig. 1) by using the
5th-generation mobile communication system (5G)
and our ultra-realistic communication technology
Kirari!.
Initiatives at the sailing regatta for the TOKYO
2020 5G PROJECT are summarized below:• Period: July 25, 2021 to August 4, 2021
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

•	Venue: Enoshima Yacht Harbor, Kanagawa Prefecture• No. of events held: 10•	Total number of participants: about 4000 (onlyathletes and related people were allowed towatch the event)•	Organization: TOKYO 2020 5G PROJECT
formed by four bodies: the Innovation Promotion
Office of the Tokyo Organising Committee of the
Olympic and Paralympic Games, NTT (participating as the technical supervisor), NTT DOCOMO, and Intel Corporation (Fig. 2).
The ultra-wide video images were also transmitted
live to Tokyo Big Sight, where the main press center(MPC) was located, to convey the realism of the eventto media personnel whose movements were restrictedwithin Japan because of the novel coronavirus(COVID-19) pandemic (Fig. 3).
Although the sailing regatta was held without spectators, we implemented a measure called a “virtual
stand” for sending support to the athletes from their
far-away families and friends (Fig. 4).
57

Feature Articles©2021 – IOC- All rights reserved Tokyo 2020

Fig. 1. Example of live video transmission via offshore wide-vision display.

Fig. 2. Logo of TOKYO 2020 5G PROJECT.

©2021 – IOC- All rights reserved Tokyo 2020

Fig. 3. Example of video transmission to MPC.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

58

Feature Articles©2021 – IOC- All rights reserved Tokyo 2020

Fig. 4. Remote cheering from the virtual stand.

4K x 3 camera-equipped drone
TOKYO 2020

Ultra-wide video-synthesis
technology / Advanced MMT

TOKYO 2020

High-speed
large-capacitytransmission

Offshore wide-vision display
with 12K horizontal resolution

TOKYO 2020

4K x 4 camera-equipped boats
5G

13-m-wide LED screen@MPC at Tokyo Big Sight

TOKYO 2020

Fig. 5. Overview of the video transmission system.

2. Configuration of the video
transmission system
The configuration of the video transmission system
for the sailing regatta is shown in Fig. 5. First, thesailing action is captured from boats or a drone
equipped with multiple 4K cameras. The captured
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

video images are synthesized as ultra-wide video
images with 12K resolution and transmitted in realtime by using Kirari! and 5G, which enables highspeed, large-capacity transmission. The transmitted
images were displayed in real time on a 55-m-wide
offshore wide-vision display floating in front of theviewing area at the regatta venue and on a 13-m-wide
59

Feature Articles

Video shooting

Pre-process: Projection transformation matrix calculation
Extract feature points
from reference images
Calculate projectiontransformation matrix

Ultra-wide video
synthesis

Synthesis process: Projection transformation and seam-line searchMediasynchronizationtransmission

Fig. 6. Processing flow of ultra-wide-video synthesis technology.

light-emitting diode (LED) display installed at the
MPC at Tokyo Big Sight.
3. Technology
3.1 U ltra-realistic communication technology
Kirari!
Kirari! is a communication technology that enables
spectators to experience a race or other sports events
as if they were in the stadium even when they are faraway from it. In this project, we implemented two
technological elements of Kirari!, video transmission
using ultra-wide video-synthesis technology andultra-realistic media-synchronization technology.
3.1.1 Ultra-wide video-synthesis technology
The process flow of ultra-wide video synthesis isshown in Fig. 6. First, the optical axes of each of fourcameras are aligned as much as possible, and thecameras are set up so that their shooting ranges overlap adjacent captured images by 20–40%. Before theactual synthesis process, as a pre-processing step tocorrect the disparity between each camera, a projection-transformation matrix is calculated from feature
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

points extracted from a reference still image. For sailing, an image was taken of the competition yachts
waiting just before the race as the reference image. To
implement the pre-processing, we optimized theentire process from acquisition of the reference
image to determining the wide-image-synthesis
parameters by compressing the image data and reducing the transmission time by two-thirds, for example.
Next, during the synthesis process, video frames werecombined using a projection-transformation matrix
calculated for each video frame to correct disparities
and find appropriate seam lines in the video images(i.e., a process called “seam search”). The synthesis
process is characterized by three features: synchronous distributed processing, accelerated seam search,and use of a graphics processing unit (GPU).
For synchronous distributed processing, the synthesis process is distributed to multiple servers (in accordance with the input and output conditions) so that ahuge amount of video processing can be executed inreal time. To synchronize the synthesis timings ineach server, the same time code is given to the video
frames when they are input, and this time code is60

Feature Articles

propagated to the transmission devices (such as
encoders) in the subsequent stages. As a result, synchronization of frames can be controlled across theentire system. The time code can be selected witheither of two methods: (i) using the time code superimposed on the input video or (ii) generating the timecode when the video is input.
The seam search is sped up in two ways: (i) using
the movement and shape of moving bodies obtainedby analyzing the video frames during the seam searchor (ii) using the seam position of the past video frame
to prevent the seam position from fluttering. If theseam search for the next frame is executed after theseam search for the previous video frame is completed, it becomes difficult to achieve real-time synthesis. Accordingly, a two-stage seam search is conducted on a reduced video frame as a pre-search, andthe pre-search result is used in the seam search of thenext frame.
For GPU-based acceleration of the seam search,almost all video processing is executed on the GPU,so processing is more intensive and faster. As a resultof using the GPU, the video input from the serial
digital interface (SDI) board is transferred to the GPUwith minimal central-processing-unit cost, and thesubsequent video processing is executed by the GPU.
For example, when synthesizing 12K (4K × 3) ultrawide video images from images captured with thefour 4K cameras, it was possible to reduce the number of servers required by two-thirds (from six to two)compared with when the GPU was not used.
3.1.2 Ultra-realistic media-synchronization technology (Advanced MMT)
The video images generated using the ultra-wide
video-synthesis technology are divided into multiple
4K images for transmission so that they can be handled by general display devices, encoders, and decoders. In this case, the image on the display was 12K
wide and 1K high, so it was divided into three 4K
images. Kirari! uses advanced MPEG Media Transport (MMT) technology to perfectly synchronize theplayback timing of individual images and sounds thatare transmitted in separate segments. Ultra-realistic
media-synchronization technology (Advanced
MMT) synchronizes and transmits multiple video
and audio with low delay by using UTC (Coordinated
Universal Time)-based synchronization control signals that comply with the media transmission standard MMT. This technology can (i) synchronize anddisplay not only wide-angle video images but also
multi-camera-angle video images (2K, etc.) that tracka specific athlete and (ii) absorb the difference in
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

delays due to the network (dedicated line, Internet,
wireless local area networks, etc.) that transmits eachcamera image or audio signal and device processing.
This enables an ultra-realistic viewing experience inwhich multiple videos and audio are completely synchronized.
3.2 Wireless communication technology
The races of the sailing regatta at the Olympic andParalympic Games Tokyo 2020 were held on six race
courses (Fig. 7). Two of those race courses, “Enoshima” and “Zushi,” were chosen for our initiatives.
Ultra-wide video images were captured from threecamera boats for the Enoshima course, and ultra-wide
images were captured using a drone camera in addition to three camera boats for the Zushi course. 5Gbase stations were set up near both courses at the
points shown in the figure.
The configuration of the network used in this project is shown in Fig. 8. For uplinking video from theboats to shore and downlinking from the video-editing base to the barge-mounted offshore wide-vision
LED display, NTT DOCOMO’s 5G line was used as
the core in conjunction with multiple wireless communications with different characteristics such as
bandwidth, distance, and directionality. NTT DOCOMO’s 4G line was used for flight control of the drone,
backup video, and control of the server located on thecamera boats.
3.2.1 5G downlink transmission
The video transmission between the video-editingbase and barge-mounted offshore wide-vision LED
display was carried out using NTT DOCOMO’s 5G
access premium service. A commercially available
customer premises equipment (CPE) terminal wasused as the receiver. During the regatta, 200-Mbit/s
video transmission was continuously operated formore than eight hours a day, achieving stable video
transmission.
3.2.2 5G uplink transmission
NTT DOCOMO’s 5G access premium service wasmainly used for transmission of drone video. A commercially available CPE terminal was used as thereceiver. Six 2K video images were transmitted at an
average bandwidth of 80 Mbit/s. NTT DOCOMO’s
4G service was used for backup of video transmission
at a total rate of 50 Mbit/s.
3.2.3 Transmission of video images from a drone
The system configuration of a drone used to capture
video images of the races is shown in Fig. 9. The
drone was fitted with three 4K cameras. Threestreams of 4K video taken by a drone were transmitted
61

Feature Articles

5G base station
Offshore wide-vision display
5G
base
stationEnoshima
Yacht Harbor
Enoshima
KamakuraZushi
5G base station

Fujisawa
Sagami

HayamaFig. 7. Map of the race courses for the sailing regatta.

Ocean race course (Enoshima/Zushi)

EnoshimaC
A drone boat

A droneVideotransmission
deviceUltra-wide
image
synthesis

Enoshima video-editing base
Advanced
MMT
deviceTransmission distance
2000–3000 m

Transmission distance
100–200 m
D
A relay boat

A camera boat
Ultra-wide
image
synthesis

Advanced
MMT
deviceTransmission
distance
100–200 m

B

Video-relay
deviceOffshore widevision display

Advanced
MMT
deviceB

video-editing baseVideotransmission
deviceTransmission distance
2000–3000 m

Advanced
MMT
deviceA

Advanced
MMT
deviceAdvanced
MMT
device5G
4.9 GHz, 5.6 GHz
4G
60 GHzFig. 8. Network-configuration diagram.

to a drone boat, from which the drone is launched andlanded via 60-GHz wireless communication. The
three video streams were time-synchronized fortransmission. On board the drone boat, wide video
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

images were synthesized and synchronously transmitted to the video-editing base at Enoshima via the
5G access premium service and Advanced MMT(Fig. 8).
62

Feature Articles

3

2

2
1

(2) Wireless transmission system between drone and drone boata. Millimeter wave transmission system (developed by NTT Access Network
Service Systems Labs.): By using 60-GHz band, it is possible to transmit
4K (20 Mbit/s) x 3 videos between long distance (over 100 m).
b. GPS gimbal system: Based on RTK-GPS tracking, antennas of the drone
and the drone boat automatically and accurately follow each other.
c. Automatic antenna switching technology (developed by NTT AccessNetwork Service Systems Labs): To avoid radio line-of-sight obstruction
by drone body, two antennas are equipped on the drone, and the proper
antenna is automatically selected based on the radio quality.
c.

b.
Automatic antenna control
by RTK-GPS based drone
position tracking(1) 4K x 3 synchronized video transmission system
Multiple 4K IP video streams
can be transmitted with timesynchronization among cameras
by using timecode.
This time, three video streams
were transmitted.

(i)

(ii)
Antennaon the drone boat

Based on the radio quality of antenna (i)and (ii), proper antenna is automaticallyselected. In this case, (ii) is selected
since (i)’s signal quality is inferior to (ii)because of the drone body.

(3) Multi-band drone-control functionality
For redundancy of drone control assuming unstable radio conditions (e.g. interference by multiple radio devices, attenuation by long distance),
2.4 GHz/4G bands are implemented as the drone controlling channels, which can be immediately switched by the drone pilot.

Fig. 9. Configuration of a drone system.

For the video transmission between the drone anddrone boat, 60-GHz wireless-communication technology developed by NTT Access Network Service
Systems Laboratories was used. For the wirelessequipment, it is necessary to keep the antenna surface
facing the drone all the time. To ensure more-stablewireless communication, we developed a technologythat automatically controls the direction of the antenna by using real time kinematic (RTK) GPS (Global
Positioning System) information so that the antennaalways points in the direction of the drone. Depending on the shooting angle of the drone, the drone may
create a blind spot for wireless communication. To
solve this problem, we developed a technology for
automatically switching the antenna in accordancewith the wireless-signal strength. The above technologies enable stable video transmission between
the drone and drone boat in a manner that gives thedrone more freedom to shoot video. As a security
measure for the drone, the flight control of the drone
was duplexed using two 4G wireless signals, one at 4
GHz and the other at 2 GHz.
3.2.4 Transmission of video images from camera
boats
The following requirements for transmission ofvideo images from the camera boats were set:
1.	Since the race course may be changed at short
notice due to weather conditions or wind direction, the camera boats should be able to move
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

flexibly.
2.	Depending on the race course used, wirelesstransmission over a distance of 2 to 3 km is
required.
To meet these requirements, we prepared a “relayboat”—separate from the camera boats—for longdistance wireless transmission to the land. For transmission of video between the camera boats and relay
boat, 60-GHz wireless communication, which has ashort transmission distance but a large data-transmission capacity, was used. To allow the relay boat tomove flexibly, the boat was fitted with a wirelessantenna with a relatively wide angle of directivity. In
addition to 5G, 4.9-GHz and 5.6-GHz wireless communications were used to transmit video images fromthe relay boat to land (Fig. 8). While these wirelesscommunications can transmit over long distances, theantenna directionality is strong, so the antenna surfaces of the transmitter and receiver must always bealigned. Two mechanisms for controlling the antenna
were developed (Table 1 and Fig. 10).
The type of boats was selected by applying a bit ofingenuity. Since it was necessary to film the races
close up, small fishing boats were selected as thecamera boats because they would not affect the navigation of the racing yachts. A large fishing boat thatcan suppress rocking even in open seas was selected
because it has to transmit wireless signals over 2 to 3km.
63Feature Articles

Table 1. Two systems used for controlling antennas.
Systems

Details

Radio

System 1
Remote control

We integrally molded a remote controllable surveillance camera and antenna
and remotely controlled the antenna using surveillance-camera images.
Specifically, we used surveillance-camera images from boats to land toremotely control the direction toward the land antenna, achieving wirelesscommunication.

Used for radios with relatively largeantenna-reception angles(30 degrees)

System 2
Auto control by GPS

We developed a technology that automatically controls the direction of theantenna by using RTK-GPS information so that the antenna always points inthe direction of the boat and land.

Used for radios with relatively small
antenna-reception angles(10 degrees)

System 1: Remote control

System 2: Auto control by GPS
On the boat
The radio antenna for land
: Remote control from land using
surveillance cameras

At the land

60-GHz antenna: Antenna operation on the boat

GPS Gimbal: Auto antenna-control system using RTK

Fig. 10. Examples of using two antenna-control systems.

For the on-land antennas, it was not always possible
to secure a roof top of a tall building along the coastline. Accordingly, a vehicle was fitted with a pole that
could be raised high above the vehicle, and theantenna was attached to the top of the pole to ensure
a clear line-of-sight to the relay boat.
The 60-GHz band enables transmission at 100 to
200 Mbit/s over a distance of 100 m (i.e., boat-to-boat
distance), while the 4.9- and 5.6-GHz bands enabletransmission at 50 to 150 Mbit/s over a distance of 2
to 3 km (boat-to-land distance). Communication
speed depended on the condition of the ocean waves.
During the approach of typhoon Nepartak, the waves
were high, so it was difficult to control the antennasof the wireless transmitters, which significantly
reduced communication speed.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

3.3 Offshore wide-vision display
To create a sense of realism as if you were watching
the races up close, highly realistic video images anda space to make use of those images are required. To
meet the first requirement, actual-size images of thecompetition yachts, a display size that covers theentire field of view, high resolution, and a display
with no noticeable dots are needed. The offshore
wide-vision display was 5 m high and 55 m wide witha horizontal resolution of 12K and vertical resolutionof 1K. Although the pitch size was 4.8 mm, as shownin Fig. 11, it was installed 40 m from the spectators’seats, so the roughness of the image was not noticeable because it exceeded the spectators’ retinal resolution.
To meet the second requirement, the space in which
the display is set up is also important. Since the64Feature Articles

Enoshima Yacht Harbor

Offshore wide-vision display

40 m

Spectator seats

Fig. 11. Location of the offshore wide-vision display.

display is like a rectangular cut-out, it inevitably
leaves an unnatural feeling in the same manner as atelevision. By floating the display on the ocean andmerging its boundaries with the natural sky and sea,
it was possible to create a natural sense of realism.
When the offshore wide-vision display was
installed, we set up an organization containing not
only specialized members who design and build
LEDs but also those who carry out structural calculations on the basis of the undulation of the sea surface,
those who navigate and moor boats, marine-procedure agents, weather forecasters, etc. and studied thefollowing points in consideration of the weather at
the actual regatta. We collected meteorological data
for Enoshima in July and August over the past 10
years to understand wind speed, wind direction, wave
height, etc., evaluated the strength of the designed
LED display and the safety of the means of mooring
the barge of the floating display, and determined thecriteria for evacuating the site of the offshore widevision display. To meet the evacuation criteria, thedisplay was operated under the condition that windspeed did not exceed 25 m and wave height (swell)
did not exceed 1.6 m. The criteria were determined
after careful considerations of the members specializing in marine weather based on not only the intensity of the LEDs and the strength of the mooring of
the floating-display barge but also the weather conditions on Enoshima Island and on the waters off Miura
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Peninsula in case of evacuation to Odaiba in Tokyo.
During typhoon Nepartak, which occurred during theregatta, weather information was constantly monitored, and possible evacuation was determined on thebasis of the above-mentioned criteria. As a result ofthat monitoring, it was determined that no evacuation
was necessary, and the display was operated without
incident.
3.4 Wave-undulation correction technology
When filming from a boat on the ocean, the captured video images will flicker due to the rocking of
the boat. We therefore developed a technology forcompensating for this video flickering in real time.
The developed technology uses optical flow to detectthe direction of the video flickering and correct it inreal time to minimize as much flickering as possible.
4. Results
During the regatta, the video transmission wentsmoothly without any major problems. Although theregatta was not attended by spectators, hundreds ofpeople, mainly athletes and related support people,
watched the video transmission every day of theregatta. We received comments from many people ofthose that expressed their surprise, excitement, andhopes for the future of sailing. Some comments were
“I was surprised by the high sense of realism that I
65

Feature Articles

have never experienced before,” “I really wish thegeneral public could see this,” “It made it easy tounderstand the race situation,” and “If this style ofspectating becomes common, the number of sailingfans will increase rapidly.” For our aim of creating “asense of realism, as if the race was being held in frontof you,” we were able to create the experience of“warping into the race space itself” by fusing thepowerful, actual-size race images displayed on theoffshore wide-vision display floating on the sea withthe actual surface of the sea, sky, and other real
spaces.
Technically, issues still had to be solved. Theseissues include (i) unstable wireless communication,
including 5G, in certain situations and (ii) slight disturbance in the video images because the camera
boats and drone could not completely absorb theundulations due to the waves and engine, respectively. However, we believe that we were successful in anenvironment that demands a very high level of performance.
5. Concluding remarks
Using commercial 5G and our in-house ultra-realistic communication technology Kirari!, we successfully transmitted live ultra-wide video images (with
12K horizontal resolution) via a 55-m-wide offshore
wide-vision display installed at the Enoshima Yacht
Harbor site and a similar but smaller display at the
MPC at Tokyo Big Sight in Tokyo. At both sites, thelive video transmissions were highly evaluated byathletes, officials, and international media. The com-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

munication technology used in this project will
enable live viewing not only within a competition
venue but also to remote locations, including overseas ones. Even during the COVID-19 pandemic, thisproject can be ranked as a showcase for achieving a
remote world that NTT is aiming for. From now
onwards, we will promote further research and development on communication technologies, mainly inthe fields of sports and entertainment, using 5G and
the All-Photonics Network of IOWN (Innovative
Optical and Wireless Network) as an infrastructure.
Acknowledgments
We thank all members of the TOKYO 2020 5G
PROJECT, the Innovation Promotion Office of theTokyo Organising Committee of the Olympic andParalympic Games, and Intel Corporation who
worked with us on the project. We also thank the
Functional areas (FAs) of the Tokyo Organising
Committee of the Olympic and Paralympic Games,
World Sailing, and Japan Sailing Federation for theircooperation in implementing this project, and the
Enoshima-Katase, Koshigoe, and Kotsu­bo Fishing
Cooperatives for providing the necessary boats weused for filming. Finally, we particularly want tothank Tatsuya Matsui, the chief engineer in charge of
EEP infrastructure, and Hikaru Takenaka, who was incharge of network construction for this sailing project.
NTT is an Olympic and Paralympic Games Tokyo
2020 Gold Partner (Telecommunication Services).

66

Feature Articles

Takashi Miyatake

Hiroshi Chigira

Senior Research Engineer, Supervisor, NTT
Human Informatics Laboratories.
He received an M.E. in engineering from Kobe
University in 1995. In the same year, he joined
NTT and engaged in research on image processing and human interface. He also worked as asystem engineer at NTT Communications andother companies. He is currently engaged inresearch of the ultra-realistic communication
technology Kirari!.

Senior Research Engineer, NTT Human Informatics Laboratories.
He received a B.E. and M.E. in engineeringfrom Waseda University, Tokyo, in 2007 and2009 respectively, and joined NTT in 2009. Since
then he has engaged in research on human-computer interaction and developed vital sensing
hardware at NTT Cyber Solution Laboratories
and NTT Service Evolution Laboratories. In
2016, he joined 2020 Epoch-Making Project,
where he has been engaged in several creative
research projects combining NTT’s cutting-edge
technologies, design, and art to celebrate theTokyo 2020 Games.

Kentaro Shimizu

Junichi Nakajima

Senior Research Engineer, Supervisor, NTT
Human Informatics Laboratories.
He received an M.S. in electronics and information engineering from Yokohama National
University, Kanagawa, in 1999 and Ph.D. inmedical research from Tokyo Women’s Medical
University in 2009. He joined NTT in 1999. He
is a member of the Institute of Electronics, Information and Communication Engineers, theInformation Processing Society of Japan (IPSJ),
the Japanese Society for Artificial Intelligence,
the Japan Society for Health Care Management,
and the Japan Society of Computer Aided Surgery.

Senior Research Engineer, NTT Human Informatics Laboratories.
He received an M.Sc. in information sciencefrom the University of Tokyo in 1998. He joined
NTT the same year, and engaged in research onvideo coding and media streaming systems. He
also developed various devices such as videophone and set-top box at NTT EAST. He is currently engaged in research of Kirari!.

Tomoyuki Hayasaka

Masakazu Urata

Senior Research Engineer, NTT Human Informatics Laboratories.
He received an M.Sc. in physics from Hokkaido University in 1996. He joined NTT the same
year and engaged in a system engineering on web
systems. He has been engaged in research on
Kirari! since 2019.

Research Engineer, NTT Human Informatics
Laboratories.
He received a B.E. and M.E. in electronicengineering from Osaka Prefecture University in1990 and 1992. He joined NTT Information
Networking Systems Laboratories in 1992 andengaged in R&D of intelligent network, e-Government, and smart card. He is currently engaged
in research on a cloud computing platform andhigh-speed networks.

Mushin Nakamura

Kenta Ogo

Senior Research Engineer, NTT Human Informatics Laboratories.
He received a B.S. in mathematics and M.E. incomputer science from the University of Tokyoin 2001 and 2003. He joined NTT in 2003, where
has been engaged in research and development(R&D) of NGN (Next Generation Network)
appliances and web-based signage systems.

Masakatsu Aoki

Senior Research Engineer, NTT Human Informatics Laboratories.
He received a B.E. and M.E. in engineeringfrom Shinshu University, Nagano, in 1994 and1996. He joined NTT Human Interface Laboratories in 1996 and worked on research related tocomputer graphics. His research interests include
human-computer interaction, life logs, and visualcommunication.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Research Engineer, NTT Human Informatics
Laboratories.
He graduated from the University of Tokyo. He
joined NTT in 1995, where he was engaged in
R&D on on-demand video distribution using
Internet Protocol communication at NTT Human
Interface Laboratories. He then launched aninternational content delivery network and largescale commercial video-distribution service at
NTT Communications. He is widely involved in
the video distribution field such as program recommendations at NTT Plala. He then conducted
research on a three-dimensional virtual society,location-linked tourism recommendations, ultrahigh reliability 360-degree virtual-reality video
distribution, and telepresence. He designed a
12K-wide long-distance maritime wireless video
transmission system for the Tokyo 2020 Games.

67

Feature Articles

Yuki Yoshida

Research Engineer, NTT Human Informatics
Laboratories.
She received a B.Sc and M.Sc. from the University of Tsukuba. She joined NTT and engaged
in R&D of human interface. In 2019, she joined2020 Epoch-Making Project, where she has been
engaged in service promotion to celebrate theTokyo 2020 Games.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Shingo Kinoshita

Vice President, Head of NTT Human Informatics Laboratories.
He received a B.E. from Osaka University in1991 and M.Sc. with Distinction in technology
management from University College London,
UK, in 2007. He joined NTT in 1991 and was asenior manager of the R&D planning section of
the NTT holding company from 2012 to 2015.
He is currently a visiting professor at the Art Science Department, Osaka University of Arts, andvisiting executive researcher at Dentsu Lab
Tokyo. He has served as a member of the Japan
Science and Technology Agency (JST) JSTMirai Program Steering Committee, member ofthe All Japan Confederation of Creativity (ACC)TOKYO CREATIVITY AWARDS 2021 Judging
Committee, and member of the Broadband Wireless Forum Steering Committee. He has been
engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machinelearning, service design, and technology management. Until recently, he had been in charge ofNTT’s Tokyo2020 initiatives, including sportswatching video technology, inclusive design forsocial issues, and promoting the use of ICT in
kabuki, entertainment, and media arts such as live
music.
He has been in his current position since 2021,
where he manages R&D on information andcommunication processing of humans based onhuman-centered principles.

68Feature Articles: Olympic and Paralympic Games Tokyo 2020
and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Badminton × Ultra-realistic
Communication Technology Kirari!
Makoto Muto, Keisuke Hasegawa, Daichi Namikawa,Seiichi Konya, Nobuhiro Hirachi, Taiji Nakamura,
Kenya Suzuki, and Shingo Kinoshita
Abstract

NTT provided technical support through Kirari!, an ultra-realistic communication technology, for theTOKYO 2020 Future Sports Viewing Project spearheaded by the Tokyo Organising Committee of the
Olympic and Paralympic Games. This article presents results of demonstrations of a next-generation
immersive technology for providing a sense of “being there,” namely through the sense of presence andsense of unity, by transmitting holographic images of the badminton matches held at Musashino Forest
Sport Plaza to a remote location (National Museum of Emerging Science and Innovation (Miraikan)).
Keywords: sports viewing, highly realistic experience, holographic video transmission

1. Overview of the TOKYO 2020 Future Sports
Viewing Project
The beauty of watching performances live is being
able to experience the sense of presence and sense ofunity that can be felt when you are actually at thevenue. In other words, it enables experiencing thesensation of having athletes or artists being right infront of you and sharing the same space with themwhen watching sports events or live concerts, rather
than viewing them as images on a flat screen. Despite
dramatic improvements in screen size, image quality,
and resolution, online streaming and live viewing still
can transmit only flat-screen images. It is therefore
not possible to fully convey a sense of presence, suchas the athlete’s in-the-moment excitement, strengthand beauty of the physical form, and sense of unity
between the athletes and audience. For the TOKYO
2020 Future Sports Viewing Project spearheaded by
the Tokyo Organising Committee of the Olympic andParalympic Games, we carried out technology demonstrations aimed at delivering the experience ofbeing at the badminton venue of the Tokyo 2020
Games to those who could not attend the actual eventby using Kirari! ultra-realistic communication techNTT Technical Review

Vol. 19 No. 12 Dec. 2021

nology to transmit holographic images of the event(Fig. 1).
We initially planned demonstrations of live viewing
by inviting the general public. However, to preventthe spread of the novel coronavirus, we cancelled the
public demonstration and instead opened “The Futureof Sports Viewing—Next-generation Immersive
Technology Demonstration Program” at the National
Museum of Emerging Science and Innovation (Miraikan) for the media from July 30 to 31, 2021.
2. System configuration
Figure 2 shows the overall system configuration
for the technology demonstrations. We set up 8K
cameras (Fig. 3) at the Musashino Forest Sport Plaza,
the venue for the Tokyo 2020 Games badminton competitions, and transmitted video images taken during
the badminton matches via a 1-Gbit/s network to thebroadcast center. We then used the real-time extraction of objects with arbitrary background, a component technology of Kirari!, to selectively extract only
the images of players and shuttlecocks from thetransmitted 8K images at the broadcast center(Fig. 4). Using highly realistic media synchronization69

Feature Articles

Fig. 1. Kirari! technology demonstration experiments of the Tokyo 2020 Games badminton competition.

The competition venue(Musashino Forest Sport Plaza)

8K camera
Systemconfiguration

Advanced
MMT
transmitter

The broadcast
centerNetwork(1 Gbit/s)Videofrom OBS

Advanced
MMT
transmitter

The remote-viewing venue(Miraikan museum)
Network(1 Gbit/s)

Advanced
MMT
transmitter

Holographic
projection
equipment

Objectextraction
equipment
Real-time
extraction of
objects with
arbitrary
background

Kirari!
technology

Highly realistic media synchronization technology (Advanced MMT)

Bird’s eyeview sports
viewing
multi-layer
aerial image
displaytechnology

Fig. 2. The overall system configuration.

technology (Advanced MMT (MPEG Media Transport)) [1], we then synchronously transmitted theextracted images along with multiple images that
included video and audio provided by the Olympic
Broadcasting Services (OBS) to the remotely locatedMiraikan museum.
We set up an approximately 100-seater spectator
stand and a full-size court equipped with holographic
projection equipment at the remote-viewing venue torecreate the actual venue (Fig. 5). The extracted
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

images of players and shuttlecocks were holographically displayed by presenting players in their actual
positions in front and behind the net using the bird’s
eye-view sports viewing multi-layer aerial image display technology. As a result, we were able to create aspace where the badminton players appeared to be
actually present in the remote-viewing venue.

70

Feature Articles© 2021 International Olympic Committee. All Right Reserved.

Fig. 3. An 8K camera installed at the competition venue.

Fig. 4. Networking in the broadcast center.

Fig. 5. The projection stage at the remote-viewing venue and state of its construction.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

71

Feature Articles(a) Object extraction

Videos taken during the competition(b) Media synchronization

Synchronous transmission of multiple
streams to the remote-viewing venue

Holographic representationin the remote-viewing venue

Fig. 6. Workflow of object extraction and synchronous transmission.

Simultaneous training andinference of two objects(Red: Player in front,
Green: Player in back)Input image• Training image (training)

Simulating thebadminton court space• Result of object extraction(inference)

Depth information

Fig. 7. Method of individual extraction of the front and back players.

3. Kirari! technologies
3.1 R eal-time extraction of objects with arbitrary
background
The real-time extraction of objects with arbitrary
background is a technology for selectively extracting
only the specific objects in images, such as players
and shuttlecocks, from the videos taken during acompetition [2] (Fig. 6(a)).
Normally, extraction of only the objects from videos requires using a green or blue background anderasing the background color by chroma keying. This
technology, however, enables extracting only theobjects from the raw images of the competition venue
in real time without the need for any special background environment.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

To apply this technology to the badminton matches,
we carried out the following five improvements on
our previous system [1] that is for less complicatedcontent.
(1)	Individual extraction of each player in the front
and back sides of the court
Separately extracting the front and back players is
not possible with the conventional technology. To
achieve this, we developed a deep learning model thatcan simultaneously extract and infer front and backplayers by inputting depth information simulating thebadminton court space. This enabled the extraction ofindividual players for a game like badminton where
players are located separately in the front and backsides of the court. We therefore succeeded in stable
and accurate extraction of objects (Fig. 7).
72

Feature Articles

Heatmap: Depth of red
Offset: Arrow

Fig. 8. Method of extracting shuttlecocks.

(2)	Improvement of output-image resolution andframe rate of players’ images
The conventional technology only supports resolutions of up to 4K and frame rates of up to 30 fps. The
maximum resolution of player images that can beachieved using 4K cameras is only approximately
640 pixels; and the roughness of aerial images
becomes evident when they are magnified and projected into life-size images. A frame rate of 30 fps
does not allow tracking of rapid movements such as
when the player is smashing a shuttlecock, leading to
disrupted images. We therefore carried out multilayering of the image frames to be processed andleveling of computing resources to support images
from 8K, 60-fps cameras. As a result, we were able toproduce smooth, high-definition images of players at
930-pixel resolution and 60-fps frame rate.
(3)	Stable extraction of subtle and rapid shuttlecock
movements
Since shuttlecocks are very small and move very
rapidly in the video, using conventional image-recognition methods results in considerable noise andflickering and in discontinuity and incompleteness ofthe shuttlecock trajectory. We therefore developed
extraction algorithms specific for the shuttlecock andsucceeded in detecting its exact position and in accurately extracting its various contours. To detect shuttlecock position, we devised a method for learning
the shuttlecock’s position and movement information
by inputting continuous frames in a convolutional
neural network (CNN) to eliminate the effects ofsmall objects similar to the shuttlecock appearing in
the video (such as seat guide lights). The position of
the shuttlecock was determined on the basis of therough position (heatmap) and correction value (offset) obtained from the CNN (Fig. 8). Extraction of the
shuttlecock image was carried out by generating theimage using a background-subtraction method and
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Fig. 9. C orrecting the black gap zone caused by the court’snet.

filtering with the extracted shuttlecock position andshape and predicted shuttlecock information (position, contour, and motion blur level) in the next
frame. This resulted in a precision of 90.7% andrecall of 90.3%, levels that can be used for realistic
sports viewing.
(4)	Automatic correction of missing parts in player
imagesAs shown in Fig. 9, when the player at the back of
the court overlapped with the net, a black gap zone
appeared on the player image. Images were correctedby inferring the missing parts on the basis of the color
information above and below the gap zone.
(5) Automatic generation of player’s shadow
Reproducing the players’ shadows appearing in theactual venue and in the remote-viewing venue
enables the creation of more natural player images.
We extracted the shadows from the video based on
the results of player-image extraction. We carried out
this step quickly by determining the range where the73Feature Articles

11.7 m12.2 m

Display (for back layer)

Thin half-mirror

7.7 m

10.4 mReal image

Virtual image

Virtual image

Full-size court and net

Real image

Display (for front layer)

Fig. 10. Image of equipment and example of multi-layer aerial image display.

shadows might appear on the basis of the height of theplayer’s jump and lighting conditions of the competition venue.
3.2 H
 ighly realistic media synchronization technology
The highly realistic media synchronization technology (Advanced MMT) is NTT’s proprietary technology developed by extending the media transmission
standard MMT and enables the transmitting of various continuous data (streams), such as video, audio,
and lighting information, while maintaining theirtime synchronization. For this project, we synchronously transmitted multiple streams, such as video of
the competition taken at the venue, audio, images ofplayers extracted from competition videos, extracted
shuttlecock images, video obtained from OBS, etc.
and displayed the necessary data on the remoteviewing venue at the appropriate timing to create ahighly realistic spatial representation (Fig. 6(b)).
3.3 B ird’s eye-view sports viewing multi-layer
aerial image display technology
To deliver the experience of watching as if viewers
were at the competition venue, we created a spacethat physically simulated the actual venue (Fig. 10).
We set up a full-size court and net, as well as a spectator stand with the same downward viewing angle asin the actual venue. We used the bird’s eye-view
sports viewing multi-layer aerial image display techNTT Technical Review

Vol. 19 No. 12 Dec. 2021

nology for projecting the holographic representationof players at the same position within that recreatedspace.
With the conventional holographic projection
method called Pepper’s ghost, aerial images are displayed by reflecting images on the display to a half
mirror tilted at a 45-degree angle. Only a single layerof an aerial image can be displayed. Single-layer display, however, cannot be used to display players infront and behind the net, as in badminton, in different
locations (layers) in the projected space.
With our technology, a spectator stand at the sameheight as the actual competition venue was set up,
and the locations of the court, net, two half-mirrors,
light-emitting diode displays, and the projectors were
optimized to create a realistic appearance of the players in front and behind the net. This enabled the holographically displaying of the two players at theircorrect positions.
The use of two half-mirrors, however, results in a
discontinuous appearance of the shuttlecock, whichmoves back and forth between the two mirrors. Since
this problem is a structural issue arising from having
two physically separated layers, there is no perfectsolution to it. To produce a more natural appearance,
we repeatedly conducted trial-and-error experimentson the timing for switching the shuttlecock’s display
layers and came to the conclusion that the best timing
for switching is when the player hits the shuttlecock.
To determine the best timing for switching, we
74

Feature Articles

Table 1. Impressions of participants.
Viewed in full scale, the sense of presence is completely different.
When the shuttle goes up high, it may be a little difficult to look where
the shuttle is. However, it feels as if the player has warped to the spot.
We can enjoy the real feeling of being at the match venue.
The full scale players were so impressive that we cheered as if beingat the match venue unintentionally.

combined multiple approaches, such as determining
the shuttlecock position through analysis of lateral
court images, determining the timing when the shuttlecock is hit through detection of hitting sounds fromthe audio during the match, and human operators
watching the game manually switching the display
layers. The combination of multiple approaches
enabled the real-time switching of shuttlecock-image
display positions.
4. Results of demonstrations
The technology demonstrations were conducted for
four days from July 30 to August 2 during the badminton tournament finals. Press representatives participated in the viewing demonstrations for not only
the recorded videos of the preliminary rounds but
also for the live coverage of the men’s doubles,
women’s singles, and men’s singles events. Many
participants commented that they were able to gain ahighly realistic experience of the matches (Table 1).
The highly realistic environment with a full-size
court and net and life-size holographic projections ofplayers apparently elicited a sensory illusion that theplayers were actually in front of the participants. This
indicates the possibility of achieving an emotional
connection at a deeper level than when watching on
television. The spectators were able to more easily
feel the energy and vibe during the competitions or
the excitement of players before the start of thematches, and even the emotions associated with winning or losing. When Kento Momota fell to his knees
when he lost in the second round of his match during
the group play stage, the realistic sense of his presence in the remote-viewing venue was very striking.
In terms of object-extraction performance, we succeeded in real-time processing of 8K, 60-fps videos.
There are issues that need to be addressed for extraction accuracy such as when images of the two players
overlap or extraction for doubles matches. Nevertheless, we succeeded in processing the images without
missing parts or errors that could hinder proper viewNTT Technical Review

Vol. 19 No. 12 Dec. 2021

Table 2. Details of lag time.
Camera system 17 ms ( 1 frame)

MMT transmission

1600 ms (4K 59.94p BT.709, rate ofencode: 40 Mbit/s, FEC: 5%)

Extraction of objects 1000 ms (including input/output buffering)
Presenting system 120 ms (4 frames at 30 fps)

ing of the games, even for players wearing different
uniforms for all the six live broadcast experiments.
We also confirmed that there were no errors in thecorrection of the black zones caused by the net and in
the addition of shadows. For the shuttlecock images,
however, more improvements are necessary due toconsiderable discontinuity depending on the viewing
position.
In terms of transmission performance, the total endto-end delay from the competition venue to theremote-viewing venue was less than 2800 ms.
Table 2 lists the lag time details. The MMT transmission delay included network transmission delays ofapproximately 1 ms between the competition venue
and broadcast center and approximately 0.1 ms
between the broadcast center and remote-viewing
venue. Delays in object-extraction processing included buffering that takes fluctuations in processing
time, etc., into consideration, but the actual delay inextraction processing excluding the buffering wasless than 400 ms.
Since the demonstrations involved one-way transmission of images from the competition venue to theremote-viewing venue, an approximately 3-s delay
did not cause major problems. However, our future
goal is to connect the two venues bi-directionally anddeliver the cheers from the audience in the remoteviewing venue back to the competition venue, inwhich case, significant lag times would become problematic, making it necessary to further reduce processing time.
We were able to confirm that the above performance of the demonstration system were sufficient in
75

Feature Articles

delivering a realistic experience of the presence of theathletes in the remote-viewing venue.

technologies
Acknowledgments

5. Summary
As part of the TOKYO 2020 Future Sports Viewing
Project, we applied Kirari! to badminton matches anddemonstrated that it can be used to deliver a realistic
experience of being at the competition venue.
Although there are issues regarding accuracy of
extracting player images and in the method for displaying the shuttlecock, we succeeded in demonstrating the possibility of a new sports-viewing experience, beyond what television can provide, for the
Olympic Games, an event that attracts attention fromall over the world.
Going forward, we will carry out improvements inextraction performance and other technical aspects,
as well as deploy the technology to other games andareas, such as for music concerts. We will also study
methods of minimizing lag time for bi-directional
connection between competition and remote-viewing
venues and conduct research and development on
Kirari! and other technologies for achieving theRemote World and eventually make future proposals
that will surprise the world using communication

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

We would like to thank the members of the Innovation Promotion Office of the Tokyo Organising Committee of the Olympic and Paralympic Games forhelping us resolve various issues in conducting thistechnology demonstration, representatives of theMiraikan museum for organizing and leading the
project, and representatives of partner companies fortheir technical support.
NTT is an Olympic and Paralympic Games Tokyo
2020 Gold Partner (Telecommunication Services).
References
[1]

[2]

Y. Tonomura, H. Imanaka, K. Tanaka, T. Morizumi, and K. Suzuki,“Standardization Activities for Immersive Live Experience (ILE),”
Journal of the ITU Association of Japan, Vol. 47, No. 5, pp. 14–17,
2017 (in Japanese).
H. Kakinuma, G. Nagao, H. Miyashita, Y. Tonomura, H. Nagata, and
K. Hidaka, “Real-time Extraction of Objects from Any Background
Using Machine Learning,” NTT Technical Review, Vol. 16. No. 12,
pp. 12–18, 2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201812fa2.html

76Feature Articles

Makoto Muto

Senior Research Engineer, NTT Human Informatics Laboratories.
He joined NTT in 2001 and engaged inresearch on information retrieval. He is currently
researching segmentating subjectives from video
for sport public viewing applications.

Keisuke Hasegawa

Taiji Nakamura

Senior Research Engineer, 2020 Epoch-Making Project, NTT Human Informatics Laboratories.
He joined NTT DATA Communications Systems (now NTT DATA) in 1991. He has over 20
years’ experience in information systems planning and development in the national public sector.

Kenya Suzuki

Engineer, NTT Human Informatics Laboratories.
He received a B.E. and M.E. in informatics
from Kyoto University in 2012 and 2014. He
joined NTT WEST in 2014 and engaged in themaintenance of network facilities. In 2016, hejoined NTT Service Evolution Laboratories,
where he has been engaged in research on media
processing for Kirari!. He is a member of theInstitute of Image Information and Television
Engineers (ITE).

Senior Research Engineer, Supervisor, NTT
Human Informatics Laboratories.
He received a B.E. in electro-communications
from the University of Electro-Communications,Tokyo, in 1993 and M.E. in electronic engineering from the University of Tokyo in 1995. He
joined NTT Human Interface Laboratories in
1995, where he has been engaged in research on
human-computer interaction and multimedia
applications including web-based signage system development. He is currently interested inmedia art projects such as Kirari!. He is a member of IEICE of Japan and the Physical Society ofJapan.

Daichi Namikawa

Shingo Kinoshita

Senior Research Engineer, NTT Human Informatics Laboratories.
He received a B.E. and M.E. from Kagoshima
University in 2007 and 2009. He joined NTT in2009 and has been researching and developing
enhanced network services and systems.

Seiichi Konya

Research Engineer, NTT Human Informatics
Laboratories.
He received a Master of Electronic Engineering from Hokkaido University in 1990 and joined
NTT Information and Communication Laboratories. He is a member of the Institute of Electronics, Information and Communication Engineers(IEICE).

Vice President, Head of NTT Human Informatics Laboratories.
He received a B.E. from Osaka University in1991 and M.Sc. with Distinction in technology
management from University College London,
UK, in 2007. He joined NTT in 1991 and was asenior manager of the research and development(R&D) planning section of the NTT holding
company from 2012 to 2015. He is currently avisiting professor at the Art Science Department,
Osaka University of Arts, and visiting executive
researcher at Dentsu Lab Tokyo. He has served asa member of the Japan Science and Technology
Agency (JST) JST-Mirai Program Steering Committee, member of the All Japan Confederationof Creativity (ACC) TOKYO CREATIVITY
AWARDS 2021 Judging Committee, and member of the Broadband Wireless Forum Steering
Committee. He has been engaged in R&D of amedia-processing technology, user interface/user
experience, communication protocols, information security, machine learning, service design,and technology management. Until recently, he
had been in charge of NTT’s Tokyo2020 initiatives, including sports-watching video technology, inclusive design for social issues, and promoting the use of ICT in kabuki, entertainment,
and media arts such as live music.
He has been in his current position since 2021,
where he manages R&D on information andcommunication processing of humans based onhuman-centered principles.

Nobuhiro Hirachi

Senior Research Engineer, NTT Human Informatics Laboratories.
He received an M.Sc. in natural science from
Chiba University in 1998. He joined NTT in thesame year and studied fiber optic access networks including optical access systems. He hasdeveloped an asynchronous-transfer-mode-based
passive optical network system for the first offiber-to-the-home services and received thePresident’s commendation. He has been engaged
in research of Kirari! since 2019.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

77

Feature Articles: Olympic and Paralympic Games Tokyo 2020
and NTT R&D—Technologies for Viewing Tokyo 2020 Games

Marathon × Ultra-low-latencyCommunication Technology
Soichiro Usui, Shinji Fukatsu, Eiichiro Matsumoto,
Maiko Imoto, Daisuke Shirai, and Shingo Kinoshita
Abstract

NTT provided ultra-low-latency communication technology to the Tokyo 2020 Real-time Remotecheering Project, which was implemented by the Tokyo Organising Committee of the Olympic andParalympic Games for the marathons of the Olympic and Paralympic Games Tokyo 2020 held in Sapporo on August 7 and 8, 2021. This project proposed a new means of watching sports by connecting the
marathon course in Sapporo with the cheering venue in Tokyo in real time. By delivering the cheers ofspectators to the athletes from the remote location in Tokyo, it was possible to create a sense of unity
between the athletes and spectators, as well as a sense of realism similar to roadside cheering while
maintaining safety and security. Our efforts concerning this project and the technologies that we used are
introduced in this article.
Keywords: ultra-low-latency communication technology, remote cheering, IOWN

1. Overview of the project
During the novel coronavirus (COVID-19) pandemic, it has become increasingly difficult to cheer
on athletes directly at competition venues. By using
new communication technologies, the Tokyo 2020
Real-time Remote-cheering Project aimed to create aworld in which athletes at a competition venue andspectators in a remote area can share the excitement,
emotion, and sense of unity at the venue (Fig. 1).
As a Gold Partner of the Olympic and Paralympic
Games Tokyo 2020, NTT has been contributing to thecreation of the most-innovative Games in history byproviding cutting-edge communication technology.
In light of the situation concerning the COVID-19
pandemic, we adopted a new mission to propose a
new form of sports viewing using communication
services and make it a springboard for the future anddecided to provide the technology for this project.
Regarding remote spectator experience, delay timeis the largest challenge in transmitting cheering fromremote venues to the competition venue. For marathons in particular, even a small delay can have asignificant effect on the ability to transmit cheering to
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

athletes running at 5 m/s. The total delay in watchinga sporting event remotely had been several seconds
each way, which includes not only the propagation
delay of light but also media-processing delay such astransmission-processing delay and compression
delay for video information; as a result, it has been
impossible to transmit cheering to players, athletes,
etc. without a delay. NTT undertook the Tokyo 2020
Real-time Remote-cheering Project with the goal ofsolving these problems by using its ultra-low-latency
communication technology to minimize the latencyof the transmission process to about 100 ms each way
and ensure the transmission of cheering to the athletes.
2. Device configuration of the project
The overall device configuration of the project isshown in Fig. 2. We aimed to create a space in whichremote spectators could feel as if they were cheering
along the roadside as the runners were running by.
Therefore, we installed light-emitting diode (LED)displays (approximately 50 m wide and 2 m high) at
both the Sapporo marathon course (in front of Sapporo
78

Feature Articles

SapporoTokyo

Fig. 1. Overview of the Tokyo 2020 Real-time Remote-cheering Project.

Marathon course (Sapporo)
LED display (approximately 50 m wide and 2 m high)EightcamerasMedia-processing device
Transmission device

Ultra-low-latency media-processing technology

20-Gbit/s high-speed
broadband line

Remote-viewing venue (Tokyo)

Disaggregated configuration for video transport

Transmission device
Media-processing device

LED display (approximately 50 m wide and 2 m high)EightcamerasFig. 2. Device configuration of the project.

Sosei Square) and Tokyo remote-viewing venue. The
displays enabled the runners and spectators to seeeach other in actual size. The total width of the LED
displays at both sites was approximately 50 m, but thelocation and spacing of the displays varied in accorNTT Technical Review

Vol. 19 No. 12 Dec. 2021

dance with the conditions at the installation site
(Figs. 3(a) and (b)). Eight cameras were set up in
front of the LED displays at both sites to capture
images of spectators and runners (Fig. 3(c)). The 4K
images from each of the two sites were transmitted to79

Feature Articles(a) 50-m-wide LED display installed at
the Tokyo venue(b) 50-m-wide LED display installed along
the Sapporo marathon course (in frontof Sapporo Sosei Square)(c) Eight cameras and microphones
installed along the marathon
courseFig. 3. The setup at both venues.

Fig. 4. NTT network equipment that provided ultra-low-latency communication technology.

the other site via a 20-Gbit/s high-speed broadband
line via a media-processing device using ultra-lowlatency media-processing technology and transmission device using disaggregated configuration forvideo transport. These two technologies constitute
our ultra-low-latency communication technology.
3. Ultra-low-latency communication technology
The Innovative Optical and Wireless Network(IOWN) is an innovative concept proposed by NTT
as a next-generation communication platform. IOWN
will enable the creation of a network and information-processing infrastructure that includes devices
capable of providing high-speed, high-capacity comNTT Technical Review

Vol. 19 No. 12 Dec. 2021

munications and vast computing resources by using
innovative technologies centered on optical technology. One of the components of IOWN, the All-Photonics Network (APN), will achieve ultra-low-power
consumption, ultra-high capacity, and ultra-low
latency by introducing photonics-based technology
into everything from the network to devices. The
ultra-low-latency communication technology used inthis project uses two APN elemental technologies:
disaggregated configuration for video transport andultra-low-latency media-processing technology(Fig. 4).

80

Feature Articles

Videotransmission
control function

Switching
controlfunctionxKD client
function

Opticaltransmission
control function

Softwarefunctiongroup

Coherent
DSP

Hardwarefunction
group

Network OS (Linux-based)
x86 architecture

White-box switch

SwitchASIC

Video PIU

Encryptionfunction

Optical transmission

SDI video signal
ASIC: application-specific integrated circuit
DSP: digital signal processor
OS: operating system

PIU: plug-in unit
xKD: quantum key distribution or post-quantum
cryptography-based key distribution

Fig. 5. Disaggregated configuration for video transport.

3.1 D isaggregated configuration for video transport
Disaggregated configuration for video transport
enables flexible changes in configuration, addition offunctions, and cost reduction by separating network
functions that were previously provided in combination and configuring them so that each function canbe controlled by standardized interfaces. We developed a plug-in unit (video PIU) in which a direct
serial digital interface (SDI)-signal acquisition function is added to a white-box switch with an opticaltransponder for long-distance transmission. The newvideo PIU enables direct transmission of uncompressed video and audio in SMPTE ST 2110* format
over optical long-distance transmission lines. As aresult, the delay between the video input on the transmitter side and video output on the receiver side was
reduced to about 1 ms, and the one-way delay(including the distance delay between Tokyo andSapporo) was reduced to about 20 ms. We also implemented a set of functions for video routing in the
network operating system, which is the control software for the white-box switch, to achieve integrated
operation of network and video devices (Fig. 5).
3.2 U
 ltra-low-latency media-processing technology
Ultra-low-latency media-processing technology
captures SDI video signals output from a camera on asubframe or line basis and executes video processing
such as geometric transformation and composition at
the video-signal level. Implementing each video process at the video-signal level instead of at the frame
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

level makes it possible to reduce the frame latency
required by conventional video processing andachieve low latency (Fig. 6).
In this project, when transmitting video from multiple cameras, the SDI video signals from each camera were made into subframes in line units, and thereceived subframes were immediately separated anddisplayed on multiple displays in a manner that
achieved even lower latency (Fig. 7).
4. Results of the project
On the days of the men’s and women’s marathons,
after a demonstration of low latency was conducted,
the remote spectators cheered for the runners. Via the
50-m-wide display, the Tokyo venue provided a senseof realism to the spectators, namely, the feeling thatthe athletes were running right in front of them, andthe spectators could show their support by clapping
and waving flags. For past remote-cheering experiences, due to delays, the timing of the cheering was
not synchronized with the athletes, so the spectators
found it difficult to feel a sense of unity while cheering for the athletes. For the Sapporo marathon course,
however, the ultra-low-latency communication technology enabled the spectators to see their cheeringreaching the runners as the runners passed by. As aresult, the spectators were able to feel the same senseof unity even in the Tokyo venue, which made them*

SMPTE ST 2110: A standard developed by the society of motion
picture and television engineers (SMPTE) for transmitting video
over Internet Protocol (IP) networks for the professional media
industries.

81

Feature Articles

Conventional video processing at the frame level

Video processing(Parallelization)

CameraSeparation

Input
buffer

Frame

Frame
Combination

Frame

SDI data
stream

SDI data
streamMonitor

OutputbufferBuffering• Separation

Buffering• Combination

Our video processing at the video-signal level
SDI data
streamCameraFrame

Video processing(Parallelization)

SDI data
streamPreprocessing

Postprocessing

MemoryMemoryMonitorFig. 6. Ultra-low-latency media-processing technology.

Camera input 1 Camera input 2 Camera input 3 Camera input 4 Camera input 5 Camera input 6 Camera input 7 Camera input 8
512 px1080 px

Cutout andaggregation

1920 px

3840 px

Geometric transformation by LUT (Look Up Table)

subframe 1

subframe 22160 px
subframe 3

subframe 4

Disaggregation

1920 px1080 px
Cameraoutput 1

Cameraoutput 2

Cameraoutput 3

Cameraoutput 4

Cameraoutput 5

Cameraoutput 6

Cameraoutput 7

Cameraoutput 8

Fig. 7. Aggregation and disaggregated processing of multiple camera images.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

82

Feature Articles©2021 – IOC/Getty for NTT- All rights reserved Tokyo 2020(a) Cheering from Tokyo to Sapporo during the women’smarathon©2021 – IOC/NTT- All rights reserved Tokyo 2020(c) Cheering from Tokyo to Sapporo during the men’smarathon©2021 – IOC/Getty for NTT- All rights reserved Tokyo 2020(b) Cheering from Tokyo reaching the runners during thewomen’s marathon©2021 – IOC/NTT- All rights reserved Tokyo 2020(d) Cheering from Tokyo reaching the runners during the
men’s marathon

Fig. 8. Scenes from the project during the marathons.

even more enthusiastic about cheering (Figs. 8(a) and(b)).
Through this cheering experience, the spectators at
the Tokyo venue commented, “I could experience thesame sense of realism and speed as cheering at theroadside,” “I felt that I could cheer from a position
closer to the athletes than possible at the roadside,which is usually crowded,” and “Above all, it was
lovely to see the cheering reach the athletes.” In otherwords, the spectators seemed to be very satisfied withthe sense of unity by having their cheering reach the
marathon venue, an outcome that had been impossible with the remote cheering. The significance of thisproject was also confirmed by the fact that some of
the marathon runners showed interest in the real-timecheering from the remote area via the display, andthat result was a world’s first. As expected, the delay
time for the transmission process was kept to 100 ms
each way. Unfortunately, typhoon Mirinae wasapproaching Japan, and the men’s marathon was held
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

in rainy weather on August 8th; nevertheless, after
ensuring safety of the athletes and spectators, wewere able to transmit cheering from Tokyo to Sapporo in spite of the rain (Figs. 8(c) and (d)).
5. Future developments
In light of the results of this project, NTT aims tocreate a new style of watching sports competitions
that allows people who are unable to cheer at a competition venue to experience the sense of presence
and unity, which could only be experienced at thevenue, in a safe and secure manner. Regarding theTokyo 2020 Real-time Remote-cheering Project, thecheering was transmitted from a special venue inTokyo. We will accelerate our research and development so that our ultra-low-latency communication
technology can be widely used and the same realtime remote cheering will be possible from homes.

83Feature Articles

NTT is an Olympic and Paralympic Games Tokyo
2020 Gold Partner (Telecommunication Services).

Soichiro Usui

Senior Manager, Strategic Business Creation
Team, Research and Development Planning
Department, NTT.
He received a B.S. in industrial engineeringand management from Tokyo Institute of Technology in 1999. He joined NTT the same year
and was involved in corporate business and network services development at NTT EAST. He
was with NTT Research and Development Planning Department from 2012 to September 2021,
where he was in charge of information and communication technology (ICT) business creation
for the entertainment and sports domain. He has
been with NTT EAST since October 2021.

Shinji Fukatsu

Senior Research Engineer, Supervisor, CyberWorld Laboratory, NTT Human Informatics
Laboratories.
He received a Ph.D. in engineering from Osaka
University in 2002 and joined NTT the same
year. He has been engaged in research and development (R&D) of human interfaces and video
streaming services as well as in planning anddevelopment of video streaming services at NTT
Plala and in promoting standardization and international development of ICT at the Ministry ofInternal Affairs and Communications. He is currently engaged in R&D of remote-world infrastructure technology.

Eiichiro Matsumoto

Senior Research Engineer, Cyber-World Laboratory, NTT Human Informatics Laboratories.
He received a B.E. in science engineering(information science) from Meiji University in1999 and joined NTT the same year, from August
he was engaged in the development of geographic information systems and voice/videocommunication systems at NTT EAST. From2008 to 2010, he was involved in the R&D of IP
television (TV)/mobile broadcasting standards
and the formulation of technical standards at
NTT. Since 2010, he has been engaged in theplanning and development of network services
and systems for video distribution systems, suchas IPTV and radio-frequency TV and researching
video-transmission-system technologies for adding high value to relay networks and access networks at NTT EAST. He has been at his current
position since 2020.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Maiko Imoto

Research Engineer, Cyber-World Laboratory,
NTT Human Informatics Laboratories.
She received an M.S. in computer science from
Ochanomizu University, Tokyo, in 2011. She
joined NTT the same year and began studying
geographical information search for mobile
phones and HTML5-based e-learning applications. She was involved in R&D on Kirari!
immersive telepresence technology. From 2017
to 2020, she was responsible for the developmentof live-video distribution platforms for broadcasters and creating live viewing services for
sports and entertainment at NTT Plala. She has
been engaged in R&D of multimedia data interactive transmission technologies.

Daisuke Shirai

Senior Research Engineer, Supervisor, Frontier
Communication Laboratory, NTT Network Innovation Laboratories.
He received a B.E. in electronic engineering,
M.E. in computer science, and Ph.D. in media
design from Keio University, Kanagawa, in 1999,
2001, and 2014. He pioneered the world’s first
4K JPEG 2000 codec system, which enables low
latency 4K60p video transmission on a Gigabit
network. He has applied his expertise acrossmultiple domains through his study of practicalapplications in digital audio and video broadcasting technology, image coding, information theory, networking, human-computer interaction, andsoftware architecture. His current research topics
include remote video production networks, ultralow latency visual communication, and its security over optical transport networks.

Shingo Kinoshita

Vice President, Head of NTT Human Informatics Laboratories.
He received a B.E. from Osaka University in1991 and M.Sc. with Distinction in technology
management from University College London,
UK, in 2007. He joined NTT in 1991 and was asenior manager of the R&D planning section of
the NTT holding company from 2012 to 2015.
He is currently a visiting professor at the Art Science Department, Osaka University of Arts, andvisiting executive researcher at Dentsu Lab
Tokyo. He has served as a member of the Japan
Science and Technology Agency (JST) JSTMirai Program Steering Committee, member ofthe All Japan Confederation of Creativity (ACC)TOKYO CREATIVITY AWARDS 2021 Judging
Committee, and member of the Broadband Wireless Forum Steering Committee. He has been
engaged in R&D of a media-processing technology, user interface/user experience, communication protocols, information security, machinelearning, service design, and technology management. Until recently, he had been in charge ofNTT’s Tokyo2020 initiatives, including sportswatching video technology, inclusive design forsocial issues, and promoting the use of ICT in
kabuki, entertainment, and media arts such as live
music.
He has been in his current position since 2021,
where he manages R&D on information andcommunication processing of humans based onhuman-centered principles.

84

Global Standardization Activities

High-accuracy Time-synchronization
Technology for Low-latency, Highcapacity Communications in the 5G
and Beyond 5G eras
Kaoru Arai and Makoto Murakami
Abstract

High-accuracy time-synchronization technology has been attracting attention for enabling services
that use high-accuracy time information, such as fifth-generation (5G) mobile communications, highfrequency financial transactions, high-definition video distribution, and smart grids. The International
Telecommunication Union - Telecommunication Standardization Sector (ITU-T) Study Group (SG) 15
and the Institute of Electrical and Electronics Engineers (IEEE) mainly carry out international standardization. In this article, we introduce the time-synchronization requirements for 5G as well as next-generation communication technologies, such as Beyond 5G and quantum communications, an overview oftime-synchronization technology/Precision Time Protocol (PTP), the standards of which have recently
been updated by IEEE, and the latest discussion topics in ITU-T SG15.
Keywords: time synchronization, PTP, 5G/Beyond 5G

1. Background oftime-synchronization technology
Time synchronization means that the time between
systems in a communications network is the sameand is consistent with Coordinated Universal Time
(UTC). Time synchronization is required for fourthgeneration (4G) and later mobile communications,and technology has been developed to deliver UTCsynchronized time information to mobile base stations. As information technology (IT) services havebecome more diversified and higher quality, theapplication of time information has expanded beyond
the mobile field. Examples include high-accuracy
time stamping in high-frequency trading in the financial sector, synchronization in distributed processing
among datacenters, frame synchronization in video
streaming, and time synchronization among instruments in large-scale scientific experiments such as
accelerators in atomic physics.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Time-synchronization technology plays an important role in various industries and scientific fields, as
described above, is discussed in many forums andstandardization organizations, and is being commercialized and introduced into services. The International Telecommunication Union - Telecommunication Standardization Sector (ITU-T) has been discussing the requirements for time synchronization incommunication networks and specifications for highaccuracy time-synchronization devices.
2. Use cases and requirements of timesynchronization in telecommunications
We first introduce the use cases of time synchronization in telecom networks that are under consideration by ITU-T and example requirements for mobile
and industrial applications.
In 4G and later mobile communications, the same
frequency is time-shared and transmitted for both85

Global Standardization Activities

Base station
With guard band
Interference

Interference
Operator A

Operator B

Operator C

User equipment
Frequency

Guard band

Time error occurs at operator B

Guard band

Operator A

Without guard band
Operator A

Operator B

D

Operator B

Operator C

Operator C
Frequency

Effective use of bandwidth for guard bands
× Interference between operators(a) Frequency-slot assignment example

U
D

D

D
U

U

U
D

D

D
U

U

U
D

D

D
U

U

U
D

D

U

U

Uplink signals

D

Downlink signals

UTime error

Interference between operators
Signal B downlink interferes with signals A and C uplink.

(b) Interference with no guard band between operators

Fig. 1. Radio-wave interference between operators due to time-synchronization errors.

uplink and downlink communications (i.e., time division duplex (TDD)) to effectively use the bandwidthof mobile base stations. For TDD, a time-synchronization accuracy of +/–1.5 µs is required. However, tomaximize the use of the radio frequency band allocated to each telecom operator in 4G/5G communications, a system without inter-operator guard bands isused. Therefore, if the time-synchronization error
increases, there are concerns about radio-wave interference between operators (Fig. 1(a)). This suggeststhat if the time error between certain operators
increases, the uplink and downlink frame patterns
may be reversed, and interference may occur withother neighboring operators (Fig. 1(b)). In Japan, 4G
and later operate without guard bands; hence, there
are concerns about the effects of interference due totime-synchronization errors [1]. To avoid this problem, it is necessary to synchronize to a common timestandard (e.g. GPS (Global Positioning System))among operators and unify the uplink and downlink
TDD frame structure. NTT clarified these interference mechanisms and detailed conditions of theiroccurrence due to time-synchronization errors, proposed consideration of them in ITU-T Study Group(SG) 15, and reflected them in requirements for timesynchronization in Recommendation G.8271 at theJanuary 2020 meeting.
The Institute of Electrical and Electronics Engineers (IEEE) has standardized time-sensitive netNTT Technical Review

Vol. 19 No. 12 Dec. 2021

working (TSN) as an industrial Ethernet standard toenable real-time monitoring and control with low
latency and time synchronization and has specified
standards for time synchronization in IEEE802.1AS.
One proposed use case is to transmit time information
over the 5G network of a telecom operator and deliver it to industrial applications assumed for TSN(Fig. 2). At the ITU-T SG15 meeting in April 2021, abudget of 900 ns was proposed as the time accuracy
required for the 5G network specified by ITU-T.
Details of the requirement will be discussed in the
future.
As future use cases of time synchronization, applications to next-generation communications, such as
quantum communications and quantum cryptography, are also being studied. In ITU-T, standardizationof quantum key distribution (QKD) is actively being
discussed. One of the methods of QKD requires highaccuracy time synchronization to detect weak photon
signals from the sender at the receiver with accurate
timing. Certain telecom operators are also interested
in time synchronization in QKD, and requirements
for synchronization technologies for these systems
are expected to be discussed in the future.
3. Enhancement of time synchronization—New
PTP standards
Next, we introduce trends in Precision Time Protocol
86

Global Standardization Activities

TSN time source

Transfer TSN time via the 5G network
TSN applications

TSN

5G network

Automation/motion control

Streaming
ITU-T
Minimum 900 ns

IEEE802.1AS

Smart grid

IEEE802.1AS

Fig. 2. Transmission of TSN time information via 5G network.

IEEE
Basic protocols
IEEE1588-2008 (PTP v2.0)
IEEE1588-2019 (PTP v2.1)

IEEE

IEEE

ITU-T

SMPTE

CERN

General industry

Electric power

Communications

Video distribution

Scientific experiments

IEEE 802.1AS(TSN)

C37.238(PTP power profile)

G.826x Series
G.827x Series(PTP telecom profile)

SMPTE ST 2059-2(SMPTE profile)

White Rabbit Project

SMPTE: Society of Motion Picture and Television Engineers
CERN: European Organization for Nuclear Research

Fig. 3. Relationships among standardization bodies for PTP.

(PTP), which is a protocol to achieve high-accuracy
time synchronization. PTP synchronizes time
between two systems by exchanging dedicated packets that are embedded time information [2]. The basic
PTP has been standardized as IEEE1588. On thebasis of IEEE1588, standardization organizations andresearch institutes in various industrial fields maintain compatibility and extend the protocol according
to requirements for various usage scenes (Fig. 3). The
ITU-T supervises the communications field anddefines PTP specifications for communications networks to meet requirements of the mobile applications discussed in the 3rd Generation Partnership
Project (3GPP). IEEE supervises the industrial field
and is expected to support private 5G*1 by using the
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

TSN standards mentioned earlier.
The current PTP was standardized as IEEE15882008 (Version 2.0) in 2008. Its extended specification
was standardized as PTP telecom profile by ITU-T. It
was then introduced to commercial communication
devices and deployed to telecom networks worldwide. A new standard was also completed as
IEEE1588-2019 (Version 2.1) and released in 2020[3]. Version 2.1 is backward compatible with Version
2.0, although the standard documentation has beensignificantly increased from about 300 pages in*1 Private 5G: Differing from 5G provided by telecom operators,
these are 5G systems built and provided locally by companies
and local governments according to the individual requirements
of industries.

87

Global Standardization Activities

Table 1. Major additional functions in IEEE1588-2019.
Main perspectives
Flexibility

Robustness

Accuracy

Additional functions

Major changes and additions

Mixed packet forwarding methods

Multicast/unicast mixable

Multi-profile

Multiple profiles superimposable on the same network

PTP domain redundancy

Multiple domain numbers can be specified for a PTP device

Security

Functions added to check PTP packet for abnormalities

Monitoring

Added slave port monitoring function

High-accuracy profile

Added parameters to achieve sub-nanosecond accuracy

Pre-calibration

Added calibration function for delay asymmetry

IEEE

W

Incompatible
IEEE1588-2002
Version 1.0

NECompatible
IEEE1588-2008
Version 2.0

IEEE1588-2019
Version 2.1

Alignment and expansion

Under alignment

ITU-T
Revisions
PTP telecom profile
G.8265.1, G.8275.1/75.2

PTP telecom profile
G.8265.1, G.8275.1/75.2

Currently being updated to support IEEE1588 Version 2.1

Fig. 4. Evolution of IEEE and ITU standards for PTP.

Version 2.0 to about 500 pages in Version 2.1 as many
new functions have been incorporated.
In response to the increasing importance of timesynchronization in applications and the emergence ofhigh-accuracy time-synchronization requirements invarious fields, IEEE1588-2019 was expanded fromthe previous PTP standards mainly in terms of flexibility, robustness, and accuracy (Table 1). Regarding
flexibility, the PTP packet forwarding method allows
both unicast and multicast packets to be mixed. In
terms of robustness, features related to security functions, quality monitoring of PTP receiving ports, andredundancy of PTP domains*2 were added to enhance
the protocol’s resistance to abnormal operations.
Accuracy has been improved from the nanosecond
level to sub-nanosecond (in the order of 0.1 ns) level,
and a calibration function was added for optical path
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

differences between uplink and downlink of opticalfibers and time-stamp errors between egress andingress in a PTP device, which affect synchronization
accuracy. To achieve compatibility between
IEEE1588-2019 and the PTP telecom profile, ITU-T
is in the process of updating the recommendations(Fig. 4). A number of telecom profile parameters
were added and modified to comply with IEEE15882019 at the April 2021 meeting.
4. Improving the accuracy of timesynchronization—PTP devices
ITU-T has been studying to improve the accuracy*2 PTP domain: A logical group of PTP devices belonging to a single time-reference device.

88

Global Standardization Activities

BC (Master): Regenerate PTP packets

TC: Transparent transmission of
PTP packets

BC (Slave): Terminate PTP packets

T-TC

T-BC

T-BC

Slave MasterSlave Master

Time error per BC step
Class
A
B
C
D

Maximum time error
100 ns
70 ns
30 ns (newly agreed)
Under discussion

Time error per TC stepClass

Maximum time error

ABC

100 ns
70 ns
30 ns? (underdiscussion)

Fig. 5. Accuracy requirements for PTP devices.

of PTP devices to meet the requirements of 5G and is
still discussing how to further improve the accuracy.
ITU-T specifies a telecom-boundary clock (T-BC) in
G.8273.2 as a device that transmits time information
over a network using PTP [2]. The T-BC terminates
PTP packets and relays them to the next device(Fig. 5). The maximum absolute time error per T-BC
was specified as Class B at 70 ns, but then a new standard was newly specified as Class C at 30 ns. Class D
regulations to achieve even higher accuracy have also
been under discussion from 2019 onwards. The telecom-transparent clock (T-TC), which transmits PTP
packets transparently without terminating them, isdefined in G.8273.3 (Fig. 5). For T-TC, discussions
on high-accuracy classes after Class C have also
begun, and certain Class C error parameters havebeen agreed at the September 2020 meeting. The
maximum absolute time error has not yet been specified, but there is a proposal to set it to 30 ns, which isequivalent to T-BC Class C. This is expected to be
discussed in the future.
5. Improving the accuracy of timesynchronization—Delay measurement
Since PTP is a protocol in which uplink and downlink latencies are assumed constant, ITU-T SG15 has
been actively discussing delay of a PTP device andoptical-fiber propagation delay, which are the maincauses of synchronization-accuracy degradation.
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Regarding delay of a PTP device, ITU-T is concerned about the effect of processing-delay variations
at lower layers that cannot be eliminated by the
hardware-timestamping function specified in
IEEE1588 (Fig. 6(a)). There is an opinion that thetime of delay should be specified with nanosecond
accuracy for data processing at the physical layer in
optical modules. It has also been pointed out that thedelay variation in frame multiplexing and forward
error correction (FEC) in digital signal processing
used for optical signal transmission and reception isseveral tens of nanoseconds. Since the delay variationof a device is directly related to the time errors of
PTP, a clear specification of the fixed delay and control of the variation are required. However, there isalso the dissenting opinion that it is difficult to specify the fixed delay of the optical module configurations and FEC processing as a standard because of thedependency on system-vendor implementation.
These issues are therefore currently under continuous
study.
Regarding optical-fiber propagation delay, thedelay difference between an uplink and downlink ofless than 1 ns must be measured for a route of PTP
packets. In optical-fiber propagation delay, the optical-path-length variation caused by the refractive
index change and mechanical expansion and contraction of the optical fiber when the ambient temperaturechanges must be considered. The calibration functionof the difference in delay specified in IEEE1588-2019
89

Global Standardization Activities

IEEE1588
Application

Time-stamp pointOS

Delay-variation compensation

MAC
Time-stamp point(Hardware processing)

PHY

Subject to delay variation
Network(a) Effects of delay variation in PTP protocol stack

PTP device

Single fiber transmission

PTP device

Wavelength A
Wavelength B

Uplink and downlink delay difference
due to wavelength difference

Delay-difference changes due
to changes in the ambient environment(b) Delay asymmetry in single-core optical-fiber transmission
MAC: Media Access Control
OS: operating system
PHY: physical layer

Fig. 6. Effects of device-internal and optical-fiber-transmission line delays.

is based on the assumption that PTP devices are
directly connected by optical fibers and that there is
no change in difference of delay during operation.
However, considering complex telecom networks,
the need to dynamically measure delay differenceswith high accuracy has been raised in ITU-T. In particular, the coherent network Primary ReferenceTime Clock (cnPRTC)*3 [4], which is a concept for
high-accuracy time-generation architecture that NTT
has proposed and standardized jointly with other telecom operators, uses a network to connect distributed
PRTC devices (time-reference devices) and compare
their time information. At the April 2021 meeting, it
was agreed that requirements of the time-transmission accuracy for the comparison link between
PRTCs should be very high—Class A: 5 ns, Class B:
1 ns. In Class B, PTP packet transmission is presumably executed using a single optical fiber for biNTT Technical Review

Vol. 19 No. 12 Dec. 2021

direction. Since uplink and downlink PTP packets aretransmitted at different wavelengths on a fiber, there
are wavelength-dispersion effects in which opticalfiber propagation delay varies depending on the
wavelength (Fig. 6(b)). To address this, experts invarious technical fields are also jointly discussing
technologies to accurately measure optical-fiber
propagation delay. Such delay can generally be measured using optical time-domain reflectometry(OTDR)*4. The use of correlation OTDR, whichenables high-accuracy measurement by modulating*3 cnPRTC: A new architecture that generates optimal time by comparing the times of distributed PRTCs with each other to improve
the time accuracy of the PRTC that serves as the time reference.
*4 OTDR: A technology for measuring the transmission loss anddistance of an optical fiber and detecting fracture points by injecting optical pulses into the fiber and measuring the reflected
pulses.

90

Global Standardization Activities

optical pulses and correlating them at the receiver, isalso under discussion. However, issues remain suchas the effect of the differences between the OTDR
measurement wavelength and PTP transmission
wavelength and how to deal with a case making it
impossible to secure the reflection path of opticalpulses because of optical amplifiers deployed along
the section to be measured.
6. Future synchronization technologies
As a synchronization signal reference, the use of
optical clocks*5 is being considered as next-generation high-accuracy atomic clocks to replace the conventional cesium atomic clocks. Optical clocks are
expected to surpass commercial cesium atomic
clocks in accuracy by more than five orders of magnitude and are attracting attention as a new standardof seconds for the future. In ITU-T SG15, proposals
have begun for new work items such as a new timesynchronization method using UTC-synchronized
optical clocks as a time source and monitoring ofoperational cesium atomic clocks with high-accuracy
optical clocks. NTT has proposed a next-generationsynchronization network concept based on opticalclocks and will continue to actively propose requirements for next-generation synchronization devices

and networks.
7. Future developments
The use of 5G and Beyond 5G networks will be
expected to diversify, and many IT services using
highly accurate time information will emerge in the
future. NTT will continue to actively contribute to thestandardization of synchronization technology bysearching for potential service requirements andmonitoring the latest technological developments.
References
[1]
[2]

[3]
[4]

M. Fujii, S. Suwa, R. Toba, and T. Toeda, “Base-station Equipmentwith the Aim of Introducing 3.5-GHz band TD-LTE,” NTT DOCOMO Technical Journal, Vol. 18, No. 2, pp. 8–13, Oct. 2016.
K. Arai and M. Murakami, “Overview of Network Synchronization
Technology Standardization in ITU-T,” NTT Technical Review, Vol.
14, No. 2, Feb. 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201602gls.html
IEEE Std 1588-2019: “IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems,”
2019.
K. Arai and M. Murakami, “Trends in Standardization of High-precision Time- and Frequency-synchronization Technology for Creating a
5G Mobile Network,” NTT Technical Review, Vol. 17, No. 1, pp.
35–42, Jan. 2019.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201901gls.html*5 Optical clock: The next generation of atomic clocks based on optical frequencies. Research institutes and universities worldwide
have been actively working on improving the accuracy of thesetechnologies. Optical clocks are promising candidates to become
the next-generation standard to replace the cesium atomic clock,the current standard of seconds.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

91

Global Standardization Activities

Kaoru Arai

Research Engineer, NTT Network Service
Systems Laboratories.
He received a B.S. and M.S. in applied physics
from Tokyo University of Science in 2010 and2012. Since joining NTT Network Service Systems Laboratories in 2012, he has been researching and developing network systems such as
clock supply systems and time synchronization
systems. He has been participating in ITU-T
SG15 activities concerning synchronization
technologies since 2014. He received the Distinguished Service Award regarding standardization
activities of synchronization technologies fromthe Telecommunication Technology Committee(TTC) of Japan in 2020. He is currently engaging
in research of next-generation synchronization
technologies.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Makoto Murakami

Researcher, NTT Network Service Systems
Laboratories.
He received a Ph.D. in electrical engineeringfrom the University of Tokyo in 2009. He initially engaged in the research and development(R&D) of long-haul transmission systems using
optical amplifiers and coherent modulation/
demodulation schemes at the emergence of thosetechnologies. After completing development anddeployment of a commercial optically amplified
submarine system, he continued R&D of wavelength-division multiplexing systems to further
increase the fiber-transmission capacity. From2001 to 2003, he worked for NTT Communications, where he was involved in the construction
and operation of international communication
networks mainly in the Asia-Pacific region. Since
2003 he has been an active participant in ITU-T
SG15 as head of the Japanese delegation and has
also been involved in R&D and standardizationof large-capacity optical transport networks. He
is currently the chairman of the transport networks and EMC (Electro-Magnetic Compatibility) Working Group at TTC of Japan. He received
the Accomplishment Award from the ITU Association of Japan in 2015 and the DistinguishedService Award from TTC in 2015.

92

Practical Field Information about
Telecommunication Technologies
Practical Field Information about Telecommunication Technologies

Case Studies of Communication
Problems with Video Transmission
Systems Using Wireless LANs
Technical Assistance and Support Center, NTT EAST
Abstract

This article introduces recent problems that have occurred when using wireless local area networks forvideo transmission and solution to these problems. This is the sixty-seventh article in a series on telecommunication technologies.
Keywords: wireless LAN, channel interference, video transmission, low-emissivity glass

1. Introduction

2. Problem case with security-camera systembuilding. One of these was on the first floor and theother was on the second floor. These wireless APs
were connected to four security cameras via wirelessrepeaters. These cameras and wireless repeaters were
installed on four poles for outdoor lighting. The
images from the cameras were stored in the cloud via
the wireless LAN and Internet. The customer reported that video streams from three cameras, which wereconnected to the wireless APs on the second floor,
were frequently interrupted.
The wireless LAN used the 2.4-GHz band, which is
generally considered to be affected by interference.
Accordingly, the maintenance engineer switchedfrom that band to the 5-GHz band (W56), which isless affected by interference and generally used outdoors. However, interruptions were not resolved, so
the Technical Assistance and Support Center (TASC)
was asked to investigate the problem.

2.1 E
 quipment configuration and problem situation
A problem with a security-camera system using awireless LAN was reported by our customer. The
equipment configuration is shown in Fig. 1. Twowireless-LAN access points (wireless APs) were
installed near the windows of a two-story office

2.2 Investigation of cause
We set the frequency used for the wireless LAN to
the 2.4-GHz band and measured the radio-wave environment and the received signal strength.
(1) Investigation of radio-wave environment
We used a wireless LAN tester, which was developed by TASC, to measure radio waves in theInternet access using wireless local area networks(LANs) has become available both for indoors andoutdoors. We use wireless LANs for a wide variety of
purposes, which include not only accessing websites,
sending and receiving emails, and social networking
but also video distribution. However, when designingand constructing wireless LAN systems for video
transmission such as to display security-camera
images or the screen of a personal computer (PC) ona large display, it is necessary to take into accountdata delays and losses. This article introduces two
recent problems that have occurred when using wireless LANs for video transmission and solutions ofthese problems.

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

93

Practical Field Information about Telecommunication Technologies

Cloud

WirelessAP

Wireless repeater
2.4-GHz band

Camera #4
images (interrupted)
Camera #3
images (interrupted)#2

Camera #2
images (interrupted)

ONU

Camera #1
images (normal)

Customer’s #1
building

PoE

Securitycamera

#4

#4

#3

#3

#2

#2

#1

#1

Carpark (outdoor)

ONU: optical network unit
PoE: Power over Ethernet

Fig. 1. Equipment configuration.

Low-emissivity metallic
coating on innersurfaces of glass
Exterior
walls

Exterior
walls

Wireless repeater #2

AP#2
24.6 mWindow glass

CameraMeasuredvalue: −31 dBm

Measured value: −71 dBm

Cross section of
exterior wall
Attenuation caused by window = measured transmission loss - distance attenuation

Fig. 2. Attenuation caused by low-emissivity glass.

2.4-GHz band and found no interference or otherwireless-LAN waves that could cause radio-wave
interference.
(2) Measurement of received signal strength
Using a measuring instrument that can measure theradio-wave strength of devices, we measured thereceived signal strength of the radio waves near thewireless APs and wireless repeaters. The received
signal strength of the radio wave for downlink transmitted from a wireless AP near a wireless repeater
was less than –60 dBm, and that for uplink transmitted from a wireless repeater near the wireless AP wasvery weak, ranging from –74 to –100 dBm, which
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

was the measurement limit.
The measured radio-wave strength for downlink
near the wireless APs and wireless repeaters was –31
and –71 dBm, as shown in Fig. 2. The results indicatethat the transmission loss between a wireless AP and
wireless repeater was about 40 dB. The theoreticalvalue of distance attenuation calculated from the distance of 24.6 m between the wireless AP and wirelessrepeater is 28 dB ( 20log (24.6 m)); therefore,
attenuation of 12 dB (about 1/20) was caused by thewindow glass in the building. The window was a typeof double glazing and the inner surfaces of the glass
were covered with a low-emissivity metal coating to94

Practical Field Information about Telecommunication Technologies

Image disturbance occurred where
the display (host unit A) and a PC
(transmission unit B) in the adjacentconference room were close to eachother through the wall.

Host
unit
B

Transmission
unit B

Host
unit
A

Transmission
unit A
Measurement location

Conference room B

Conference room A*Furniture is omitted in this figure.

Fig. 3. Equipment configuration.

reduce radiant heat from the sun.
2.3 Cause of problem
From our investigation, it was presumed that thelarge attenuation caused by the low-emissivity glass
interrupted the video streams from the security cameras. The radio waves for uplink from the wirelessrepeater were significantly weakened, so the communication between the wireless APs and wirelessrepeaters probably become unstable.
2.4 Solution
In this case, by moving the wireless APs away fromthe low-emissivity window glass, which prevents thetransmission of radio waves, it is possible to increase
the amount of radio-wave transmission through thewall other than the glass. When we moved the wireless APs, the received signal strength between thewireless APs and each wireless repeater increased
and improved radio-wave throughput, eliminating
video interruptions.
3. Problem case withvideo-image-transferring system
3.1 E
 quipment configuration and problem situation
A problem with a security-camera system using awireless LAN was reported from our customer. The
equipment configuration is shown in Fig. 3. A wireless display adapter (WDA), which is used to display
NTT Technical Review

Vol. 19 No. 12 Dec. 2021

the screen of a PC on a large display via a wirelessLAN, has two units. Its host unit is connected to thedisplay and its transmission unit is connected to the
PC. At the customer’s premises, many WDAs were
installed in conference rooms and meeting spaces.
The WDAs used the 5-GHz band of wireless LAN,and channel bonding was used to ensure the 40-Mbit/s
bandwidth, which is required for video transmission.
The 5-GHz-band wireless LAN has a dynamic frequency selection (DFS) function that automaticallyswitches the channel to be used if radar waves, etc.
are received. In this equipment configuration, W52, afrequency band that is not used by radar waves, wasused to avoid communication interruption due tochannel switching by DFS. This band was also divided into two channel groups (36/40ch and 44/48ch) to
avoid inter-channel interference. Despite theseattempts, the video images displayed on the large
display were disrupted in certain areas.
3.2 Investigation of cause(1) Investigation of radio-wave environment
We used a spectrum analyzer to measure the radiowave environment in the 5-GHz band, and no radio
waves other than those of WDAs were measured.
(2)	Investigation of transmission-waveform spectrum
We measured the transmission-waveform spectrum
near host unit A when video disturbance occurred(Fig. 4).
The measurement results indicate that transmission95

Practical Field Information about Telecommunication Technologies

36.40ch44.48chTransmission
unit A

Transmission
unit A

Transmission
unit B

Waveform disturbanceoccurs

10 dB(a) Only transmission unit
A is communicating(b) Immediately after
communication withtransmission unit B starts(c) 2 to 3 seconds after
communication starts

Fig. 4. Measurement of transmission-waveform spectrum using spectrum analyzer.

units A and B used different channel groups and thatat the measurement location shown in Fig. 3, thereceived signal strength from transmission unit B was
about 10 dB higher than that from transmission unit
A. They also indicate that two to three seconds after
transmission unit B started communication, the transmission waveform from transmission unit A was
disrupted (Fig. 4(c)) and, simultaneously, the video
image was disrupted.
3.3 Cause of problem
It was presumed that this disturbance was causedby the radio wave from transmission unit B, which
was about 10 dB higher than that from transmission
unit A, reached host unit A, although conference
rooms A and B were separated by a wall. The radio
wave from transmission unit B reached the host unit
A; therefore, the communication between transmission unit A and host unit A was affected. This disturbance is presumed due to adjacent-channel interference, which is caused by the arrival of strong radio
waves on the adjacent channel.
3.4 Solution
The basic solution against problems like this case isto improve the radio-wave environment. To improve
such an environment, it is necessary to ensure thedistance between the wireless-LAN devices, to prevent radio-wave interference between the wirelessLAN devices by installing sheets of radio-wave
shielding, to shorten the distance between the host

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

and transmission units or not to obstruct the line ofsight between the host and transmission units.
However, it was difficult to implement the abovementioned solution in this case; therefore, we set up
the same communication channel for units A and forunits B and controlled the transmission timing temporally. It became possible to stabilize the communication in the wireless section and eliminate video
disturbance. It should be noted that when the samechannel is used for multiple wireless devices, thetransmission rate will be divided in proportion to thenumber of devices; therefore, it is necessary to verify
operation before installation of the devices to ensurethat the bandwidth required for video transmission
can be secured.
4. Conclusion
Recent problem cases with video transmission systems using wireless LANs and their solutions wereintroduced in this article. TASC develops tools thatcan be used in the field to analyze and identify thecause of problems in wireless-LAN and wirelessbased systems.
The EMC Engineering group in TASC is continuously engaging in technical support to the field
regarding problems caused by lightning, malfunction
due to electromagnetic phenomena, interference ofradio waves, induction from power line, and so on, toensure reliability of telecommunication services.

96

External Awards
Electronics Society Award
Winners: Masahiro Nada, Toshihide Yoshimatsu, NTT Device Innovation Center; Hideaki Matsuzaki, NTT Device Technology Laboratories
Date: March 31, 2021
Organization: The Institute of Electronics, Information and Communication Engineers (IEICE) Electronics Society
For their pioneering research and development of high-speed avalanche photodiodes with a vertical-illumination structure for opticalfiber communications.
FIT Encouragement Award
Winner: Yuya Omori, NTT Device Innovation Center
Date: August 27, 2021
Organization: Information Processing Society of Japan (IPSJ)
For “Complexity Reduction Based on Equivalence and Continuityof Convolutional Layers in CNN Inference.”
Published as: Y. Omori, D. Kobayashi, S. Yoshida, S. Hatta, H.
Uzawa, K. Nakamura, and K. Sano, “Complexity Reduction Basedon Equivalence and Continuity of Convolutional Layers in CNN
Inference,” Proc. of 20th Forum on Information Technology(FIT2021), Vol. 1, pp. 179–180, Aug. 2021.
Best Paper Award (Research track)
Winners: Yukako Iimura and Shinobu Saito, NTT Computer andData Science Laboratories
Date: September 8, 2021
Organization: IPSJ Special Interest Group on Software Engineering(SIGSE)For “Practical Report on Software Development Approach for
Promoting Workstyle Flexibility.”
Published as: Y. Iimura and S. Saito, “Practical Report on Software
Development Approach for Promoting Workstyle Flexibility,” Proc.
of IPSJ/SIGSE Software Engineering Symposium 2021, pp. 14–22,
Sept. 2021.
Distinguished Contributions Award
Winner: Doohwan Lee, NTT Network Innovation Laboratories
Date: September 15, 2021
Organization: IEICE Communications Society
For his contributions as a peer reviewer for papers submitted to the
IEICE Communications Society.
Young Scientist Presentation Award
Winner: Koji Sakai, NTT Basic Research Laboratories

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

Date: September 21, 2021
Organization: The Japan Society of Applied Physics (JSAP)
For “Development of 3D-cultured Neuronal Network in Graphene-based Self-folding Electrode Array.”Published as: K. Sakai, T. Teshima, T. Goto, H. Nakashima, and M.
Yamaguchi, “Development of 3D-cultured Neuronal Network in
Graphene-based Self-folding Electrode Array,” The 68th JSAP
Spring Meeting 2021, Mar. 2021.
Young Scientist Presentation Award
Winner: Ai Ikeda, NTT Basic Research Laboratories
Date: September 21, 2021
Organization: JSAP
For “Novel Superconducting (CaCuO2)n/(Ca2Fe2O5)m Superlattices Prepared by MBE.”
Published as: A. Ikeda, Y. Krockenberger, Y. Taniyasu, and H. Yamamoto, “Novel Superconducting (CaCuO2)n/(Ca2Fe2O5)m Superlattices Prepared by MBE,” The 68th JSAP Spring Meeting 2021, Mar.
2021.
Young Scientist Presentation Award
Winner: Takahiro Kashiwazaki, NTT Device Technology Laboratories
Date: September 21, 2021
Organization: JSAP
For “Terahertz-order Broadband Measurement of Squeezed Light
by Optical Parametric Amplification for Ultra-fast Optical Quantum
Computing.”
Published as: T. Kashiwazaki, N. Takanashi, A. Inoue, T. Kazama,
K. Enbutsu, R. Kasahara, T. Umeki, and A. Furusawa, “Terahertzorder Broadband Measurement of Squeezed Light by Optical Parametric Amplification for Ultra-fast Optical Quantum Computing,”
The 68th JSAP Spring Meeting 2021, Mar. 2021.
Young Scientist Presentation Award
Winner: Takuma Tsurugaya, NTT Device Technology Laboratories
Date: September 21, 2021
Organization: JSAP
For “Photonic Reservoir Computing Using Low-power-consumption SOA.”
Published as: T. Tsurugaya, T. Hiraki, M. Nakajima, T. Aihara, N.-P.
Diamantopoulos, T. Fujii, T. Segawa, and S. Matsuo, “Photonic Reservoir Computing Using Low-power-consumption SOA,” The 68th
JSAP Spring Meeting 2021, Mar. 2021.

97

Papers Published in Technical Journals andConference Proceedings
Divide-and-conquer Verification Method for Noisy Intermediate-scale Quantum Computation
Y. Takeuchi, Y. Takahashi, T. Morimae, and S. TaniarXiv:2109.14928, Oct. 2021.
Several noisy intermediate-scale quantum computations can be
regarded as logarithmic-depth quantum circuits on a sparse quantum
computing chip, where two-qubit gates can be directly applied on
only some pairs of qubits. In this paper, we propose a method toefficiently verify such noisy intermediate-scale quantum computation. To this end, we first characterize small-scale quantum operations with respect to the diamond norm. Then by using these charac-

NTT Technical Review

Vol. 19 No. 12 Dec. 2021

terized quantum operations, we estimate the fidelity ψt | ρ̂ out|ψtbetween an actual n-qubit output state ρ̂out obtained from the noisy
intermediate-scale quantum computation and the ideal output state(i.e., the target state) |ψt . Although the direct fidelity estimation
method requires O(2n) copies of ρ̂out on average, our method requires
only O(D3 212D) copies even in the worst case, where D is the denseness of |ψt . For logarithmic-depth quantum circuits on a sparse chip,
D is at most O(log n), and thus O(D3 212D) is a polynomial in n.

98