NTT Technical Review

July 2022 Vol. 20 No. 7

Front-line Researchers

Ryuichiro Higashinaka, Visiting Senior Distinguished Researcher, NTT Human Informatics
Laboratories/NTT Communication Science Laboratories

Rising ResearchersTakuhiro Kaneko, Distinguished Researcher, NTT Communication Science Laboratories

Feature Articles: Optical and Wireless Transmission Technologies for IOWN/6G


Research and Development for Pioneering a New Communications Paradigm with Wide-area
Coverage

Future Development of Digital Coherent Optical Transmission Technology

Research and Development of Scalable Optical Transport Technologies

R&D Activities of Core Wireless Technologies toward 6G Radio AccessFeature Articles: ICT Platform for Connected Vehicles Created by NTT and ToyotaOverview of Technical Development and Verification in the Connected-vehicle Field



Activities and Results of Field Trials—Reference Architecture for a Connected-vehicle
Platform

Activities and Results of Field Trials—Network Edge Computing Platform

Real-time Spatiotemporal Data-management Technology (Axispot™)

Selective Vehicle-data-collection AlgorithmVertically Distributed Computing TechnologyLane-specific Traffic-jam-detection TechnologyTechnology for Calculating Suddenness Index for Aggregated Values

Regular Articles
Unsupervised Depth and Bokeh Learning from Natural Images Using Aperture Rendering
Generative Adversarial Networks

Global Standardization Activities


Report of the 9th ITU-T TSAG (Telecommunication Standardization Advisory Group) Meeting

External Awards

Front-line Researchers

Put Your Curiosity First and Try
Anything—Researchers Should Do
What They Want to Do; Whatever
It Takes
Ryuichiro Higashinaka
Visiting Senior Distinguished
Researcher, NTT Human Informatics
Laboratories/NTT Communication
Science Laboratories
Abstract
Dialogue systems have made great progress over the last few
years due to the application of deep learning and the spread oftechnologies that enable people to interact with voice agents on
smartphones, personal robots, and other devices. Ryuichiro
Higashinaka, a visiting senior distinguished researcher of NTT
Human Informatics Laboratories/NTT Communication Science
Laboratories and professor of the Graduate School of Informatics, Nagoya University, is aiming to createa dialogue system that allows humans and computers to understand each other and intelligently collaborate by clarifying the principles of natural-dialogue interaction between them. We interviewed him about
the progress of his research and his attitude as a researcher.
Keywords: dialogue system, common ground, AI

Create a world in which humans and computers
can understand each other by pursuing common
ground between them
—It has been about three and a half years since thelast interview. Can you tell us about the research you
are currently working on?
In my last interview, in 2018, I focused on thequestion-answering system used in the “Shabette
Concier” voice-agent service and dialogue systems.
NTT Technical Review

Vol. 20 No. 7 July 2022

Since then, my research has further advanced andexpanded into other fields. Natural language processing has rapidly improved through deep learning, andits possible applications have multiplied considerably. For example, the advent of BERT (Bidirectional
Encoder Representations from Transformers), a natural-language-processing model, has caused a paradigm shift of sorts and dramatically improved theperformance of natural language processing. Over thelast few years, I have been applying natural-languageprocessing techniques based on deep learning to1

Front-line Researchers

Narikiri AI Amadeus Kurisu

A time machine is notpossible. But it cannot
be said that it is
impossible.

May I ask why you thinkso?

The reason for thinking
so is simple: sciencehasn’t advanced enoughto interfere with the past.

What makes you think it
is not impossible?

We cannot change thepast. But we can change
the future. That’s why.

Fig. 1. Narikiri AI Amadeus Kurisu chatbot.

artificial intelligence (AI) systems.
In 2019, as part of the “Todai Robot Project—Can
a robot get into the University of Tokyo?” launchedby the National Institute of Informatics, we developed an AI robot—using a technology similar to
BERT and the latest technology at the time—that
takes the written-English subject of the National Center Test for University Admissions and scored a highmark of 185 points (deviation score of 64.1). NTT
released a press release about this achievement [1].
We also recently created the chatbot called “Narikiri AI Amadeus Kurisu” (Fig. 1), which has greatly
improved the accuracy of responses in dialogues
compared with the previously introduced “Ayase AI”
chatbot. Narikiri AI (narikiri means impersonation)
uses a unique method of creating a character of achatbot based on training data obtained from users
and fans who describe how a certain character speaks,
thinks, etc. We received 45,000 samples for the training data of Narikiri AI Amadeus Kurisu, and by using
deep learning, we were able to create a character withhigh accuracy in response generation. Then, in July
2021, we created an opportunity for fans to interact
with the chatbot by using the direct-message featureof Twitter. Although limited to three days, the event
was very well received; in fact, some participants had
hundreds of conversations with the chatbot, and theevent received a high rating of 4.59 out of 5 in a survey. My paper on Narikiri AI was accepted by a top
conference on language processing [2], so I’m proudthat this work has been recognized as a significant
academic achievement. However, some of the chatbot’s responses were not suitable for the character, so
we are currently analyzing user feedback and improving the chatbot.
NTT Technical Review

Vol. 20 No. 7 July 2022

—The conversation between computers and humans
seems to be evolving. Have we reached a level atwhich computers and humans can mutually understand each other?
I’m afraid we haven’t reached that level yet. To
enable computers and humans to understand eachother and communicate, we need to develop a systemthat can build common ground between them. Common ground is the content that is understood by theparticipants in conversation with each other. I believethat once common ground between computers andhumans can be established, mutual understanding canbe increased and communication between them will
evolve rapidly, making collaboration between dialogue systems and humans possible. Although common ground is a concept that has been discussed in
textbooks for more than 20 years, it has not been
engineered or implemented in a system. However, I
decided to pursue this theme because I believe that if
common ground cannot be established, dialogue systems won’t progress, and the next generation of such
systems won’t be viable.
Common ground is invisible because it is in themind, which made research on common ground difficult. To visualize the process of establishing common ground, we set up a task called CommonLayout
in which each of the two participants views their own
figure-arrangement screen and cooperatively determines the placement of figures while interacting witheach other via text chat and collected large-scaledialogue data between people executing this task
(Fig. 2) [3]. We quantified common ground by constantly recording the placements of the figures andmeasuring the distance between each figure placement
2Front-line Researchers

Worker B’s display

Worker A’s display

Start EndTime

Start End

07:49

Layoutinterface

Time

07:49

== Worker A started ==== Worker B started ==A: Nice to meet you.
B: Nice to meet you too.
B: What shall we make?
A: Do you have any
preferences?
B: I don’t have any idea.
A: I’m thinking too.
B: The circle in the center
seems to be a nose.
A: Then let’s make a face.

== Worker A started ==== Worker B started ==A: Nice to meet you.
B: Nice to meet you too.
B: What shall we make?
A: Do you have any
preferences?
B: I don’t have any idea.
A: I’m thinking too.
B: The circle in the center
seems to be a nose.
A: Then let’s make a face.
Chatinterface

Message

Message

Send

Send

Fig. 2. CommonLayout task.

(the sum of vector differences defined between any
two figures) during the task. In other words, we madeit possible to visualize the degree to which common
ground was being established at each point during the
dialogue. By analyzing these data, we found that theprocess of establishing common ground can be categorized into several clusters. This research was presented at the annual conference of the Association for
Natural Language Processing and received an award
for excellence in March 2021. In the process of analyzing these data, we also learned that so-called
naming plays a key role in building common ground.
In particular, humans seem to build common groundby naming figures and matching their perceptionsof those figures with those of others [4].
In addition to the above-mentioned work, to clarify
the effect of modality (channels of information suchas text, audio, and video) and social relationships on
the process of constructing common ground, I’m
extending my research on the CommonLayout task
conducted via text chat to construct and analyze a
corpus (a large-scale collection of natural language
and other data) composed of audio and video. We
found that it is easier to establish common groundwhen audio and video are used rather than when only
audio is used. We also found that it was easier toestablish common ground when the participants wereacquainted rather than when they met for the firsttime. The reader might think these findings are obviNTT Technical Review

Vol. 20 No. 7 July 2022

ous. However, the process of building common
ground differs according to the conditions, and I
believe that these differences will lead to important
findings. By analyzing the effects of modality andsocial relationships on the establishment of common
ground, we hope to create a dialogue robot that cansmoothly establish common ground between computers and humans. I’d also like to establish common
ground from various angles, and we are working on
the task of naming the shapes of the pieces in a tangram (a puzzle composed of seven pieces dividing asquare) while the participants are having a dialogueduring the task.
I want to accelerate my research and expand it
by combining different disciplines—How much research on common ground has been
done?
Although researchers in the philosophy of language
have examined how humans establish common
ground, engineering approaches are few and far
between; in fact, only a few researchers around theworld are focusing on this theme. I’m keeping myeyes open and reviewing the literature, but there are
currently not many relevant reported studies. Therefore, to create a future in which AI and humans cancollaborate, I’m accelerating research on this theme3Front-line Researcherswith determination. I’m also promoting joint
research. For example, the above-mentioned tangram-naming task is pursued in a joint research with
Shizuoka University, and I’m collaborating with Professor Yugo Takeuchi, an expert in cognitive science.
I’m also working with Professor Yasuhiro Minamiof the University of Electro-Communications on
researching the naming in building common ground I
mentioned earlier. In addition to studying speech processing, Professor Minami is also studying language
development in young children. Through our collaboration, we are thus able to pursue engineering andlanguage development in combination. I’m also collaborating with Professor Kazunori Takashio, an
expert in social robotics at Keio University, on theeffect of the modality and social relationships inbuilding common ground. Through these collaborations, I’m advancing my research on common groundby combining various disciplines.
—You seem to be engaged in academically andsocially significant activities, including active joint
research.
I think there is academic significance in doing what
has been avoided so far. When researching dialoguesystems, it is relatively easy to analyze the verbal
exchanges that are visible as text and manifested as
phenomena. However, it is difficult to analyze andevaluate what the other party is thinking when they
say those words. The research on the CommonLayout
task is to visualize those thoughts and eliminate thatdifficulty. My research is progressing, and I’m confident that the day is near when we will be able to build
a dialogue system that enables humans to work
together with AI when thinking about and deciding
on product placement and other issues.
I think that research on dialogue systems is currently at a standstill. That is to say, deep learning is
still limited in what it can do. We can have a conversation with an AI system, but it is still only for a shorttime. If the system forgets yesterday’s communication, intellectual collaboration between humans andAI will not be established. I want to develop AI
capable of long-term communication to be a partner
for us humans so that it can accompany us throughour lifetime.
I believe that by pursuing common ground, we canmake human society better 20 years or so from now.
As I say in my book “Chatting Skills of AI” (KADOKAWA), I hope to create a world in which humans
and computers can build a two-way relationship and
NTT Technical Review

Vol. 20 No. 7 July 2022

enhance mutual understanding.
Be willing to go beyond your area of expertise—What qualities are required of researchers today?
Compared with 2001, when I joined NTT, it is now
possible to conduct many experiments relatively
inexpensively through crowdsourcing and othermeans. Therefore, I think that researchers are in a
fortunate environment. On the contrary, companies
with vast financial and human resources, such as
GAFA (Google, Apple, Facebook, and Amazon), areinvesting in research and development to rapidly
bring new products and services to the world. The
improved research environment has increased thetempo of research and development and intensifiedcompetition.
To survive in these times, researchers are required
to constantly take on new challenges. It is also necessary to keep in mind our impact on—and contribution
to—society as we pursue our research. To meet theserequirements, I believe it is important to be willing to
go beyond our own area of expertise. Many problems
can only be solved by incorporating knowledge fromother research areas, and we are no longer in an era of
sticking to only one area of specialization such as
language processing; rather, we are required to havea multifaceted and broad perspective. For example,
when building a dialogue system, we must keep inmind the development of laws, etc. and consider society, users, etc. in a composite manner; otherwise,even if we develop a system born of outstanding
research, we will not be able to roll it out.
In fact, dialogue systems were not the theme thatinterested me when I joined the company. I wanted topursue machine translation because I loved words,
but the theme assigned to me was dialogue, which I
found so interesting that I ended up doing it for 20
years. Since that experience, I’ve been working withthe spirit of “let’s try everything.” Basically, I neverrefuse a request.
—What would you like to say to young researchers?
I believe that researchers should do what they want
to do; whatever it takes. I also believe one of theimportant roles of a senior distinguished researcher is
to set an example and show that one needs to do what
one wants to do. It is important to be interested in newthings, have a multifaceted and broad perspective
beyond one’s area of expertise, and not be afraid to4

Front-line Researchers

try anything. Why not start by putting your curiosity
first and trying anything? When we are young, wesometimes think that one mistake means it’s all over.
Immediately after I joined NTT, I was worried that if
I did not get a paper accepted by a top conference
within about three years, I’d be on my way out. Fortunately, I managed to get a paper accepted with thehelp of my mentor; even so, I was still afraid of
failure and thought I might be branded an unfit
researcher.
However, I don’t want young researchers to worry
about failing. That’s because they can use their failures to move on to the next stage of their development. To put it another way, if you think of years ofservice as the denominator and failure as the numerator, the denominator is small when you are young, so
the damage of failure may be severe, but as thedenominator becomes larger, that severity is dilutedby your longer service, so the way you perceive failure changes.
As a member of a corporate organization, you will
sometimes be obliged to follow company policy andinstructions from supervisors. However, it is important not to abandon what you want to do to focus on
your obligation but to connect with peers as well aswith researchers who share similar interests. In mycase, the connections that I’ve made with the academic community have been a great help. It is a goodidea to build relationships not only with people inside
your company but also with people outside the company from a young age.
References[1]

Press release issued by NTT, “AI Achieved a Score of 185 on the
English Written Exam of the National Center Test for University
Admissions in 2019,” Nov. 18, 2019.
https://group.ntt/en/newsrelease/2019/11/18/191118a.html

NTT Technical Review

Vol. 20 No. 7 July 2022

[2]

[3]

[4]

K. Mitsuda, R. Higashinaka, T. Li, and S. Yoshida, “Investigating
Person-specific Errors in Chat-oriented Dialogue Systems,” Proc. of
the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 464–469, Dublin, Ireland, May
2022.
K. Mitsuda, R. Higashinaka, Y. Oga, and S. Yoshida, “Dialogue Collection for Recording the Process of Building Common Ground in a
Collaborative Task,” Proc. of the 13th Language Resources and
Evaluation Conference, pp. 5749–5758, Marseille, France, June 2022.
Y. Saito, K. Mitsuda, R. Higashinaka, and Y. Minami, “Analysis of
Naming in the Process of Building Common Ground in Dialogue,”
Proc. of the 28th Annual Meeting of the Association for Natural Language Processing, pp. 38–42, Mar. 2022 (in Japanese).

Interviewee profile
Ryuichiro Higashinaka received a B.A. in environmental information, Master of Media andGovernance, and Ph.D. from Keio University,
Kanagawa, in 1999, 2001, and 2008. He joined
NTT in 2001. Since 2020, he has been a professor
at the Graduate School of Informatics, Nagoya
University and visiting senior distinguishedresearcher at NTT. His research interests include
building question-answering systems and spoken
dialogue systems. From November 2004 to
March 2006, he was a visiting researcher at the
University of Sheffield in the UK. He received theMaejima Hisoka Award from the Tsushinbunka
Association in 2014 and the Prize for Science andTechnology of the Commendation for Scienceand Technology by the Minister of Education,
Culture, Sports, Science and Technology in 2016.
He is a member of the Institute of Electronics,
Information and Communication Engineers, theJapanese Society for Artificial Intelligence, theInformation Processing Society of Japan, and the
Association for Natural Language Processing.

5

Rising Researchers

Learning 3D Information from 2D
Images Using Aperture Rendering
Generative Adversarial Networks
toward Developing a Computer that
“Understands the 3D World”
Takuhiro Kaneko
Distinguished Researcher, NTT
Communication Science Laboratories
Abstract
When people look at photos, they can estimate three-dimensional (3D) information, such as depth, from their experience andknowledge, but computers have difficulty in doing so because
they cannot have such experience and knowledge. We spoke to
Takuhiro Kaneko, a distinguished researcher who developed anovel deep learning model that can learn 3D information fromstandard 2D images such as those on the web.
Keywords: generative adversarial networks, unsupervised learning, depth and bokeh

Learning of depth and bokeh effects fromnatural images with aperture rendering
generative adversarial networks—What is research on “learning of depth and bokeh
effects from natural images” about?
It is difficult to record the three-dimensional (3D)world in which we live in as it is. For this reason, it iscommon to record and store two-dimensional (2D)
images such as photographs instead of 3D information.
When people look at photos, they can estimate 3D
information, such as depth, from their previous expeNTT Technical Review

Vol. 20 No. 7 July 2022

rience and knowledge. However, computers havedifficulty in doing so because they do not have such
experience or knowledge. In the future, when we
think about scenarios where robots support our lives,
it will become essential for computers to understandthe 3D world. The easiest way for computers to learn
is to provide a large number of pairs of 2D images
and 3D information as training data. Learning would
be easy because they know the correct answer. However, this method requires special devices such as
depth sensors and stereo cameras and is costly.
For this reason, in this research, we created a deeplearning model that can learn 3D information fromstandard 2D images such as those taken with ordinary
6

Rising Researchers

All-in-focus
image
Randomvariable

Light
field

Aperture

Bokeh
image

Image
generator

Mechanism simulating the camera aperture

Depth
generator

Wide aperture
Depth
map

Bokeh image

Ray oflight
Narrow apertureAll-in-focus image

Ray oflight

Fig. 1. Process flow in the AR-GAN generator.

cameras and those on the web. When we look at thephotos, we see that the focus is usually on the object,
and the background is often blurry. Using these defocus cues as clues, we carry out learning of 3D information, especially depth and bokeh effects. In otherwords, if the problem of projecting 3D information
onto 2D images through a camera is the forward
problem, for this research, the goal is to solve theinverse problem. This problem is challenging because
it is the so-called “ill-posed problem” of estimating
3D information from only 2D images that lack various information.
—Specifically, through what mechanisms are you
carrying out learning?
This research is based on a technology called agenerative adversarial network (GAN). GAN is a socalled “unsupervised learning model,” which does
not require pre-determined correct answers, and consists of two neural networks; namely, a generator anddiscriminator. The generator generates a “fakeimage” from a given random variable. On the other
hand, the discriminator distinguishes two types ofimages, “real images” and the “fake images” generated by the generator. Because they work adversarially, wherein the generator tries to trick the discriminator while the discriminator tries to identify the fakeimages precisely, learning can be carried out while
competing, and as a result, the generator can produce
realistic images.
NTT Technical Review

Vol. 20 No. 7 July 2022

Since GAN is a specialized technology for producing 2D images, it has no connection with the 3D
world. We, therefore, proposed the “aperture rendering GAN (AR-GAN),” which incorporates the opticalproperties of the camera into the GAN. “Aperture”
refers to the aperture of the camera. By incorporating
optical constraints due to the aperture of the camera
when projecting the 3D world into 2D images, thegenerator can learn by associating 2D images, depth
maps, and bokeh effects.
Figure 1 shows the process flow in the AR-GAN
generator. The “image generator” is also included in
the GAN and generates an all-in-focus image when
given a random variable. The depth generator generates the depth map that is paired with the image, andis unique to AR-GAN. The system then uses thesepairs of generated data to enable a mechanism simulating the camera aperture. The light field consists of
25 images, which represent what the image looks likein the aperture of the camera. In the center, the image
created by light coming in through the center of theaperture is displayed. When the object is offset fromthe center to the right, an image of the object viewed
from the right part of the aperture is displayed; andwhen the object is offset to the top of the aperture, animage of the object viewed from the top part of theaperture is displayed.
It may be easier to imagine the mechanism by moving your face up, down, left, and right while looking
at the object in front of your eyes. Objects with focus
are clearly visible because they are in the same
7

Rising Researchers

position and do not move when you move your face,but objects that are farther away move more when
you move your face. Therefore, summation of theseimages produces images with bokeh effects. This is
how the mechanism enables synthesizing a bokeh
image on the basis of the predicted depth map andall-in-focus image using a camera with an opticalconstraint on the light field.
—How is the progress of the research and what are
the challenges going forward?
Currently, we are able to generate flower images,
bird images, and human face images. Learning cantake days, but once it is done, the system can generate
in a few seconds bokeh images that cannot be distinguished from the real images. At present, I think that
if we narrow down the types of objects, we would be
able to generate reasonable images.
As for the challenges, one is that depending on thetype of object, some are easier to learn while others
are not as amenable to learning. For example, forhuman face images, the shape and position of theparts are fixed to some degree, making it easy to learn
them. On the other hand, images with many variations, such as those of animals taken from different
viewpoints and distances, are difficult to learn.
Also, the larger the image size, the more details
need to be synthesized, and the more processing time
it takes, making learning more difficult. I think there
will be various issues that will emerge as we expand
the scope of applications in the future. As expected,
the difficulties inherent with ill-posed problems
remain.

NTT Technical Review

Vol. 20 No. 7 July 2022

Developing a computer that understands the
3D world—What will this technology enable in the future?
Our mission as researchers is to “develop computers that are highly compatible with humans.” For thisreason, it is essential to understand the 3D world; but
the cost of data collection is likely to be a barrier toapplying the technology to a wider range of fields. In
that regard, I think this research is beneficial because
it is excellent in terms of data collection. In the future,
we will be able to create robots that can move freely
around the 3D world, build 3D worlds without dissonance in virtual space, and develop tools to create
3D objects in virtual space.
Another advantage is that optimized models can bebuilt as long as we can collect 2D images. For example, if you gather photos taken by a well-known photographer, you can build a model that learns thebokeh effects unique to that photographer. Currently,
communication using images such as through social
media has become very common. If we can easily add
the bokeh effects, it may become easier to create
more attractive photos. I think this is particularly useful in the three areas of robotics, content generation,and entertainment.
—What are the future prospects and initiatives oncollaborations with other fields?
Since it is basic research, it is difficult to set specific targets for practical use at this time; but we plan
to continue to improve performance by increasing
accuracy and resolution. We have focused on thecamera aperture, but it is interesting to note that adding physical constraints enables creating a more reliable computer.
One of the key technical areas of the Innovative
Optical and Wireless Network (IOWN) initiative is“Digital Twin Computing,” where computations are
performed using digital twins of various industries,
things, and humans. In order to merge the real world
with the virtual world, it is necessary for computers
to properly understand the real world, so I believe that
this technology can also contribute to these areas.
I feel that the importance of integrating media generation technology with various fields is increasing as
the technology matures. Currently, we are conducting
research that combines computer science, such as
computer vision and machine learning, with physics,
such as optics. Going forward, we will continue to8

Rising Researchers

focus our efforts on collaborations with people fromother fields, such as computer graphics for image
creation and photonics for photography, as well as oncross-discipline implementation.
—Could you give a message to young researchersand future business partners?
NTT laboratories are conducting extensive researchranging from basic research to applications, and inparticular, our laboratory, NTT Communication Science Laboratories, is conducting research on how to
improve communication between humans andbetween humans and machines. In the past, we haveoften been limited to basic research, but recently, thedistance between basic research and applied researchhas been narrowing, and I feel that the opportunities
for engaging in research while considering real-world
problems are increasing. We are also seeing more andmore cases of the technology presented at international conferences being embedded in applications
and deployed as a service on the web. I think it isinteresting to see the output, for example, by creating
a solution that actually converts voice, rather thanstopping at tinkering with formulas.
There are limitations to doing research individually.
Our laboratory is also actively collaborating with
universities and accepting interns, so I hope that we
can continue to actively collaborate particularly withstudents and young researchers who want to create
something or change something.
As for business partners, since we have been able to

NTT Technical Review

Vol. 20 No. 7 July 2022

create interesting ideas and technologies from theresearch side, we need their help in linking theseideas and technologies to services. On the other hand,
we can get ideas for research by receiving feedback
from people in the service field who have profoundknowledge of real problems. So, going forward, we
hope to continue to actively collaborate also withbusiness partners.

Interviewee profile
Takuhiro Kaneko received his M.E. degreefrom the University of Tokyo in 2014. He joined
NTT the same year as a member of NTT
Communication Science Laboratories. In 2020,
he completed his Ph.D. at the University ofTokyo. He has been a distinguished researcher at
NTT Communication Science Laboratories since2020. He is engaged in research on computer
vision, signal processing, machine learning, anddeep learning, particularly dealing with image
synthesis, speech synthesis, and voice conversion. He received the Hatakeyama Award fromthe Japan Society of Mechanical Engineers, the
ICPR Best Student Paper Award in the 21stInternational Conference on Pattern Recognition(ICPR 2012), and the Dean’s Award for Best
Doctoral Thesis from the University of Tokyo.
For details: https://www.kecl.ntt.co.jp/people/
kaneko.takuhiro/

9Feature Articles: Optical and Wireless Transmission
Technologies for IOWN/6G

Research and Development for
Pioneering a New Communications
Paradigm with Wide-area Coverage
Shigeru Iwashina, Katsuhiro Shimano, Koichi Takasugi,
Kazunori Akabane, and Takashi Saida
Abstract

Research and development at NTT Network Innovation Laboratories aims to establish elemental technologies for next-generation communication networks envisioned under NTT’s Innovative Optical andWireless Network (IOWN) and the 6th-generation mobile communication system (6G). These technologies, which include advanced and high-capacity backbone optical transmission networks and extended
coverage of wireless communications, are being developed to support dramatic changes in the information society such as the expansion of remote work due to the COVID-19 pandemic. This article introduces optical/wireless transmission technologies and systemization technologies currently being
researched and developed at NTT Network Innovation Laboratories.
Keywords: optical transmission technology, wireless transmission technology, IOWN/6G

1. Introduction
Social activities that enable remote participation
have become widespread as reflected by the adoptionof remote work as a countermeasure against the
COVID-19 pandemic. To support these activities,
there are growing expectations for the developmentof next-generation communication networks envisioned under NTT’s Innovative Optical and Wireless
Network (IOWN) and the 6th-generation mobile
communication system (6G) featuring, for example,
advanced and high-capacity backbone optical transmission networks and extended coverage of wirelesscommunications. Research and development (R&D)
at NTT Network Innovation Laboratories aims toestablish communication technologies that can
exploit the physical waves of diverse frequency bands
including light, radio, and sound waves and enablelong-distance, high-speed, and high-capacity transmission in a wide range of areas including communication media such as optical fiber, water, and air.
Researchers at NTT Network Innovation LaboratoNTT Technical Review

Vol. 20 No. 7 July 2022

ries understand that their mission is to pioneer a newcommunications paradigm by achieving the above
research targets using scientific knowledge based on
physics and mathematics such as that related to electromagnetic wave propagation and light propagation,
digital signal processing, and media processing.
As shown in Fig. 1, the research scope of NTT Network Innovation Laboratories covers a wide range ofareas. Researchers usually dealt with optical fiber andradio wave propagation, but today, the research scope
has broadened as far as low-frequency bands, such as
sound waves, and research targets have extended
beyond air and enclosed spaces, such as optical fiber,
to include all types of spaces and media from water toouter space as communication media. Research inunderwater acoustic communication targeting water,
for example, aims to achieve information transfers
beyond the Mbit/s class capable of video transmission. To achieve extreme-high-capacity wirelesstransmission toward 6G [1], NTT Network Innovation Laboratories is also researching and developing
new areas such as orbital angular momentum (OAM)10Feature Articles

Computing network platform technology
Natural transmission technology

Medium: Pursue principles
on the physical layer: Physical application,systemizing(Spatial
multiplexing)

Enclosed
space

Scalable opticaltransport

Wave adaptation& control(Single core)
Quantum
cryptography

FLEX
digital-coherent
Extreme L1
network(Boundary region)
Free-space optics
Satellite sensing(Air,
outer space)Open(On land)space

Underwater acoustic comm.

(Underwater)

100G digitalcoherent TX

100G digitalcoherent RX

Terabit wirelessIntelligent space formation& comprehension technology(Sound)Low(Radio)(Light)Frequency band

High

Fig. 1. Research scope of NTT Network Innovation Laboratories toward IOWN/6G.

multiplexing transmission wireless communications,
free space optics (FSO) assuming air and outer space
as media, and quantum cryptography communications for achieving the ultimate in security.
At NTT Network Innovation Laboratories, we arealso engaged in specific R&D toward the development of the All-Photonics Network (APN) [2], which
is one of the three main pillars of NTT’s IOWN initiative. This includes the R&D of elemental technologies for achieving a next-generation optical communication network that can easily provide high-capacity optical communication paths to a variety of customers as expected of the APN. Specific projects
include the development of a low-power digital signal
processor (DSP) for achieving 400-Gbit/s-class signal transmission through digital-coherent opticaltransmission technology, demonstration of bulk
extension of the optical amplification bandwidth
through wideband optical parametric amplifiers, andthe development of technology for automatically setting high-capacity optical communication paths.
The remainder of this article introduces a numberof initiatives in cutting-edge technologies undertaken
by NTT Network Innovation Laboratories toward
IOWN/6G divided into frontier communication technologies, wave propagation technologies, and innovative transport technologies.

NTT Technical Review

Vol. 20 No. 7 July 2022

2. Frontier communication technologies
An outline of frontier communication technologies
is shown in Fig. 2. With the aim of achieving cyberphysical systems such as smart cities, we are working
on optical-path automatic optimization/control technology for connecting computing resources distributed over a wide area on the IOWN APN and highspeed remote data-transfer technology for achieving
high-capacity, low-latency data transfers. These
technologies will lead to the establishment of an optical network platform for ultra-wide-area distributed
computing. We are also working on natural media
processing technology to transmit information on theconfiguration of physical space in real time with low
latency toward ultra-high-presence communication
including video and on multimodal wireless environment-comprehension/prediction technologies to predict, set, control, and manage communication quality
and the environment through sensing and artificial
intelligence (AI). We are also taking up the challengeof establishing secure-transmission technology forenvisioning the use of quantum computers.
2.1 O ptical network platform for ultra-wide-area
distributed computing
At NTT Network Innovation Laboratories, we aredeveloping technology for dynamically connecting
11

Feature Articles

Intelligent transport

Frontier transport
Prediction/inference

Comprehension/visualization

Set, control, manage

Multimodal wirelessenvironment-comprehension/
prediction technologies
Long-distance, high-capacity,
high-security communications

Disaggregated equipment configuration, open scheme
IOWN APN (optical)

Sensor device
IOWN (wireless)Quantum cryptography
communications technology

Non-IP optical path
Optical network platform for ultra-wide-area distributed computing
Optical-path automatic optimization/control technology
High-speed remote data-transfer technology

Natural media processing
technology

Fibersensing
Self-driving robot

Environment spaceformation/
comprehension

Fig. 2. Outline of frontier communication technologies.

computing resources, such as accelerators and memory devices, distributed over an ultra-wide area on the
APN through the optimal design and control of transmission modes and optical paths on the basis of optical characteristics. This technology will enable theconstruction of optical paths that extract maximum
optical transmission performance. Our aim is theestablishment of an optical network platform toward
ultra-wide-area distributed computing essential to thedevelopment of smart cities through high-speed
remote data-transfer technology, which will makeexclusive use of bandwidth on ultra-long-distance
optical paths and synchronize remote memory devices with extreme low latency.
2.2 Natural media processing technology
We are developing spatial reconstruction technology with extreme low latency with a view to ultrahigh-presence communication. This will be accomplished by accommodating information on the configuration of physical space in APN optical paths as atime series of high-speed digital signals, enhancing
high-speed media transfers with low-latency features
to enable long-distance transmissions with extreme
low latency, and using lightweight machine learning.
Looking to the future, we aim to establish cross-layer
AI monitoring technology to collect the input/output
of people, things, and the environment across multiple layers and make inferences from those data. Our
goal with this technology is to enable automatic optimization even during system operation through state
NTT Technical Review

Vol. 20 No. 7 July 2022

monitoring and detection of anomalies and theircauses.
2.3 M
 ultimodal wireless environment-comprehension/prediction technologies
To satisfy a variety of requests for network services
to be provided by IOWN, we are developing technologies for using diverse types of physical space
information obtained using cameras and sensors forpredicting communications quality several seconds
into the future and for maintaining high-quality communications at all times through optimal communication means. Our goal is to establish environmentcomprehension technology that uses the relationshipbetween physical space information and wirelesscommunication system information to extract theformer, such as location information, from the latter.
2.4 Q
 uantum cryptography communications
technology
NTT Network Innovation Laboratories is exploring
new technology areas based on optical technologies
toward long-term innovative contributions to the
IOWN initiative. Specifically, we aim to use opticalfiber transmission technology and communication
protocol technology to achieve a quantum cryptography communications system that overcomes the limitations in distance and communication speeds inconventional quantum key distribution. We will alsoendeavor to establish security and transmission technologies for the age of quantum computers.
12

Feature Articles(1) Underwater acoustic
communication technology(2) Ultra-wide-area, high-speed
network configuration technology(3) Terabit-class wirelesstransmission technology

Free space propagation

Satellite sensing

Spatio-temporalequalization,
spectrumdivisiontransmission
Resources

Secure confined spaces

100G digitalcoherent RX

100G digitalcoherent TX(4) Wave adaptation andcontrol technology

OAM-MIMO
transmission

FSO

Core network

Shielding

Ultra-high-speed fronthaul/backhaul……

IoT terminals

Extreme coverage

4K/8K uncompressedvideo transmission

High speed, large capacity

Physically impossible
to intercept

Bent beam

Free wireless space

Creation of wireless value as a social
platform of the IOWN/6G era
Robust core technologies: radio-wave propagation technology, wireless signal processing technology

Fig. 3. R&D related to wave propagation technology.

3. Wave propagation technologies
Wireless communications in the IOWN/6G era is
expected to have a variety of features. These include
extreme coverage extension that includes coverage inthe sea and outer space in addition to terrestrial communications, high-speed/high-capacity transmission
irrespective of the points being connected, and flexible wireless space formation that confines radio
waves to the targeted area, all with the aim of creating
new value in wireless communications. At NTT Network Innovation Laboratories, we are working onfour key technologies toward the creation of wirelessvalue as a social platform for the IOWN/6G era: (1)underwater acoustic communication technology, (2)ultra-wide-area, high-speed network configuration
technology, (3) terabit-class wireless transmission
technology, and (4) wave adaptation and control technology (Fig. 3).
3.1 U
 nderwater acoustic communication technology
Wireless communication technology based onradio waves capable of high-speed, stable, and longdistance communication has not been established forunderwater applications; as a result, unmannedprobes and undersea heavy equipment used forresource development on the ocean floor or for port
construction typically connect with a support vesselon the surface of the sea using a communication cable
100 m or longer to achieve remote operations [3].
This state of affairs has been a major issue from the
NTT Technical Review

Vol. 20 No. 7 July 2022

viewpoint of work efficiency. We aim to establish
Mbit/s-class underwater acoustic communication
technology to enable wireless remote control ofunderwater equipment and devices. This technology
will include spatio-temporal equalization technology
to compensate for significant waveform distortion of
acoustic waves and spectrum-division transmission
technology to achieve high speeds through broadband transmission.
3.2 U ltra-wide-area, high-speed network configuration technology
Areas in which people do not live, such as in highly
rural and mountainous areas and the open sea, may
not have a terrestrial wireless communication infrastructure for economic reasons. Our aim is to achieveultra-wide-area, high-speed networks in areas where
no wireless communication infrastructure has been
set up. To this end, we are pursuing satellite Internetof Things (IoT) platform technology [4] to enable thecollection of sensor data on a global scale and FSO
platform technology to enable 100-Gbit/s-class highspeed communications in areas such as the open sea.
3.3 T erabit-class wireless transmission technology
Terabit-class wireless transmission technology is
considered necessary for the fronthaul and backhaul
in the IOWN/6G era [5]. We aim to further increasetransmission bandwidth and the amount of spatial
multiplexing essential to terabit-class wireless transmission through digital signal processing technologies.
13Feature Articles

These include OAM and line-of-sight multi-input
multi-output (MIMO) schemes for achieving spatial
multiplexing and high-frequency-band Butler circuit
configuration technology for increasing the bandwidth of OAM multiplexed transmission.
3.4 Wave adaptation and control technology
In wireless communications, radio waves tend to
spread in all directions even outside the area targeted
for communications. This leakage of radio signals
outside the intended area results in a drop in confidentiality, increase in interference, and power loss asuniversal problems in wireless communications. We
aim to establish the ultimate in wireless communications by preventing the leakage of radio waves using
wave-control techniques. These will include terminal-coordinated, user-centric radio access network
technology that links not only base stations but also
terminals to form wireless space in an adaptive manner and multi-shape wave-control technology to fully
control radio-wave trajectories.
4. Innovative transport technologies
NTT Network Innovation Laboratories is pursuinginnovative transport technologies for achieving
extreme-high-capacity optical paths as a platform for
IOWN. In particular, we are researching and developing digital-coherent optical transmission technology
including the development of a low-power, 1-Tbit/sclass DSP for optical communications as well as
opto-electronic integration technology to link lightand electricity and analog and digital circuits. We arealso working on innovative Layer 1 networking technology to generate new value for users and operatorsand pioneering elemental technologies toward a
dramatically new user experience (UX) in the APN.
With the aim of efficiently accommodating massive
amounts of traffic in the future, we have taken on thedevelopment of innovative extreme-high-capacitytransmission technology with a 10-Pbit/s-classthroughput per node, which is enabled by scalable
optical transport platform technology driving thedevelopment of optical signal processing technology.
4.1 F
 LEX digital-coherent optical transmission
platform technology
NTT Network Innovation Laboratories aims toestablish FLEX digital-coherent optical transmission
platform technology as an elemental technology foreconomically achieving IOWN (Fig. 4). This technology will achieve long-distance and low-power
NTT Technical Review

Vol. 20 No. 7 July 2022

high-capacity optical transmission of the 1-Tbit/sclass essential to the construction of the APN. In
addition to conventional long-distance transmission
oriented to carriers, this technology is also applicable
to short-range transmission such as datacenter interconnects. For rapidly expanding application areas, itaims to achieve optimal optical paths by using a digital signal processing function to make flexible changes to the transmission scheme, type of compensation
processing, etc. in addition to high-accuracy transmission path monitoring.
4.2 S
 calable optical transport platform technology
A petabit-class optical network will be needed toaccommodate the massive volume of communications traffic that will be generated by the expansion ofcloud services and further penetration of smartphones. As part of our efforts to meet this need, we
have demonstrated bulk extension of the opticalamplification bandwidth through wideband opticalparametric amplification repeaters [6] and undertaken the development of large-capacity spatial-multiplexing transmission technology using core multiplexing and mode multiplexing [7]. Our aim is topioneer innovative extreme-high-capacity transmission technology and optical signal processing technology to make this possible and establish scalable
optical network platform technology of the 1-Pbit/sclass per link.
4.3 Extreme Layer 1 networking technology
We aim to generate value for users and operators
through Layer 1 networking and contribute to theimplementation of the APN by demonstrating specific use cases together with collaborators. To thisend, we are developing elemental technology fordramatically enhancing the UX through instantaneous creation of a Layer 1 communication path toany location and elemental technology for revolutionizing operations by enabling network configuration
changes to be made at will through Layer 1 switching
without interrupting communications. Through thesetechnical developments, we are exploring a variety ofuse cases such as demonstrating that remote e-sports
matches can be conducted fairly even at the professional level by using Layer 1 communication pathdelay/adjustment technology [8].
5. Conclusion
This article outlined cutting-edge technologies now
14

Feature Articles

APN

WDM

WDM
Optical node

Optical repeater

Optical node

Enables flexible changes tobaud rate, modulation
scheme, compensation
processing, etc. according
to transmission conditions
Achieves low-power operation
by applying advanced
processes and innovative
signal processing algorithms
Improves characteristics
through DSP digitalcompensation and total designof analog/electrical and opticalcomponent characteristics

System auto calibration

Flex transponders,
100G–1 Tbit/s/

DSP

+
f

+
f

f

Trans- Electrical Opticalband
mission band

Opticalfront-end

Optical-components/DSP integrated Co-PKG(opto-electronic integrated device)

Maximizes transmission characteristics
through high-accuracy transmission pathmonitoring and coordinated compensationof digital processing and optical processing

WDM: wavelength division multiplexing

Fig. 4. Digital-coherent optical transmission platform technology for rollout in APN.

being pursued by NTT Network Innovation Laboratories toward IOWN/6G. Going forward, our goal isto achieve early establishment of various elemental
technologies toward the deployment of IOWN/6G
scheduled for 2030 by collaborating with business
partners and specialists in a variety of industrial
fields.

[4]
[5]

[6]
[7]

References
[1]
[2]
[3]

Y. Kishiyama, S. Suyama, and S. Nagata, “Trends and Target Implementations for 5G evolution & 6G,” NTT Technical Review, Vol. 19,
No. 11, pp. 18–25, Nov. 2021. https://doi.org/10.53829/ntr202111fa1
Website of NTT R&D, IOWN, https://www.rd.ntt/e/iown/0002.html
Specialists Center of Port and Airport Engineering (SCOPE), “Trends
and Outlook for New Ocean Technologies and Unmanned Construction,” SCOPE NET, Vol. 70, Sept. 2014 (in Japanese).

NTT Technical Review

Vol. 20 No. 7 July 2022

[8]

Press release issued by NTT, May 29, 2020 (in Japanese).
https://group.ntt/jp/newsrelease/2020/05/29/200529a.html
Z. Zhang, Y. Xiao, Z. Ma, M. Xiao, Z. Ding, X. Lei, G. K. Karagiannidis, and P. Fan, “6G Wireless Networks: Vision, Requirements, Architecture, and Key Technologies,” IEEE Veh. Technol. Mag., Vol. 14,
No. 3, pp. 28–41, Sept. 2019.
Press release issued by NTT, Jan. 28, 2021 (in Japanese). https://group.ntt/jp/newsrelease/2021/01/28/210128b.html
Press release issued by NTT, “World’s First Transpacific-class Transmission Experiment Employing Mode-multiplexed Optical Signals—
Key Achievement for Future High-capacity Optical Transport Network Infrastructure Supporting the Spread of 5G and IoT Services,”
Mar. 9, 2020.
https://group.ntt/en/newsrelease/2020/03/09/200309b.html
Press release issued by NTT, “Unprecedented Optical Network Technologies to Support Esports Achieving Breathtaking Gaming UX in
Remote Esports Events,” Nov. 2, 2021.
https://group.ntt/en/newsrelease/2021/11/02/211102b.html

15

Feature Articles

Shigeru Iwashina

Kazunori Akabane

Vice President, NTT Network Innovation
Laboratories.
He received a B.E. and M.E. from Tokyo Institute of Technology in 1993 and 1995. Sincejoining NTT DOCOMO in 1995, he has been
engaged in the development of core network
equipment for mobile communications systems,
network architecture research, and network virtualization research. After conducting R&D on
mobile communications systems ranging from
2G to 5G, he was assigned to the R&D Planning
Department of NTT in July 2019 to promote
R&D collaboration with partners to develop andimplement NTT’s IOWN initiative. He has beenin his current position since July 2020.

Director, Wave Propagation Laboratory, NTT
Network Innovation Laboratories.
He received a B.E. and M.E. from Keio University in 1994 and 1996. He joined NTT Wireless
Systems Laboratories in 1996, where he has been
involved in the R&D of software-defined radio
systems and IoT/machine-to-machine wirelessaccess systems. He is currently engaged in R&D
of high-speed underwater acoustic communication and terabit-class wireless transmission
toward IOWN/6G.

Katsuhiro Shimano

Director, Transport Innovation Research Laboratory, NTT Network Innovation Laboratories.
He received a B.E., M.S., and Ph.D. from the
University of Tokyo in 1993, 1995, and 1998.
After joining NTT in 1998, he started researching advanced optical waveguide devices for communication. From 2002 to 2003, he was a visiting
scholar at Ginzton Laboratory, Stanford University. From 2006 to 2008, he was a director at NTT
Electronics while on leave from NTT laboratories. His current interest is in next-generation
photonic transport technologies. He is a memberof the Institute of Electronics, Information andCommunication Engineers (IEICE), the Japan
Society of Applied Physics (JSAP), and the Institute of Electrical and Electronics Engineers(IEEE), and a fellow of Optica (formerly OSA).

Takashi Saida

Director, Research Planning Section, NTT
Network Innovation Laboratories.
He received a B.S. from Waseda University in1991 and M.S. from the University of Tokyo in
1993. He joined NTT in 1993, where he has been
involved in R&D on optical network control andmanagement, network virtualization, and highperformance software switches. His current
interests include improving the performance of
optical and wireless networks.

Koichi Takasugi

Executive Research Engineer, Director of
Frontier Communication Laboratory, NTT Network Innovation Laboratories.
He received a B.E. in computer science from
Tokyo Institute of Technology in 1995, M.E.
from Japan Advanced Institute of Science andTechnology in 1997, and Ph.D. in engineeringfrom Waseda University in 2004. He wasinvolved in the design and standardization of the
Next-Generation Network architecture. He wasalso active in the AI field, such as diagnosing
diabetes through machine learning. He is currently leading research on the network architecture and protocols in optical and wireless transport networks.

NTT Technical Review

Vol. 20 No. 7 July 2022

16

Feature Articles: Optical and Wireless Transmission
Technologies for IOWN/6G

Future Development of Digital
Coherent Optical Transmission
Technology
Yoshiaki Kisaka, Etsushi Yamazaki, Hideki Nishizawa,and Takashi Saida
Abstract

Digital coherent optical transmission technology—which digitally incorporates physical quantities
such as amplitude, phase, and polarization and maximizes transmission performance through advanced
signal processing—is the latest generation of optical transmission technology; however, the technology
must be further developed to implement the Innovative Optical and Wireless Network (IOWN). The latest trends and future development of digital coherent optical transmission technology are described inthis article from the perspectives of high-speed and high-capacity transmission, low-power devices, andsoftware-based autonomous control.
Keywords: optical transmission technology, digital coherent, All-Photonics Network

1. Introduction
Since the dawn of the Internet, Internet traffic has
continued to grow exponentially in Japan and aroundthe world as various services emerge and spread. To
support this growth in traffic, optical-fiber transmission technology has continued to evolve generationally through the introduction of new technologies
such as wavelength division multiplexing and optical-amplification relay. The latest generation of thesetechnologies is digital coherent optical transmission
technology, which captures all the physical quantities
of light (i.e., polarization, amplitude, and phase) as
digital data and compensates for distortions in optical-fiber transmission lines and optoelectronic devices through advanced signal processing [1].
To implement the Innovative Optical and Wireless
Network (IOWN), which aims to integrate the realand virtual worlds into a rich society that embraces
diversity, optical transmission technology must also
evolve to provide even higher capacity, lower power
consumption, and higher functionality. The latest
trends and future development of digital coherent
NTT Technical Review

Vol. 20 No. 7 July 2022

optical transmission technology are introduced interms of high-speed and high-capacity transmission,compact and low-power devices, and software-based
autonomous control.
2. Initiatives concerning further increase in
speed and capacity
The expansion of communications within andbetween datacenters and the penetration of applications such as on-demand video streaming and cloudcomputing have further increased the demand for
optical transmission with higher capacity. Regarding
high-capacity networks, capacity per wavelength
channel must be increased before the network can bebuilt economically. Evolution of optical transmission
capacity per wavelength channel in the cases ofoffline experiments and in commercial systems isshown in Fig. 1(a). High-capacity transmission
exceeding 1 Tbit/s per wavelength has been experimentally demonstrated. In these demonstrations,modulation speed significantly increased and modulation schemes with higher modulation levels/high-order
17

(a)

(b)
5000
Real-time implementation

500
(c)

100–600 Gbit/s
32–64 Gbaud
16-nm FinFET CMOS

50
2010

100/40 Gbit/s
40-nm CMOS

2012

2014

100/150/200 Gbit/s
QPSK/8QAM/16QAM
20-nm CMOS

2016

2018

2020

Year

Full C-band WDM (~34 nm, 4.2 THz)

175-GHz grid
Before transmission
After 1280 km
After 3840 km

1528

1538

1548
1558
Wavelength (nm)

2022

1568

1
NGMI limit
1 Tbit/s/
1.2 Tbit/s/

0.95
NGMI

Capacity per lambda (Gbit/s)

Offline experiment

Optical power (10 dB/div.)

Feature Articles

0.9
0.85
0.8

0

1000

2000

3000

4000

5000

Transmission distance (km)CMOS: complementary metal-oxide semiconductor
FinFET: fin field-effect transistor
QPSK: quadrature phase shift keying

Fig. 1. Progress of transmission capacity per wavelength and examples of transmission experiments exceeding 1 Tbit/s.

modulated signals were introduced. The modulation
speed in the first generation of digital coherent optical transmission (100 Gbit/s per wavelength) was 32
Gbaud, but it has increased to 100 Gbaud and higher[2, 3]. This increased modulation speed was a resultof increasing the speed of analog components such as
analog-to-digital converters (ADCs) and digital-toanalog converters (DACs), optical modulators, drivers, photodiodes, and transimpedance amplifiers[4–6]. Probabilistic constellation shaping (PCS) [7],which can achieve a capacity close to the Shannon
limit, has attracted attention and is used together withquadrature amplitude modulation (QAM) composedof 64 or more constellation points.
Signal distortion caused by analog components inthe transmitter and receiver is a major issue in regard
to increasing the capacity of optical transmission
systems. As the symbol rate increases and the multimodulation level becomes higher, signal distortions
are caused by frequency-bandwidth limitation andcrosstalk in the analog components and the electricalwiring in printed circuit boards and other components. These distortions are compensated for by setting optimal filter coefficients, which are estimatedby calibration, for each digital equalization filter (i.e.,
one for transmission and one for reception). NTT
NTT Technical Review

Vol. 20 No. 7 July 2022

Network Innovation Laboratories is developing technology to compensate for such waveform distortion
with high accuracy. There are two problems withdigital estimation: (i) increasing the peak-to-average
power ratio and lowering signal quality [8] and (ii)tracking the time variation of waveform distortiondue to operating temperature, age deterioration, andcontrol dither. To ensure sufficient transmission performance, it is important to divide the roles of senderside pre-equalization and receiver-side equalization.
By applying our research and development (R&D)
technologies to address these issues, we have commercialized an application-specific integrated circuitthat achieves digital coherent transmission at a rate of
up to 600 Gbit/s [9].
Regarding the development of a next-generation
optical transmission system, NTT Network Innovation Laboratories has further increased modulation
speed to 168 Gbaud and experimentally demonstrated transmissions of a 1-Tbit/s light signal over 3840
km and of 1.2-Tbit/s light signal over 1280 km [2].
We achieved these high modulation speeds by using
two analog multiplexer (AMUX) front-end integrated
modules [5], which respectively use polarizationmultiplexed PCS-16QAM and 36QAM modulation
schemes. This light signal is wavelength-division18

Feature Articles

multiplexed (WDM) with optical frequency spacingof 175 GHz. The spectrum of the WDM signal before
and after transmission is shown in Fig. 1(b). The
transmission line consists of pure-silica-core fiberwith an 80-km optical-amplification relay section anduses both backward-pumping distributed Raman
amplification and erbium-doped-fiber amplification.
Each amplification achieves spectral-utilization efficiencies of 5.71 and 6.85 bit/s/Hz, respectively. Measured dependence of normal generalized mutual
information (NGMI) on transmission distance for
1-Tbit/s and 1.2-Tbit/s signals is shown in Fig. 1(c).
As shown in the figure, NGMI values were above
0.857, namely, error-correction threshold for 21%
redundancy level, for the 1-Tbit/s signal after 3840km transmission and for the 1.2-Tbit/s signal after
1280-km transmission. This result indicates thaterror-free transmission is possible. Note that in thisexperimental demonstration, we only used the
C-band, but we are conducting R&D with the aim of
expanding the bandwidth to the L- and S-bands.
3. Initiatives concerning miniaturization andpower reduction
To handle ever-increasing communication traffic,
optical networks must be able to continuously
increase capacity. However, the installation space andpower supply of optical transmission equipment arelimited; accordingly, achieving higher capacity
necessitates smaller, lower-power optical interfaces.
Since miniaturization of devices lowers their heatdissipation performance, low power consumption as
well as miniaturization and high-density mounting of
the devices that configure the optical interface are key
factors in achieving miniaturization. It is therefore
essential to reduce the power consumption of a digital
signal processor (DSP) because it accounts for a large
portion of the power consumption of optical interfaces.
Digital coherent optical transmission technology
was first implemented in long-distance transmission
systems configuring the backbone networks of telecommunications carriers. As the technology has
matured, miniaturization, power reduction, and cost
reduction have progressed, and the technology has
become more widely used in short-distance transmission systems such as metro networks and datacenterinterconnect (DCI) networks. This technology is
expected to be applied to access networks and networks within datacenters in the future. Low-power
technologies will also become increasingly important
NTT Technical Review

Vol. 20 No. 7 July 2022

in regard to creating future massively multi-parallel,
high-capacity optical transmission systems using
multicore fibers with spatial multiplexing.
The progress in miniaturization and power reduction of coherent transceivers is shown in Fig. 2. The
DSPs developed and practically applied at NTT laboratories, i.e., installed in a coherent transceiver, areshown below the horizontal axis. Coherent transceivers are mainly used for metro and DCI networks, andtheir miniaturization and power-consumption reduction are rapidly progressing in a similar manner tolow-power DSPs. Currently, 400-Gbit/s pluggable
transceivers, such as the C form-factor pluggable 2
digital coherent optics (CFP2-DCO) and quad small
form-factor pluggable double density (QSFP-DD),are in practical use.
DSPs for optical communication are required tooperate at extremely high speeds and low power consumption, so the state-of-the-art complementary
metal-oxide semiconductor (CMOS) process isalways applied to reduce power consumption. However, this power consumption cannot be reduced sufficiently by improving the CMOS process. Accordingly, application of new signal-processing algorithms, selection of functions in accordance withapplication [10], and optimization of performance
have progressed in regard to functions such as reduction in the digital sampling rate, wavelength-dispersion compensation, adaptive equalization, and forward-error correction (FEC). Although the application of state-of-the-art CMOS processes and innovations in signal-processing algorithms have significantly reduced the power consumption of digital circuits, the power consumption of analog circuits, suchas DACs and ADCs, has not yet been significantly
reduced, partly due to the effect of the higher transmission and reception signal speeds of those circuits.
For that reason, the power consumption of analog
circuits accounts for a larger percentage of the total
power consumption in each generation of DSP, andone of the major challenges for further power reduction is to reduce the power consumption of analog
circuits.
With the aim of achieving further miniaturizationand lower power consumption, co-package mounting
technology, in which a DSP and silicon-photonicsbased optical transmitter-receiver device called
coherent optical subassembly (COSA) [11] aremounted in a single package, has been researched anddeveloped. High-density mounting of the DSP and
COSA enables significant miniaturization, while
shortening the high-speed analog electrical wiring
19

Feature Articles

2010

2015

2020

Board mountPower consumption

OIF-compliant 5 x 7" module
OIF-compliant
4 x 5" module
OIF-com
CFP

CFP2-DCOOSFP/QSFP-DD
QSFP

100G
40 nm

LP 100G
20 nm

LP 200G
16 nm

LP 400G
7 nm

LP: low power
OIF: Optical Internetworking ForumOSFP: octal small form-factor pluggable

Fig. 2. Miniaturization and power reduction of coherent transceivers.

between the DSP and COSA minimizes losses andkeeps signal drive power low. To implement the
future IOWN-based All-Photonics Network, we areconducting R&D aimed at drastically reducing power
consumption by offloading functions that are currently executed by digital signal processing to opticalprocessing. NTT Network Innovation Laboratories
has demonstrated a significant improvement in opticalsignal-to-noise ratio (SNR) tolerance for 800-Gbit/sclass optical transmission by compensating for signal-waveform distortion caused by the bandwidthcharacteristics of the optical transmitter and receiverin conjunction with digital signal processing andoptical processing [12]. This performance improvement makes it possible to simplify digital signal processing, thus reduce the power consumption of the
DSP.
4. Initiatives concerning automated control
using software
Software-defined wide-area network (SD-WAN)
technology, which separates the physical network anddevice hardware from their control planes and manages them using software, has been commercially
NTT Technical Review

Vol. 20 No. 7 July 2022

introduced in the market for forwarding equipment
such as routers, and a system that connects remoteuser sites and enables centralized control using software is now in place. In the era of full-fledged IOWN,
user terminals equipped with various transmissionfunctions, such as routers with coherent modules andwhite-box switches, will become increasingly popular. To connect these terminals while attaining low
latency and low power consumption by reducing thenumber of optical-electrical conversions, using directconnection via a carrier network is being considered[13], and it is expected that software-based automatic-configuration technology for optical transmission
networks will be implemented.
However, for optical transmission below layer 1,
complex physical factors such as wavelength dependence of optical amplifiers and nonlinear opticaleffects in fiber have been considered a barrier to automated control using software as in the case of SDWANs. To dynamically connect arbitrary user ends, a
new method for quickly estimating transmission-pathcharacteristics that determine transmission distanceand capacity of optical transmission networks andreal-time measurement of transmission quality using
receivers are required.
20

Feature Articles

Estimate transmission quality andcalculate optimal transmission mode

GNPy

Controller(1) Collect basic informationand functions possessed by
the transmitter/receiver DSP(3) Set optimal transmission
mode and establish optical
path in a few minutes

TAI

TAI

Vendor-A device

Vendor-B device

Base A

Base B(2) Determine transmission-path status in the mode thatcan be connected to both transmitter and receiver sides

Fig. 3. Automatic optimization of optical-path transmission mode between different vendors and devices.

The Open Optical & Packet Transport project groupof the Telecom Infra Project [14], which aims todefine open technologies, architectures, and interfaces in optical and Internet protocol networking, isdeveloping GNPy [15], an open-transmission-path
design tool based on transmission-quality estimation
using Gaussian-noise models, and a transponder
abstraction interface (TAI) [16], which separates the
hardware and software of a transponder. NTT Network Innovation Laboratories is contributing to openness in these areas and aims to implement a technology that automatically optimizes the transmission
mode of optical paths between different vendors anddevices and establishes optical-path connections in ashort time by using open interfaces and tools such asa TAI and GNPy (Fig. 3).
Manual work has traditionally been required toconnect such optical paths, and the process from service order to optical-path connection took a longtime, ranging from several days to several weeks. As
schemes for digital coherent optical transmission
become more sophisticated and transmission modes
become more diverse, the design and coordination of
optical paths are becoming more complex. Our developed technology calculates the optimal transmission
NTT Technical Review

Vol. 20 No. 7 July 2022

mode (modulation scheme and FEC type)—even in
an environment in which multi-vendor transmission
equipment has been deployed—by collecting suchinformation via a TAI and estimating transmissionpath conditions and transmission quality by means of
GNPy and other methods. Moreover, by setting that
information to the transmitter and receiver DSPs, an
optical-path connection is established in a few minutes [17]. Specifically, these procedures/processes
are carried out with the following sequence of steps.
(1)	The basic information and functions possessed by the transmitter and receiver DSPs
(configurable modulation scheme, FEC type,etc.) are collected via the TAI, and the opticalpath connection is established after setting atransmission mode that can be connected toboth the transmitter and receiver.
(2)	Via step (1), parameters, such as bit error rate
and wavelength dispersion, are collected fromthe receiver DSP, and the quality of the transmission channel is estimated. From theseparameters, the optimal transmission mode iscalculated—with appropriate margin settings—on the basis of geometric SNR estimated by GNPy.
21

Feature Articles(3)	On the basis of the calculated transmissionmode, parameters such as modulation method
and FEC type for the transmitter and receiver
DSPs are set, and an optical path between
transmission devices of different vendors is
opened in a few minutes.
5. Concluding remarks
To implement optical transmission technology that
supports IOWN, NTT Network Innovation Laboratories will research and develop digital coherent opticaltransmission technology from the viewpoints of highspeed and large-capacity transmission, low-powerdevices, and software-based autonomous control.
References
[1]
[2]

[3]

[4]

[5]

K. Kikuchi, “Fundamentals of Coherent Optical Fiber Communications,” J. Lightwave Technol., Vol. 34, No. 1, pp. 157–179, 2016.
M. Nakamura, F. Hamaoka, M. Nagatani, Y. Ogiso, H. Wakita, H.
Yamazaki, T. Kobayashi, M. Ida, H. Nosaka, and Y. Miyamoto, “192Gbaud Signal Generation Using Ultra-broadband Optical Frontend
Module Integrated with Bandwidth Multiplexing Function,” Proc. of
the Optical Fiber Communication Conference and Exhibition (OFC)
2019, Th4B.4, San Diego, USA, Mar. 2019.
M. Nakamura, T. Sasai, K. Saito, F. Hamaoka, T. Kobayashi, H.
Yamazaki, M. Nagatani, Y. Ogiso, H. Wakita, Y. Kisaka, and Y. Miyamoto, “1.0-Tb/s/λ 3840-km and 1.2-Tb/s/λ 1280-km Transmissions
with 168-GBaud PCS-QAM Signals Based on AMUX Integrated
Frontend Module,” Proc. of OFC 2022, San Diego, USA, Mar. 2022.
M. Zhang, C. Wang, X. Chen, M. Bertrand, A. S. Ansari, S. Chandrasekhar, P. Winzer, and M. Lon ar, “Ultra-high Bandwidth Integrated Lithium Niobate Modulators with Record-low Vπ,” Proc. of
OFC 2018, Th4A.5, San Diego, USA, Mar. 2018.
Y. Ogiso, J. Ozaki, Y. Ueda, H. Wakita, M. Nagatani, H. Yamazaki, M.

NTT Technical Review

Vol. 20 No. 7 July 2022

Nakamura, T. Kobayashi, S. Kanazawa, T. Fujii, Y. Hashizume, H.
Tanobe, N. Nunoya, M. Ida, Y. Miyamoto, and M. Ishikawa, “Ultrahigh Bandwidth InP IQ Modulator for Beyond 100-GBd Transmission,” Proc. of OFC 2019, M2F.2, San Diego, USA, Mar. 2019.
[6] M. Nagatani, H. Wakita, H. Yamazaki, Y. Ogiso, M. Mutoh, M. Ida, F.
Hamaoka, M. Nakamura, T. Kobayashi, Y. Miyamoto, and H. Nosaka,“A Beyond-1-Tb/s Coherent Optical Transmitter Front-end Based on
110-GHz-bandwidth 2:1 Analog Multiplexer in 250-nm InP DHBT,”IEEE J. Solid-State Circuits, Vol. 55, No. 9, pp. 2301–2315, 2020.
[7] T. Yoshida, M. Karlsson, and E. Agrell, “Short-block-length Shaping
by Simple Mark Ratio Controllers for Granular and Wide-range Spectral Efficiencies,” Proc. of the 43rd European Conference on OpticalCommunication (ECOC 2017), Gothenburg, Sweden, Sept. 2017.
[8] T. Sasai, M. Nakamura, E. Yamazaki, A. Matsushita, S. Okamoto, K.
Horikoshi, and Y. Kisaka, “Wiener-Hammerstein Model and Its
Learning for Nonlinear Digital Pre-distortion of Optical Transmitters,” Opt. Express, Vol. 28, No. 21, pp. 30952–30963, Oct. 2020.
[9] ExaSPEED-TERA, https://www.ntt-electronics.com/en/products/
photonics/exaspeed-tera.html[10] O. Ishida, K. Takei, and E. Yamazaki, “Power Efficient DSP Implementation for 100G-and-beyond Multi-haul Coherent Fiber-optic
Communications,” Proc. of OFC 2016, W3G.3, Anaheim, USA, Mar.
2016.
[11] S. Yamanaka and Y. Nasu, “Silicon Photonics Coherent Optical Subassembly for High-data-rate Signal Transmissions,” Proc. of OFC
2021, Th5F.2, San Francisco, USA, June 2021.
[12] A. Matsushita, M. Nakamura, K. Horikoshi, S. Okamoto, F. Hamaoka, and Y. Kisaka, “64-GBd PDM-256QAM and 92-GBd PDM64QAM Signal Generation Using Precise-digital-calibration aided by
Optical-equalization,” Proc. of OFC 2019, W4B.2, San Diego, USA,
Mar. 2019.
[13] IOWN Global Forum, Technology, https://iowngf.org/technology/[14] The Open Optical & Packet Transport group, Telecom Infra Project,https://telecominfraproject.com/oopt/[15] GNPy, https://gnpy.readthedocs.io/en/master/[16] TAI on GitHub, https://github.com/Telecominfraproject/oopt-tai[17] K. Anazawa, S. Kuwabara, T. Sasai, T. Mano, T. Inoue, T. Inui, and H.
Nishizawa, “Automatic Modulation-format Selection with White-box
Transponders: Design and Field Trial,” Proc. of the 26th Optoelectronics and Communications Conference (OECC 2021), JS2A.14,
Oct. 2021.

22

Feature Articles

Yoshiaki Kisaka

Senior Research Engineer, Supervisor, Transport Innovation Research Laboratory, NTT Network Innovation Laboratories.
He received a B.S. and M.S. in physics from
Ritsumeikan University, Kyoto, in 1996 and1998. In 1998, he joined NTT Optical Network
Systems Laboratories, where he was engaged in
R&D on high-speed optical communication systems including 40-Gbit/s/ch WDM transmission
systems and the mapping/multiplexing scheme inthe optical transport network (OTN). From 2007
to 2010, he joined NTT Electronics Technology
Corporation, where he was engaged in the development of 40/100-Gbit/s OTN framer large-scale
integration. His current research interests are inhigh-speed and high-capacity optical transport
networks using digital coherent technology.
Since 2010, he has contributed to R&D of DSP
for 100-Gbit/s and beyond optical transmission.
He is a member of the Institute of Electronics,
Information and Communication Engineers(IEICE).

Hideki Nishizawa

Senior Research Engineer, Supervisor, Transport Innovation Research Laboratory, NTT Network Innovation Laboratories.
He received a B.E. and M.E. in physics from
Chiba University in 1994 and 1996. In 1996, hejoined NTT Network Service Systems Laboratories, where he was engaged in research on photonic switching systems. In 2001, he moved to
NTT Phoenix Communications Inc. (now NTT
BizLink Inc.), where he worked on the development of visual communication services. In 2007,
he moved to NTT Research and Development
Planning Department, where he was involved inplanning R&D activities for future networks.
Since 2014, he has been with NTT Network
Innovation Laboratories. His current researchinterests include disaggregated optical transport
networks that use digital coherent technology
and virtualization technology. He is a member of
IEICE.

Takashi Saida

Etsushi Yamazaki

Senior Research Engineer, Supervisor, Transport Innovation Research Laboratory, NTT Network Innovation Laboratories.
He received a B.E. and M.E. in electronicengineering from the University of Tokyo in2000 and 2002. He is currently with NTT Network Innovation Laboratories, where he has been
engaged in R&D for optical transmission systems and photonic networks. He was a visiting
researcher at Stanford University from 2018 to2019. He is a member of the Institute of Electrical and Electronics Engineers (IEEE) and IEICE.
In 2012, he received the Kenjiro Sakurai Memorial Prize from the Optoelectronics Industry andTechnology Development Association (OITDA)
of Japan.

NTT Technical Review

Vol. 20 No. 7 July 2022

Director, Transport Innovation Research Laboratory, NTT Network Innovation Laboratories.
He received a B.E., M.S., and Ph.D. from the
University of Tokyo in 1993, 1995, and 1998.
After joining NTT in 1998, he started researching advanced optical waveguide devices for communication. From 2002 to 2003, he was a visiting
scholar at Ginzton Laboratory, Stanford University. From 2006 to 2008, he was a director of
NTT Electronics while on leave from NTT laboratories. He is currently a director of NTT Network Innovation Laboratories. His current interest is in next-generation photonic transport technologies. He is a member of IEICE, the Japan
Society of Applied Physics (JSAP), and IEEEand a fellow of Optica (formerly OSA).

23

Feature Articles: Optical and Wireless Transmission
Technologies for IOWN/6G

Research and Development of
Scalable Optical Transport
Technologies
Yutaka Miyamoto, Kei Watanabe, and Kazuhide Nakajima
Abstract

This article discusses the current state and prospects of scalable optical transport technologies that candramatically expand the optical amplification and electrical signal processing bandwidths for achievinga Pbit/s-class long-range optical network toward IOWN (the Innovative Optical and Wireless Network)
All-Photonics Network. We explain the optical parametric amplification repeater technology, which has
the potential to achieve longer transmission distances while expanding the amplification bandwidth ofconventional optical amplification repeaters by more than 2.5 times. We also look into the space divisionmultiplexing optical communication technology that uses multiple-input and multiple-output signal
processing and has the potential to increase transmission capacity by more than 10 times at the same
cladding diameter as conventional optical fibers.
Keywords: ultra-large-capacity optical communications, optical parametric amplification repeater,
mode-multiplexing optical communications

1. Introduction
The fifth-generation mobile communications (5G)
system was introduced commercially in Japan in
FY2020. The 5G system is forecast to evolve into an
infrastructure for the Internet of things society inwhich all things, including autonomous driving, connect to highly reliable, low-latency, and large-scale
networks. With the acceleration of the evolution of
network infrastructure technologies through thedevelopment of the Innovative Optical and Wireless
Network (IOWN) All-Photonics Network (APN),
next-generation Beyond 5G technologies following
the 5G system are expected to be widely used as network service technologies in the 2030s. The continued evolution of the network infrastructure is considered essential for flexibly supporting changes in the
global industrial structure and in lifestyles brought
about by the COVID-19 pandemic.
A high-capacity network infrastructure that supports the evolution of broadband services requires thecontinued evolution of high-capacity optical transNTT Technical Review

Vol. 20 No. 7 July 2022

port networks. NTT, as shown in Fig. 1, has been
pursuing the continued development of high-capacity
optical transport systems and networks based on single-mode fiber (SMF) cables, which were introducedin the 1980s. The system capacity per fiber core,
along with the continuous technological innovationsof various optical communication systems, has
evolved at a rate of nearly 1.4 times per annum (1000times in 20 years), wherein the system capacity is
expected to exceed 1 Pbit/s by the 2030s. Digital
coherent optical communication technology using
digital signal processing has been commercialized asa transmission technology that takes advantage of thecoherent nature of lightwave and maximizes thetransmission characteristics of SMF. In 2019, an optical transport network with a capacity of 16 Tbit/s per
fiber core was commercialized [1]. However, it has
become clear that the physical capacity limit (capacity crunch) of SMF, a transmission medium that has
supported long-distance optical transport network
infrastructure, exists near the 100-Tbit/s capacity,which is approximately 10 times that of the current
24

Re

SDM

1P
Physical limit of SMF

100T
10T
1T

x

Digital coherent
transmission

m

ste

sy

C

WDM andopticalamplification

1T
100G

TDM

10G

1G

: Research (with SDM)
: Research (without SDM)
: Commercial system in NTT

100M
10M

l
cia

ar er
/yemm
4
.
o
1

100G
10G

h

arc

se

10P

1980

1990

2000

2010

2020

1G
100M

Channel capacity (bit/s)

Transmission capacity per fiber core (bit/s)

Feature Articles

2030

Year
SDM: space division multiplexing
TDM: time division multiplexing
WDM: wavelength division multiplexing

Fig. 1. Evolution of ultra-large-capacity optical communication system technologies.

capacity, pointing to the need for innovative technologies to achieve continued growth in system capacity.
This article describes two approaches to overcome
the capacity crunch toward the IOWN APN throughscalable optical transport technologies. The first is abroadband optical parametric amplification repeater
technology that economically increases the SMF
capacity by expanding to more than twice the opticalsignal bandwidth of optical amplification repeaters
using conventional SMF. The second is a space division multiplexing (SDM) optical communication
technology that creates multiple independent parallel
communication channels multiplexed by introducing
new degrees of freedom, such as multi-cores andmulti-modes, in a single strand of optical fiber toachieve high-capacity communication in the Pbit/sclass capacity beyond the limit of current SMF. The
following sections describe the recent progress in theindividual technology elements.
2. Broadband optical parametric amplification
repeater technology
For the current optical fiber transmission systems,
large-capacity transmission has been achieved bymultiplexing optical signals of approximately 100
wavelengths into the optical wavelength band ofaround 4 THz, which is the amplification bandwidthof an erbium-doped optical fiber amplifier (EDFA),and by expanding the transmission capacity per
NTT Technical Review

Vol. 20 No. 7 July 2022

wavelength of an optical signal using digital coherent
technology*1. For the APN, which is part of the
IOWN initiative proposed by NTT, we are aiming tobuild a flexible optical network that uses abundant
wavelength resources. Along with increasing conventional capacity per wavelength, we are also aiming to
expand the available wavelength resources (optical
wavelength band). NTT has been pursuing researchand development (R&D) focusing on optical parametric amplification*2 using a periodically poledlithium niobate (PPLN)*3 waveguide as a wide-band
and low-distortion optical amplification technology*1 Digital coherent technology: A transmission method that combines massive digital signal processing and coherent detection.
Coherent detection enables a high-sensitivity receiver for the amplitude and phase modulated optical signals by introducing interference between the light source placed on the receiving side andreceived light signal. In addition to increasing spectral efficiency
through polarization multiplexing and modulation schemes using
amplitude and phase of optical signals, digital coherent technology can achieve higher receiver sensitivity through high-precision
distortion compensation of optical signals using digital signal
processing in combination with the coherent detection.
*2 Optical parametric amplification: Light at a specific wavelengthis amplified through interaction between light of different wavelengths using the nonlinear optical effects generated in the material. High-nonlinear fibers and lithium niobate (LiNbO3) crystal
are known as nonlinear media.
*3 PPLN: An artificial spontaneous-polarization crystal in which thedirections of positive and negative charges in the crystal are forcibly inverted at a fixed period in a nonlinear medium, e.g.,
LiNbO3. PPLN enables a nonlinear optical effect that is significantly higher than that of the original LiNbO3 crystal.

25

Transmission capacity for a single-core SMF (Tbit/s)

Feature Articles

Technology for expanding
the electric signal bandwidth(AMUX, etc.)

120

NTT (EDFA)

Bro

adb

100

80

60

NTT (PPLN)

andOther companies

opt

ical

par

am

etri
c a

mp

Target (2030)

lific

Extension of wavelength resources
by PPLN optical parametricamplification (> 2.5x)

40

atio

n re

lay

(> 1

0T

Hz)

Current ED
FA opticalamplification relay (4 T
Hz)

20

0
1000

2000

3000

4000

Transmission distance (km)
AMUX: analog multiplexer

Fig. 2. E xpansion of existing SMF transmission characteristics through broadband optical parametric amplification repeater
technology and electric signal bandwidth expansion technology. All plots are experimental results with the channel
rate ≥ 800 Gbit/s and optical amplification bandwidth > 4 THz.

[2]. Optical parametric amplification with a PPLN
waveguide can only amplify a single polarized signal
and produce phase conjugated light that is unnecessary for normal optical amplification. These factors,
therefore, posed further challenges in widening theamplification bandwidth and in achieving stable
response for frequent insertion and removal of optical
wavelengths in response to traffic demand in the
future APN with the optical amplification of polarization-multiplexed optical signals currently used indigital coherent systems.
Therefore, we proposed an amplifier configuration
using multiple modular PPLN waveguides (PPLN
modules) to achieve stable amplification of polarization-multiplexed optical signals and the amplification
bandwidth of 10.25 THz with a gain of at least 15 dB.
Using the ultra-high-speed signal-generation technology [3] with a symbol rate of over 100 Gbaud andusing polarized multiplexed digital coherent signalsof 800 Gbit/s per wavelength as verification signals,
we confirmed low-distortion signal amplification in
the gain-saturation region for both single-wavelength
and wavelength-multiplexed signal inputs. We also
confirmed the high-speed response to input signal
NTT Technical Review

Vol. 20 No. 7 July 2022

switching at 1 wavelength and 41 wavelengths
through emulation of the high-frequency variation in
the number of wavelengths expected for the utilization of wavelength resources in the APN. We introduced the optical parametric amplifier as a wideband
inline repeater and demonstrated that the optical signal bandwidth can be expanded to more than 10.25
THz, more than 2.5 times that of conventional technology, using wavelength-multiplexed signals of 800
Gbit/s per wavelength [4].
Going forward, we will pursue research on longdistance transmission performance at 100 Tbit/s,
which is close to the performance limit of current
optical fiber communication systems, as shown in
Fig. 2, by combining broadband optical parametricamplification repeater technology and ultra-highspeed signal-generation technology with high symbolrate.
3. SDM optical communication technology using
mode multiplexing
SDM optical communication technologies are
expected to overcome the SMF capacity crunch and26

Feature Articles

Standard cladding diameter SDM
125 µmTarget (2030)

Spatial multiplex number

100

SDM fiber core-multiplexing cable
SDM optical amplificationand processing node30

Multi-mode
multiplexing (NTT)
Other organizations

FY2022–2023
10-mode class
MIMO long-distance transmission

103Multi-core
multiplexing (NTT)Non-standard cladding diameter SDM
250 µm

1
1

10

100

1000

Transmission distance (km)Multi-core
multi-mode
multiplexing (NTT)
Other organizations

Fig. 3. O
 vercoming the capacity crunch using mode-multiplexed MIMO signal processing and standard cladding diameter
mode-multiplexed fiber mounting technology (spatial multiplexing = 1 is the current SMF transmission system
capacity).

achieve a larger capacity. In consideration of the
manufacturability of SDM optical fiber cable, which
serves as the novel transmission medium, should beequivalent to the standard cladding diameter (125µm) of the currently widely used SMF. When using
multiple cores and propagation modes to transmit
signals using SDM optical fiber with the standard
cladding diameter, especially in areas with more thanfour space multiplexes, strong mode couplings
between each spatial mode occur, and cross talk andspatial mode dispersion (SMD) lead to significant
distortion of the signal waveform. In the current digital coherent optical transmission system using SMF,
polarization multiplexing is applied to transmit independent signals using two different polarizations. The
dynamic waveform distortion caused by the combination of polarization rotation in the transmission
path and the delay difference between the polarizedwaves (polarization mode dispersion) is adaptively
compensated by 2 × 2 multiple-input and multipleoutput (MIMO) signal processing*4 implemented in
the digital signal processing circuitry in the receiverto achieve high-quality transmission. However, if thisis simply expanded and applied to the mode-multiplexed SDM transmission system, the size of the
MIMO signal processing circuit increases proportionately to the square of the spatial multiplex numNTT Technical Review

Vol. 20 No. 7 July 2022

ber. Furthermore, the number of digital filter taps
required for MIMO signal processing must be
expanded accordingly since the SMD inherent in
multi-mode fiber is 10 times larger than polarization
mode dispersion and accumulates proportionally tothe square root of the transmission distance in multimode fibers with large mode coupling.
To overcome the above technical challenges, we areinvestigating high-capacity, long-distance opticaltransport technologies that actively use and control
the spatial modes (Fig. 3). Specifically, we are aiming
to establish: (1) spatial mode-control optical-fiber
cabling technology with a standard cladding diameterof 125 µm suitable for fiber-optic cable installation
environments and mass production, (2) mode-multiplexing MIMO processing configuration technologythat takes into account the dynamic optical characteristics attributed to the cable installation properties,and (3) the fundamental technology that organically*4 MIMO signal processing: A technology that transmits and receives one or more signals on a transmission path with multiple
signal propagation paths (propagation modes and cores) using thesame carrier frequency (wavelength). It is a widely used technology in radio communications. In optical communications, MIMO
with two inputs and two outputs (2 × 2) using two orthogonal polarization modes within SMF has been commercialized using
digital coherent technology as a polarization multiplexing technology.

27

Feature Articles

links the spatial mode-multiplexing optical amplification repeater technology integrating (1) and (2). As
an example of the results of recent studies on modemultiplexing MIMO processing configuration technology [5], in the mode-multiplexing optical communications using six independent spatial modes, wehave successfully demonstrated long-distance transmission over 6000 km by proposing an optical amplification repeater system and MIMO signal processing
system that have strong compensation characteristics
against transmission-loss and propagation-delay differences between different spatial modes. We havealso successfully demonstrated the effectiveness of anovel implementation technology for controlling
optical characteristics in mode-multiplexing transmission fiber in current terrestrial optical fiber cablestructures [6]. To establish these fundamental technologies, we are accelerating R&D in cooperation
with external partners, which is partially supportedby the National Institute of Information and Communications Technology [7].
4. Summary
In this article, we described the current state andfuture prospects for broadband optical parametricamplification repeater technology and mode-multiplexed SDM optical communication technology thatare being studied as scalable optical transport technologies to overcome the capacity crunch toward the
IOWN APN.

NTT Technical Review

Vol. 20 No. 7 July 2022

References
[1]

[2]

[3]

[4]

[5]

[6]

[7]

Press release issued by NTT Communications on Dec. 9, 2019 (inJapanese).
https://www.ntt.com/about-us/press-releases/news/article/2019/1209.
html
T. Umeki, T. Kazama, T. Kobayashi, K. Enbutsu, R. Kasahara, and Y.
Miyamoto, “Low-noise Amplification and Nonlinearity Mitigation
Based on Parametric Repeater Technology,” NTT Technical Review,Vol. 17, No. 5, pp. 20–26, 2019.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201905fa3.html
Press release issued by NTT, “World’s First Experimental Long-distance WDM Optical Transmission at 1 Tbps per Wavelength—
Expected to Provide Large-capacity Network Communications for
IoT and 5G Services,” Mar. 7, 2019.
https://group.ntt/en/newsrelease/2019/03/07/190307a.html
T. Kobayashi, S. Shimizu, M. Nakamura, T. Umeki, T. Kazama, R.
Kasahara, F. Hamaoka, M. Nagatani, H. Yamazaki, H. Nosaka, and Y.
Miyamoto, “Wide-band Inline-amplified WDM Transmission Using
PPLN-based Optical Parametric Amplifier,” IEEE J. Lightwave Technol., Vol. 39, No. 3, pp. 787–794, 2021.
Press release issued by NTT, “World’s First Transpacific-class Transmission Experiment Employing Mode-multiplexed Optical Signals—
Key Achievement for Future High-capacity Optical Transport Network Infrastructure Supporting the Spread of 5G and IoT Services,”
Mar. 9, 2020.
https://group.ntt/en/newsrelease/2020/03/09/200309b.html
Press release issued by NTT, “World’s First Attempt at Optical Transmission Property Control by Optimizing Optical Cable Structure—
Milestone for Realizing Long-haul and Large-capacity Mode-multiplexing Optical Transmission Links,” Mar. 9, 2020.
https://group.ntt/en/newsrelease/2020/03/09/200309a.html
Y. Miyamoto, K. Nakajima, T. Hasegawa, E. L. T. de Gabory, R.
Sugizaki, and R. Nasase, “R&D of Spatial-mode-controllable Optical
Transmission System,” IEICE General Conference, Planning Symposium, BI-3-1, Mar. 2022 (in Japanese).

28

Feature Articles

Yutaka Miyamoto

NTT Fellow, Director, Innovative Photonic
Network Research Center, NTT Network Innovation Laboratories.
He received a B.E. and M.E. in electrical engineering from Waseda University, Tokyo, in 1986
and 1988 and Dr.Eng. in electrical engineeringfrom the University of Tokyo in 2016. He joined
NTT Transmission Systems Laboratories in
1988, where he engaged in R&D on high-speed
optical communications systems including the
10-Gbit/s first terrestrial optical transmission
system (FA-10G) using EDFA inline repeaters.
He was with NTT Electronics Technology Corporation between 1995 and 1997, where heengaged in the planning and product development of a high-speed optical module at the data
rate of 10 Gbit/s and beyond. Since 1997, he has
been with NTT Network Innovation Labs, where
he has researched and developed optical transport technologies based on 40/100/400-Gbit/schannel and beyond. He is currently investigating
and promoting the future scalable optical transport network with Pbit/s-class capacity. He is amember of the Institute of Electrical and Electronics Engineers (IEEE) and a fellow of theInstitute of Electronics, Information and Communication Engineers (IEICE).

Kazuhide Nakajima

Senior Distinguished Researcher, NTT AccessNetwork Service Systems Laboratories.
He received an M.S. and Ph.D. in electricalengineering from Nihon University, Chiba, in
1994 and 2005. In 1994, he joined NTT AccessNetwork Service Systems Laboratories, where
he has been engaged in research on optical-fiber
design and related measurement techniques. He
has been acting as a rapporteur of Question 5 of
the ITU-T (International Telecommunication
Union - Telecommunication Standardization
Sector) Study Group 15 since 2009. He is a member of IEEE, Optica (formerly OSA), IEICE, andJSAP.

Kei Watanabe

Senior Research Engineer, HeterogeneousMaterials and Devices Research Group, NTT
Device Technology Laboratories.
He received a B.E. in electrical and electronics
engineering and M.E. and Dr.Eng. in mathematical and material science from Kobe University,
Hyogo, in 1998, 2000, and 2003. In 2003, hejoined NTT Photonics Laboratories, where heengaged in research on silica-based optical waveguides. From 2008 to 2009, he was a visiting
researcher at Ghent University, Belgium, where
he engaged in research on silicon photonics.
After returning to NTT, he has been engaged inresearch on indium phosphide high-speed modulators and nonlinear optical devices based on
PPLN waveguides. He is a member of the Japan
Society of Applied Physics (JSAP), IEICE, andthe IEEE Photonics Society.

NTT Technical Review

Vol. 20 No. 7 July 2022

29

Feature Articles: Optical and Wireless Transmission
Technologies for IOWN/6G

R&D Activities of Core Wireless
Technologies toward 6G Radio Access
Jun Mashino, Yosuke Fujino, and Riichi Kudo
Abstract

Technical research for the 6th-generation mobile communication system (6G) expected to be implemented in the 2030s has been advanced to achieve extreme high data rate/capacity, extreme low latency,
extreme high reliability, and extreme coverage extension for non-terrestrial areas. In this article, we
introduce the following core wireless technologies developed by NTT Network Innovation Laboratories
toward 6G, i.e., orbital angular momentum multiple-input multiple-output (OAM-MIMO) multiplexing
transmission, underwater acoustic communication, and wireless-link-quality prediction.
Keywords: OAM multiplexing, underwater acoustic communication, wireless-link-quality prediction

1. Introduction
Mobile communication systems, which are now
essential for our daily lives, have evolved once every
ten years. The 5th-generation mobile communication
system (5G) service has been provided since 2020 inJapan, and much research for 6G, expected to be
implemented in the 2030s, has been conductedaround the world. The vision with mobile communication systems has been providing mobile multimedia functions including displaying websites, movie
content, and reinforcement of application software in
the 3G/4G era. The vision in the 5G/6G era will be
expected to take on a role as a society infrastructurethat can address various social issues and support
diverse industries.
Use cases, target performance, and technology candidates expected for 5G evolution and 6G are
described in NTT DOCOMO’s 6G White Paper [1],including the concept of 6G combined with innovative networking and information technologies for theInnovative Optical and Wireless Network (IOWN)
led by the NTT Group. Technical development in 6G
addresses not only basic performance enhancement(e.g., extreme high data rate/capacity, extreme low
latency, extreme high reliability, and extreme massive
connectivity) but also new challenges including
extreme coverage extension to the sky, sea, and space
NTT Technical Review

Vol. 20 No. 7 July 2022

(Fig. 1). In this article, we introduce the following
three core wireless technologies developed by NTT
Network Innovation Laboratories toward 6G, i.e.,
orbital angular momentum multiple-input multipleoutput (OAM-MIMO) multiplexing transmission,underwater acoustic communication, and wirelesslink-quality prediction.
2. OAM-MIMO multiplexing transmission forterabit-class wireless communication
The benefit of 5G is high-speed transmission byintroducing wide bandwidth in the high-frequency
band called millimeter waves, which is being used in
mobile communication systems for the first time. The
final target in 5G radio access is 20-Gbit/s transmission in future evolution, and 6G radio access is
expected to reach 100-Gbit/s transmission by exploring a higher-frequency band known as the sub-THz
band to obtain more radio resources. However, a highdensity of base-station antennas has to be deployedcompared with current systems due to the problemsof straightness of radio waves and coverage holes dueto blocking in such a higher-frequency band. Therefore, a flexible network and easy installability are
required for xHaul* connecting between base-stationequipment. If wireless high-capacity xHaul isachieved, it will be possible to provide a highly dense30Feature Articles

Source: DOCOMO 6G White Paper [1]E2E: end to end
eMBB: enhanced mobile broadband
mMTC: massive machine-type communications

QoS: quality of service
URLLC: ultra-reliable and low latency communications

Fig. 1. Requirements for 6G wireless technology.

deployment solution of base-station equipment inareas where there is difficulty in optical wiring or
temporary expansion of base-station equipment.
However, wireless xHaul for 6G requires an extremely high capacity of over 1 Tbit/s (terabit-class) transmission in consideration of radio access evolution
described above, function split between base-stationequipment, and accommodation of multiple types ofbase-station equipment (Fig. 2).
NTT Network Innovation Laboratories has conducted advanced research on OAM multiplexing
technology. OAM is a physical property of electromagnetic waves characterized by a helical phase front
in the propagation direction. Since these characteristics can be used to create multiple orthogonal channels, wireless communication using OAM canincrease radio-spectrum efficiency. Given an electromagnetic wave having this OAM property, the traceof the same phase takes on a helical shape in the
propagation direction. Different data signals can betransmitted simultaneously by transmitting different
signals using radio waves having different OAM
modes. Since radio waves having this OAM property
cannot be received without a receiver having the same
number of phase rotations at the time of transmission,
multiple radio waves having different OAM modes caneventually be separated without mutual interference.
OAM-MIMO multiplexing transmission, originally
NTT Technical Review

Vol. 20 No. 7 July 2022

proposed by NTT, uses coaxial uniform circular array(UCA) antennas for OAM transmission and reception
in consideration of practical implementation. We
combined OAM and existing MIMO technology bytreating UCAs with different radii as sub-array antennas for MIMO transmission so that an extreme
amount of spatial multiplexing can be achieved while
preventing the use of higher-order OAM modes,
which is inappropriate for long-distance transmission(Fig. 3). Load reduction for baseband digital signal
processing is one of the advantages of OAM-MIMO
since signal processing can be separated into analog
passive circuits for OAM and digital signal processing for MIMO. This contributes to low-energy consumption and achieving transmission with muchwider bandwidth.
An indoor experimental trial was conducted in the
28-GHz band to challenge the limit of spatial multiplexing. The results indicate that OAM-MIMO canmultiplex 21 spatial streams in combination withvertical/horizontal polarization and achieve highcapacity transmission of 201.5 Gbit/s [2] (Fig. 4). An
outdoor experimental trial was also conducted in the
40-GHz band to prove the feasibility of long-distance*

xHaul: Unified transport network infrastructure connecting between base-station equipment, e.g., fronthaul, midhaul, and backhaul.

31

Feature Articles

Distributedunit

High-capacity
wireless transmission

High-capacity
optical transmission
Front haul
(Terabit-class)Radio unit

Radio unit

(100 Gbit/s)

Radio accessareas(100 Gbit/s)• Complement of the opticaltransmission on specific occasions• Temporal extension of basestation equipment

Fig. 2. An example of xHaul for base-station equipment.

Multiplex streams by using
orthogonality between OAM modes

OAM-MIMO
multiplexed signals

Set of OAM
multiplexed signals

MIMO multiplexing

Fig. 3. Concept of OAM-MIMO multiplexing transmission.

MIMO
multiplexing

Antennas fortransmitter/receiver

Prototype transceiver

Vertical polarization
Horizontal polarization

Simultaneoustransmission

UCA #4

17

18

19

20

21

UCA #3

12

13

14

15

16

UCA #2

7

8

9

10

11

UCA #1

2

3

4

5

6

−2

−1

0
+1
OAM mode+2

CenterOAM
multiplexing

1

Fig. 4. Transceiver antennas and assignment of 21 spatial streams.

NTT Technical Review

Vol. 20 No. 7 July 2022

32

Feature Articles

Spatio-temporal equalization

Distortion-compensated signal

antenna #NPowerantenna #1

PowerSpatially removing delayed waves that cause waveform distortion
Direct path

Delayed path
Delayed path

ReceiverTime
Direct path

Delayed path
Delayed path

Extraction

RemovalFIR filter
Time

FIR filter

Delayed path
Direct path

Signal distortion caused
by multipath waves

Transmitter
Delayed path

FIR: finite impulse response

Fig. 5. Concept of spatio-temporal equalization.

transmission. It was successful in 117-Gbit/s transmission for 200 m [3]. We are now pushing forwardwith our research to achieve terabit-class wirelesscommunication required for xHaul in 6G by expanding the bandwidth up to 10 GHz and combining itwith 20–40 spatial streams in advanced OAM-MIMO
technologies.
3. Underwater acoustic communication toextend coverage into the ocean
The development of offshore resources, such as oil
and gas fields, has been growing, and remotely operated vehicles (ROVs) are used for offshore development. Unmanned remote construction using ROVs is
also being researched to improve worker safety andoperational efficiency in port and harbor construction.
Since the Mbit/s-class underwater communication
capable of transmitting high-definition video has not
yet been commercialized, these ROVs are connected
to support vessels on the sea by long wired cables.
The main drawback of wire-controlled ROVs is theirhigh operational cost due to large support vessels anddedicated operators to hoist the long and heavy
cables. Therefore, Mbit/s-class high-speed underwatercommunication that enables wireless remotely conNTT Technical Review

Vol. 20 No. 7 July 2022

trolled ROVs is highly desired. Underwater communication using various media such as radio waves, soundwaves, and light have been studied. We have focusedon sound waves, which are suitable for stable longdistance communication, and are working to improve
the speed of underwater acoustic communication.
Not only in underwater acoustic communication but
in any wireless communication, waveform distortion
occurs as a result of combining many waves from different paths. Therefore, wireless communication
requires equalization to compensate for this waveform
distortion. Due to the extremely slow propagation
speed of sound waves compared with radio waves,waveform distortion drastically changes in underwater
acoustic communication, which makes equalization
more complicated. The conventional equalization
technique, which compensates for the inverse responseof waveform distortion, cannot track fast fluctuations
in waveform distortion of underwater acoustic communication because the inverse response cannot be
estimated in time. As a result, the communication
speed of conventional underwater acoustic communication has been limited to several tens of kbit/s [4].
To overcome the fast fluctuations in waveform distortion, we proposed an equalization technique called
spatio-temporal equalization [5] (Fig. 5). This technique33

Feature Articles

1. Physical-space-information acquisition (Camera image)Wireless communication device

2.
Objectdetection
Camera images

3.
Link-quality
prediction

Predicted
link quality

Object information

Fig. 6. Link-quality prediction using camera images.

achieves equalization by spatially removing thedelayed waves that cause waveform distortion using
adaptive beamforming. Spatio-temporal equalization
does not require the estimation of the inverse responseof waveform distortion, enabling Mbit/s-class highspeed acoustic communication.
We conducted transmission experiments in theocean to verify the effectiveness of this technology
and succeeded in transmitting 5.12 Mbit/s at a distance of 18 m and 1.2 Mbit/s at a distance of 60 m [6].
We are currently working on achieving more than 1
Mbit/s at a distance of 300 m by combining bandwidth-division transmission technology, which uses
multiple transmitters, with different resonance frequencies to transmit broadband signals.
4. Wireless-link-quality-prediction technologies
using physical space information
In the 6G era, the use of physical space information
will be more familiar because of various activities forSociety 5.0 [7], which was proposed as a future society that Japan should aspire to. The huge amount ofphysical space information will be stored in cyber
space and become more accessible from everywhere.
NTT Technical Review

Vol. 20 No. 7 July 2022

NTT Network Innovation Laboratories started toresearch the physical space information use to promote the evolution to wireless communication systems [8, 9]. The throughput and capacity in wirelesscommunication systems have been basically enhanced
using wireless system information and settings. The
requirements for wireless communication systems
are diversified and advanced because of emerging
services. This raises the expectation of highly reliablewireless accesses. Physical space information use is
one of the keys to evolve wireless communication
systems since the long-term movement of communication devices and surrounding objects can beextracted from the physical space information. Longterm prediction enables proactive actions with sufficient time to prepare the advanced controls anddevelop the roles of wireless communication systems.
Throughput prediction using camera images isshown in Fig. 6 as an example of our work. Cameras
first obtain images including communication device.
A device is then detected from the obtained imagesby using object-detection algorithms with a pre-learned
model. Finally, the wireless-link-quality-prediction
model predicts link quality by using the detected34

Feature Articles

object information. The exteriors of communication
devices are learned using object-detection algorithms
that are based on deep learning, and the detected position and movements of communication devices on
the image are used to predict future link quality. Sincea large number of cameras and sensors are expected
to be connected to the IOWN All-Photonics Network,
sustainable system design is also being researched toenable the addition and deletion of cameras and sensors for prediction systems. The artificial-intelligence-driven experimental rooms in which multimodal features including physical space information
and wireless system parameters are automaticallymeasured are being developed to evaluate the performances of various machine learning technologies
using big data measured in an experimental environment. The accurate prediction in the short and longterm enables proactive controls, such as switching tobetter wireless connections, to change the application
data rate or control the movement of communication
devices and surrounding objects by detecting the link
disconnection beforehand. We believe that physicalspace-information-based wireless-link-quality prediction will extend the use cases of reliable wirelesscommunication.

NTT Technical Review

Vol. 20 No. 7 July 2022

References
[1]
[2]

[3]

[4]
[5]
[6]

[7]
[8]

[9]

NTT DOCOMO, “DOCOMO 6G White Paper,”https://www.docomo.ne.jp/english/corporate/technology/
whitepaper_6g/
Y. Yagi, H. Sasaki, T. Yamada, and D. Lee, “200 Gbit/s WirelessTransmission Using Dual-polarized OAM-MIMO Multiplexing on 28
GHz Band,” Proc. of IEEE Globecom Workshops on HCPtP, Waikoloa, USA, Dec. 2019.
Y. Yagi, H. Sasaki, T. Semoto, T. Kageyama, T. Yamada, J. Mashino,
and D. Lee, “Field Experiment of 117 Gbit/s Wireless Transmission
Using OAM Multiplexing at a Distance of 200 m on a 40 GHz Band,”Proc. of IEEE International Conference on Communications Workshops, Montreal, Canada, June 2021.
S. Sendra, J. Lloret, J. M. Jimenez, and L. Parra, “Underwater Acoustic Modems,” IEEE Sens. J., Vol. 16, No. 11, pp. 4063–4071, June
2016.
Y. Fujino and H. Fukumoto, “Spatio-temporal Equalization Technology for High-speed Underwater Acoustic Communications,” IEICE
Commun. Mag., Vol. 60, pp. 284–297, Mar. 2022 (in Japanese).
H. Fukumoto, Y. Fujino, M. Nakano, K. Sakamoto, and T. Tsubaki,“Field Experiments Demonstrating Mbps-class Underwater Acoustic
Communication with Spatio-temporal Equalization,” Proc. of IEEE
Global OCEANS 2020, Biloxi, USA, Oct. 2020.
Cabinet Office, Government of Japan, “Society 5.0,” https://www8.
cao.go.jp/cstp/english/society5_0/index.html
R. Kudo, M. Cochrane, K. Takahashi, T. Inoue, and K. Mizuno,“Experimental Validation of Link Quality Prediction Using Exact
Self-status of Mobility Robots in Wireless LAN Systems,” IEICE
Trans. Commun., Vol. E103-B, No. 12, pp. 1385–1393, July 2020.
R. Kudo, K. Takahashi, T. Inoue, and K. Mizuno, “Using Vision-based
Object Detection for Link Quality Prediction in 5.6-GHz Channel,”
EURASIP J. Wireless Com. Network., 207, Oct. 2020.

35

Feature Articles

Jun Mashino

Senior Research Engineer, NTT Network Innovation Laboratories.
He received a B.E. in electrical and electronicengineering and M.E. in communications andcomputer engineering from Kyoto University in2003 and 2005. He joined NTT Access Network
Service Systems Laboratories in 2005, where hehas been engaged in the research and development of intelligent interference compensation
technologies and signal processing for future
wireless communication systems. From 2016 to
2019, he engineered uplink massive MIMO technology for 5G as a senior research engineer at
NTT DOCOMO 5G Laboratory and led proof ofconcept projects for social implement of 5G.
Since 2019, he has been a senior research engineer at NTT Network Innovation Laboratories.
His current research interests are in high-capacity wireless transmission technologies including
OAM-MIMO toward 6G.

Riichi Kudo

Senior Research Engineer, Supervisor, NTT
Network Innovation Laboratories.
He received a B.S. and M.S. in geophysics
from Tohoku University, Miyagi, in 2001 and2003 and Ph.D. in informatics from Kyoto University in 2010. In 2003, he joined NTT Network
Innovation Laboratories. He was a visiting fellow
at the Center for Communications Research(CCR), Bristol University, UK, from 2012 to2013, and worked for NTT DOCOMO from2015 to 2018. He is now working for NTT Network Innovation Laboratories.

Yosuke Fujino

Senior Research Engineer, NTT Network Innovation Laboratories.
He received a B.E. and M.E. in electrical andelectronic engineering from Shizuoka University
in 2002 and 2004. He joined NTT Network Innovation Laboratories in 2004 and engaged inresearch on physical layer signal design, transceiver architecture, and signal processing forInternet of Things wireless systems. The current
focus of his research is wireless communication
technology in non-terrestrial areas such as underwater, underground, and space.

NTT Technical Review

Vol. 20 No. 7 July 2022

36

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Overview of Technical Development
and Verification in the Connectedvehicle Field
Atsushi Koizumi and Shin Mitsuhashi
Abstract

As technological innovations, such as connectivity, automation, sharing, and electrification, bringmajor changes to the automotive industry, expectations for and the importance of information and communication technology (ICT) are increasing rapidly in the industry. The NTT Group and Toyota Motor
Corporation are collaborating on the research and development of an ICT platform for connected vehicles. They conducted joint field trials from 2018 to 2020 and established the basic technology throughvarious use cases and verification of the platform. In the Feature Articles in this issue, the NTT Group
operating companies and NTT laboratories that are participating in the collaboration present the detailsof the field trials, the results, technologies applied, value provided, and future issues.
Keywords: connected vehicle, IoT, big data

1. Overview of connected vehicles
A connected vehicle, which is a mobility device
equipped with communication capability, consists ofvarious components, including a vehicle, communication networks (wired and wireless), and cloud computing (Fig. 1). To implement these components, it isnecessary to develop, use, and combine a wide rangeof technologies, including networks for exchanging
data, edge computing for data processing in the proximity of connected vehicles, a platform for storingand processing collected data, platform for analyzing
and using data, and software updates.
As the potential for using big data expands, thevolume of data to handle is expected to increase dramatically. Therefore, information and communication technology (ICT) platforms, such as networks
and datacenters, that receive data from connectedvehicles are growing in importance.
2. Potential of vehicle big data
Connected vehicles are already in use, and the market for them will continue to expand. If the large
NTT Technical Review

Vol. 20 No. 7 July 2022

amount of data held by connected vehicles can beprocessed quickly and at low cost, it will become possible to use information that cannot be captured withcurrent sensors and provide faster and more reliable
services. This will in turn not only improve convenience and efficiency but also enable people to drive
more safely and securely, alleviate or eliminate traffic
congestion, and shorten travel time, which will contribute to achieving carbon neutrality (Fig. 2).
3. Purpose and areas of joint researchand development
In March 2017, the NTT Group and Toyota Motor
Corporation agreed to collaborate in developing,verifying, and standardizing technologies needed in
the connected-vehicle field by combining Toyota’s
vehicle-related technologies and NTT Group companies’ ICT-related technologies.
By sharing the technologies and expertise of eachcompany and using big data obtained from vehicles,
the NTT Group and Toyota will work together toresearch and develop the technologies needed tosolve problems facing society, such as traffic accidents
37

Feature Articles

Vehicle big data

Data analysis andutilization platform
Data storage andprocessing platform

Cloud

Networking
Edge computing

OTA(Software update)
OTA: over the air programming

Fig. 1. Overview of a connected vehicle.

New mobility
services

Solving social
issuesCar sharing/ride sharing
Remote vehicle inspection service
Telematics insurance

Safe travel (zero accidents, zero
traffic congestion)
Contribution to carbon neutrality
Contribution to reducing damage
caused by natural disasters

Fig. 2. Value provided by vehicle data.

and congestion, and provide new mobility services tocustomers. The collaboration is aimed at creating asustainable smart mobility society from a global perspective.
The collaboration activities can be broadly divided
into the following three technical areas:(1)	Data collection/storage/analysis platform:
Establish a mechanism for storing and analyzing data sent from millions or tens of millions
of vehicles.
(2)	Internet of Things (IoT) network and datacenter: Establish an optimal arrangement of networks and datacenters for collecting data fromvehicles around the world.
NTT Technical Review

Vol. 20 No. 7 July 2022(3)	Next-generation communication technology:
Verify technology for vehicles to use the 5thgeneration mobile communication system(5G) and verify the applicability of edge computing.
In December 2018, we set the goal of processing alarge volume of data from within and outside vehicles
to reproduce the physical space in the real world in
real time (within seconds) and with a precision of tensof centimeters. We carried out field trials that covered
the end-to-end mobility system including vehicles,
networks, and datacenters.

38

Feature Articles

Data collection, storage,
and analysis platform

Datacenter

Network

Edge computing

5G/LTE

Connected vehicle

Fig. 3. Overview of the ICT platform for connected vehicles.

4. ICT platform for connected vehicles
The ICT platform for connected vehicles connects
vehicles equipped with communication capability
with edge computing nodes and the cloud throughLTE (Long-Term Evolution), 5G, and IoT networks.
It also uses computing resources to collect, store, andanalyze data that are held or collected by vehicles(Fig. 3).
Data from connected vehicles are uploaded to datacenters through the mobile network and edge nodes.
The datacenter then collects, stores, and analyzes thedata and returns the analysis results to vehicles
through the network as necessary. Although this data
flow is not much different from that of smartphones
and small IoT devices, a mechanism for real-time,large-scale, and accurate processing of such data will
become one of the important infrastructures for supporting people’s daily lives and society at large.
5. Method of conducting joint researchand development
The joint research and development was conductedin two alternating activities: technical studies by aworking group and verification of its study results
using actual vehicles (Fig. 4).
In the working group weekly meetings, engineersfrom the two parties discussed multiple themes inparallel. The working group’s discussion results wereverified using a testbed that involved more than a
NTT Technical Review

Vol. 20 No. 7 July 2022

hundred physical servers, 5G, other communication
links, and actual vehicles. The verification results
were then fed back to the working group. This cycle
was then repeated over a short period. The working
group members were proud of the fact that the twoparties grew together by bringing together their technologies and expertise and by repeating the above
cycle, therefore quickly solved various technicalissues.
6. Activities in the collaboration onconnected vehicles
The field trials of the ICT platform for connectedvehicles were conducted from FY2018 to FY2020.
The following two articles in this issue give an overview and the details of the field trials from the perspectives of NTT DATA and NTT Communications,which played a central role in the trials. These articles
also introduce the technical results and achievements
obtained and future challenges identified in the trials.
(1)	Activities and results of field trials—referencearchitecture for a connected-vehicle platform:
The overall architecture and verification of theplatform through use cases and verification
results are presented [2].
(2)	Activities and results of field trials—networkedge computing platform: An overview, thefeatures, and methods of the platform are presented [3].
The ICT platform for connected vehicles involves a39

Feature Articles

Verification
using actual vehicles

Technical study
working group

Large-scale testbed for verification with
Weekly meetings with tens of engineers
NTT Digital Twin Computing Research Center actual connected vehicles
NTT Smart Data Science Center
NTT Communications
100+ server environment
NTT DOCOMONTT DATAReal network
Repeat the cycle in/5G verification
Toyota Motor
environmenta short period
Denso
Toyota Tsusho
Toyota Mapmaster
Actual vehicle
Toyota Systems

Fig. 4. Method of conducting the joint research and development.

variety of technical challenges. The following fivearticles in this issue focus on representative technologies that have been studied and developed to solvethese challenges and provide overviews of these technologies and value they provide, including verification through use cases and field trials, and identify
remaining challenges.
(3)	Real-time spatiotemporal data-management
technology (Axispot TM): This technology
stores, searches, and analyzes large volumesof data on dynamic objects [4].
(4)	Selective vehicle-data-collection algorithm:
This technology determines data-collection
priorities on the basis of meta-information,such as vehicle positions and time, observation range, and the effects of shielding by surrounding vehicles [5].
(5)	Vertically distributed computing technology:
This technology effectively uses limited server resources by dynamically changing the
response-processing server on the basis ofvehicle status [6].
(6)	Lane-specific traffic-jam-detection technology: This technology detects lane-specific traffic jams by collecting and analyzing dashcam
video and driving data to achieve optimal lanenavigation [7].
(7)	Technology for calculating suddenness index
for aggregated values: To reduce the amountof processing on the ICT platform, this method calculates an index for periodic and suddenchanges in aggregated values collected fromvehicles on the basis of the degree of deviationfrom the ordinary state [8].

7. Results and future outlook“The Technical Document on the ICT Platform forConnected Vehicles” was compiled and published in
November 2021 to report on the results and challenges of the three-year field trials [9]. We are hopingthat the results of our activities will be widely used
not only by those in the ICT and mobility industries
but also by those in other industries.
The NTT Group and Toyota will continue toimprove the speed, efficiency, and sophistication of
the ICT platform for connected vehicles in preparation for the further spread of such vehicles. We will
also continue to develop technologies that will contribute to solving problems facing society, such as
traffic accidents and congestion, by effectively using
big data collected from vehicles and by creating value
from such data.
We will also use the obtained technical results toplan social implementation of the ICT platform forconnected vehicles and the deployment of the verified technologies in smart cities. We will develop
technologies for providing high-speed, high-capacity
communications and vast computing resources
beyond the limits of the conventional platform andcollaborate with various companies, organizations,and service providers to create and provide newmobility services, thereby contributing to achievingcarbon neutrality and creating a sustainable smart
mobility society that brings safety and security topeople.
References[1]

NTT Technical Review

Vol. 20 No. 7 July 2022

Press release issued by Toyota and NTT, “Toyota and NTT Agree to40Feature Articles

[2]

[3]

[4]

[5]

Collaborate on ICT Platform R&D for Connected Cars Collaboration
Seeks to Establish Technologies for the Creation of a Smart Mobility
Society,” Mar. 27, 2017.
https://group.ntt/en/newsrelease/2017/03/27/170327a.html
Y. Chiba, “Activities and Results of Field Trials—Reference Architecture for a Connected-vehicle Platform,” NTT Technical Review, Vol.
20, No. 7, pp. 42–47, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa6.html
R. Noji, T. Kamei, K. Funabiki, D. Murata, and Y. Kanemaru, “Activities and Results of Field Trials—Network Edge Computing Platform,” NTT Technical Review, Vol. 20, No. 7, pp. 48–53, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa7.html
A. Isomura, N. Shigematsu, I. Ueno, N. Oki, and Y. Arakawa, “Realtime Spatiotemporal Data-management Technology (AxispotTM),”NTT Technical Review, Vol. 20, No. 7, pp. 54–60, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa8.html
M. Takagi, K. Matsuo, R. Nakada, and K. Mori, “Selective Vehicledata-collection Algorithm,” NTT Technical Review, Vol. 20, No. 7,

Atsushi Koizumi

Senior Manager, R&D Produce Group, NTT
Research and Development Planning Department.
He received a B.A. in commerce from Waseda
University, Tokyo, in 1995. After working on
system development in the public sector at NTT
DATA, he was seconded to the NTT DATA Institute of Management Consulting Inc. After returning to NTT DATA, he worked on sales reform
and marketing in the Corporate Planning Department then engaged in business planning in theorganization dedicated to business process outsourcing and robotic process automation. Currently, he is at NTT and engaged in the researchand development of connected vehicles.

NTT Technical Review

Vol. 20 No. 7 July 2022

[6]

[7]

[8]

[9]

pp. 61–67, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa9.html
K. Matsuo, M. Takagi, R. Nakada, and K. Mori, “Vertically Distributed Computing Technology,” NTT Technical Review, Vol. 20, No. 7,
pp. 68–75, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa10.html
K. Mori, Y. Yokohata, A. Hayashi, T. Hata, and M. Kamiya, “Lanespecific Traffic-jam-detection Technology,” NTT Technical Review,Vol. 20, No. 7, pp. 76–82, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa11.html
A. Hayashi, Y. Yokohata, T. Hata, K. Mori, and M. Kamiya, “Technology for Calculating Suddenness Index for Aggregated Values,” NTT
Technical Review, Vol. 20, No. 7, pp. 83–88, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa12.html
NTT Topics, “Tech Document on ICT Platform for Connected Vehicles Released by NTT and Toyota,” Nov. 29, 2021.
https://group.ntt/en/topics/2021/11/29/techdoc_rel_ntt_toyota.html

Shin Mitsuhashi

Senior Manager, Transport and Mobility Business Producer, Strategic Business Creation
Team, NTT Research and Development Planning
Department.
He is currently engaged in creating mobilityrelated services.

41

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Activities and Results of Field
Trials—Reference Architecture for aConnected-vehicle Platform
Yu Chiba
Abstract

The NTT Group and Toyota Motor Corporation are collaborating on research and development of an
information and communication technology platform for connected vehicles. They conducted joint fieldtrials and verified the platform across a variety of use cases from 2018 to 2020. They also establishedbasic technologies in the course of these trials. This article presents an overview of the reference architecture for the connected-vehicle platform, which collects, stores, and uses controller area network data(vehicle control data) and image data sent from in-vehicle devices. It also reports on the technical results
obtained and challenges identified during the implementation of the platform and the field trials.
Keywords: connected vehicles, IoT, big data

1. Characteristics of connected vehicles andtechnical challenges they pose
As the number of connected vehicles grows rapidly,
the amount of data obtained from them, such as controller area network (CAN) data, sensor data, andimage data, is growing dramatically. How to process
this enormous amount of data efficiently and in realtime presents a major technical challenge for a largescale connected-vehicle platform. Connected vehicles also have several unique characteristics: theyneed to use a mobile network for communication;
move at high speed; have a long life cycle; and theamount of data to be handled varies greatly from hour
to hour even within a single day. Therefore, using
connected-vehicle data across a variety of use cases
poses many complex technical challenges.
The NTT Group and Toyota Motor Corporation arecollaborating on research and development of an
information and communication technology platform
for connected vehicles. In conducting field trials toverify this platform, we set our goals from the perspectives of three particularly important component
technologies: the processing of a large amount of
data, real-time performance of the processing of such
NTT Technical Review

Vol. 20 No. 7 July 2022

data, and the degree of precision in the data processing. We have thus studied a connected-vehicle platform that can work across various use cases of connected vehicles and automated driving (Fig. 1).
2. Use cases for the field trials
The field trials were conducted using test vehicles
on public roads for three years from 2018 to 2020.
The main objective of these trials was to establish thetechnologies for and evaluate the performance of alarge-scale platform that will be able to handle millions of connected vehicles in the future. We verified
the platform using the following three sample use
cases, which are likely to be put into practice (Fig. 2).
(1)	Generation of a static map: The datacenter
analyzes vehicle-location data and image data
sent from connected vehicles and generates ahigh-precision static map required for automated driving.
(2)	Obstacle detection and notification to following vehicles: Dangerous obstacles, such as
falling rocks on the road, are detected using
image data from cameras mounted on connected vehicles. The datacenter manages this42

Feature Articles

Verification classification
Use
case

Component
technologies

Technical-development subject

Goal

Architecture

End-to-end architecture that collects datafrom connected vehicles at the datacenter,
processes it, and returns analysis results

Operate asa system

Number ofconnected
vehicles (datavolume)Scalable distributed processing technologythat can simultaneously collect and store
large amounts of vehicle and image data

30 million
vehiclesReal-time
performance

Component technology that collects, stores,
and distributes information about the locationsof fast-running vehicles, etc., in real time

Precision

Technology to precisely estimate thelocations of vehicles, signs, traffic signals,
etc., on the basis of image data

7s

10 cm

Fig. 1. Technical items and target values.

Data collection(from vehicles)

Understanding of current status and prediction(example)

Data on internal
status of vehicle• CAN and GPS data
Data on monitored
surroundings• Video and sensor data

(1) Generation of (2) Obstacle detection anda static map notification to following vehicles

Data collection (from Data processing
external sources) Data integration(3) Lane-specific congestion detection

Detailed road information• High-precision maps
Road traffic conditions• Traffic jams
• Road constructions• Accidents

GPS: Global Positioning System

Fig. 2. Sample use cases.

information and notifies the following vehicles of danger.
(3)	Lane-specific congestion detection: Bothrecurring and non-recurring traffic jams oneach lane are detected using statistical information in CAN data and real-time image data.
The cause of a non-recurring traffic jam is
identified by analyzing images of the front
point of the traffic jam.
We evaluated the feasibility of these three use cases
through the field trials. Among the component technologies needed for a large-scale connected-vehicle
NTT Technical Review

Vol. 20 No. 7 July 2022

platform, we focused on the above three component
technologies and set the following goals: the processing of large amounts of data, real-time processing ofsuch data, and the degree of precision in the dataprocessing.
3. Reference architecture for theconnected-vehicle platform
Data from connected vehicles are collected via
wireless networks (LTE (Long-Term Evolution) network and the 5th-generation mobile communication
43

Feature Articles

Communication
deviceConnected vehicle

Networkedge

Reception andannotation

StatuscontrolAnnotation
processing

Routecontrol

Queue

CAN data processing
CAN stream

Storage

CAN batch

Dynamic map
Spatiotemporal
data
processing

Image data processing
Image stream

StorageDynamic
map

Image batch
Communication
deviceQueue

Notification and delivery

Static map

Delivery management

4G/5G
Connected vehicle network

Overall system configuration of the field trials

Fig. 3. Reference architecture.

system (5G) network) and stored in servers in datacenters. The data are analyzed, and analysis results
are sent back to connected vehicles. We incorporated
this end-to-end system into the reference architecture
for the connected-vehicle platform. This referencearchitecture consists of six platforms: a network-edge
computing platform, which has a real-time link withconnected vehicles, a reception and annotation platform, which receives data from edge computing
nodes, CAN data processing platform and image dataprocessing platform, both of which process data,
dynamic map platform, which manages data, andnotification and delivery platform, which manages
communication from datacenters to connected vehicles.
We implemented this architecture using opensource software programs that had become de facto
standards in the global market. The goal of this collaboration is to develop technologies that will lead tostandardization with a policy of establishing technologies that are more open and not dependent on the
proprietary software of a specific company. Therefore, the software is structured in such a way that it
can be revised as technology advances and innovations emerge (Fig. 3).
4. Verification of use cases
A server environment for the field trials was implemented in a datacenter based on the reference architecture described above. The feasibility of the following three use cases was verified using test vehicles.
(1) Generation of a static map
In this use case, an operator at the datacenter first
NTT Technical Review

Vol. 20 No. 7 July 2022

sends an instruction to connected vehicles to generate
a map. When a connected vehicle running in the target area receives the instruction, it sends CAN dataand image data to the datacenter. The datacenter
executes preprocessing for generating a map from theimage data, estimates the locations of traffic lights
and other objects from the image data, generates map
data, and registers the data in the map database. We
verified all these processes (Fig. 4).
(2)	Obstacle detection and notification to following
vehiclesIn this use case, the datacenter uses image datafrom onboard cameras and learned obstacle data toinfer an object in images. If the object is an obstacle,
its location is estimated and registered in the dynamic
map database. A challenge in this use case is realtime performance. For this to be practical, it is necessary to detect an obstacle and notify the vehicles
approaching in the rear of it within 7 seconds. We
adopted an architecture that offloads parts of thisprocessing to the network-edge computing nodes,
which greatly improved real-time performance. We
further improved this performance by sending danger
notifications in two stages. The first notification issent promptly to the vehicles in a wider area without
taking time to narrow down the affected area. The
vehicle in danger in a specific lane is then identified
and a second notification is sent to that vehicle(Fig. 5).
(3) Lane-specific congestion detection
In this use case, the datacenter identifies the causeof a traffic jam in three steps. In Step 1, the datacenter
analyzes CAN data of running vehicles in real timeand narrows down the candidate traffic-jam front
44

Feature Articles

Reception andannotation

Operator

Annotation
processing
(4)
Queue

(5)
Image dataprocessing
(6)

(3)

Dynamic map

CAN data processing

Spatiotemporal
data
processing

Mapping
processing(7)

Location estimation
processing

Target-image-collection area
Delivery andnotification(2)

Image-collectionarea notification(1)

Static map

Dynamic
map

Architecture and process flow for generating a static map

Fig. 4. Generation of a static map.

Location
estimation

Datacenter

AxispotTM
high-speed spatiotemporalengineVertical distribution
technology at edge

Inference
process

Edge

Notificationof details

Obstacle
information

Initialnotification(obstacle
notification)

Vehiclelocation

Synchronous
Asynchronous

Vehicle-selection
algorithm

Vehicle selection using vehicle
location and angle of view

Initialnotification
Notificationof details

VehicleObstacle

Obstacle

Obstacle

Architecture for obstacle detection and technology for its deployment

Fig. 5. Obstacle detection and notification to following vehicles.

point on the basis of vehicle density per mesh and perlane. Recurring traffic jams are excluded because
their causes can be identified in advance; thus, we
focused on non-recurring traffic jams. In Step 2, thedatacenter collects image data from the vehicles running in the vicinity of the potentially congested lane
identified in Step 1. Using the image data, the datacenter determines whether there is a traffic jam, its
NTT Technical Review

Vol. 20 No. 7 July 2022

location, and its front point, and identifies its cause.
In Step 3, the datacenter notifies the vehicles runningin the affected lane of the cause of the traffic jam
obtained in Step 2 (Fig. 6).

45

Feature Articles[Step 1]
Narrow down
candidate trafficjam front point[Step 2]
Identify cause of traffic jam using images[Step 3]
Notify only the affectedvehicles of traffic-jam
information

Aggregate CAN data

Send data-collection
request to vehicles

Identify cause basedon image analysis

Notify affected vehicles oftraffic-jam information

Spatiotemporal data
aggregation technology

Vehicle-selection
technology

Adjacent-lane
speed estimation

Polygon searchtechnology

ad

on

lyg

po

obstacleroSend CAN data

Receive datacollection request

Send image data

Receive and display
traffic-jam information

Process for identifying cause of traffic jam

Fig. 6. Lane-specific congestion detection.

5. Verification of the platforms of thereference architecture
We evaluated the following platforms of the reference architecture.
(1) Reception and annotation
We evaluated this platform by focusing on the performance limits of CAN data and image data processing. Specifically, we evaluated the load performanceof annotation processing and queuing processing andidentified bottlenecks under a simulated condition inwhich 5 million vehicles were sending data at intervals ranging from a minimum of 1 second up to 10
seconds.
(2) CAN data processing
We evaluated this platform in terms of real-time
performance and throughput. We implemented realtime processes, such as the regular storage of vehicle
location information, through CAN stream processing and evaluated the performance limit of theirrespective response times.
(3) Image data processing
We evaluated this platform in terms of real-time
performance and throughput. We implemented realtime image processes, such as obstacle detection,
through image stream processing and evaluated theperformance limit of their respective response times.
NTT Technical Review

Vol. 20 No. 7 July 2022

(4) Notification and delivery processing
We evaluated this platform in terms of the cost ofimage collection, which can become a bottleneck ineach use case. An algorithm for selective collection ofvehicle data, which was developed by NTT, was used
to reduce the amount of image data collected. We
evaluated the performance of communication infrastructure technologies assuming that this algorithm
was used.
The verification of the network-edge computing
platform is not described here because it is describedin another article in this issue: “Activities and Resultsof Field Trials—Network Edge Computing Platform”[1].
6. Issues identified and future activities
We verified the reference architecture for threesample use cases using test vehicles running on public roads. In evaluating the communication infrastructure technologies needed for a large-scale connectedvehicle platform, we set goals for data volume, realtime performance, and precision (Fig. 7). Regarding
data volume, we used simulation data and successfully verified that the data processing platform canhandle 30 million vehicles. Regarding real-time performance, we used network-edge computing and46

Feature Articles

Verification classification

Use
case

Architecture

Number ofconnected
vehicles(data volume)

FY2018Generate
static map

FY2019
Ac

hie

ve

d

FY2020Identify cause of traffic jam

5 million
vehicleshie

Tens of millions
of vehicles

veComponent
technologies

ve

d

d

ve

Operate asa systemd

30 million
vehicles—(Simulation evaluation)

hie

10+ s
(without backgroundload)

Precision

hie

hie

Ac

Real-time
performanceAc

Ac

Detect obstacles
Ac

Goal

7s
(with backgroundload, fastest)

5s

ve

d

(with backgroundload, average)
Un
ac
hie
ve
d

10+ m

Several m

Tens of cm(V-SLAM)(landmarks used)(IMU used)

7s

10 cm

Fig. 7. Activity results.

achieved an average notification time (time taken
before sending notification) of 5 seconds (the target
was within 7 seconds) in the obstacle-detection use
case. We attempted to improve precision by using
visual simultaneous localization and mapping(V-SLAM)*, landmark location information, and an
inertial measurement unit (IMU). Despite these
efforts, we were not able to achieve our target of
10-centimeter precision. This will be an issue for
future technological development.
While we achieved our initial goals, except for precision, we also identified a number of technicalissues. The collection, storage, and utilization oflarge amounts of image and other data impose aheavy burden on networks and server resources. For
these technologies to be implemented in society, it isnecessary to develop technologies that are not only

functionally feasible but also more efficient and lessexpensive. To achieve these targets, we are developing technologies that use edge computing to distribute processing loads and use resources efficiently. We
will also expand the application of these technologies
to cases in which they can address social issues, suchas contributing to a low-carbon society, a contributionthat is particularly expected of the automobile industry.
Reference[1]

R. Noji, T. Kamei, K. Funabiki, D. Murata, and Y. Kanemaru, “Activities and Results of Field Trials—Network Edge Computing Platform,” NTT Technical Review, Vol. 20, No. 7, pp. 48–53, July 2022.
h t t p s : / / n t t - r ev i ew. j p / a r c h ive / n t t t e c h n i c a l . p h p ? c o n t e n t s =ntr202207fa7.html

Yu Chiba*

V-SLAM: A technology that simultaneously determines the system’s location and generates an environmental map based on
three-dimensional information of its surroundings.

NTT Technical Review

Vol. 20 No. 7 July 2022

Senior Manager, Manufacturing IT Innovation
Division, NTT DATA Corporation.
He received an M.S. in computer science fromTohoku University, Miyagi, in 2005 and joined
NTT DATA the same year. He was involved in
the research and development of software forlarge-scale systems in the financial sector as wellas the development of many systems in public
and enterprise sectors. After working as a branding manager in the Public Relations Department,
he has been in charge of technological development of a connected-vehicle platform since 2018.

47

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Activities and Results of Field
Trials—Network Edge Computing
Platform
Ryosuke Noji, Takayuki Kamei, Kaito Funabiki,
Daisuke Murata, and Yuken Kanemaru
Abstract

Toyota Motor Corporation and the NTT Group conducted field trials to verify an information andcommunication technology platform for connected vehicles over a three-year period beginning in 2018
and shared their respective technologies and expertise regarding connected vehicles. Distributed processing using edge nodes increased the efficiency and processing speed of the platform. However, thedistribution of processing sites created new challenges. To address these challenges, we developed anarchitecture in which multiple network functions are allocated to edge nodes and verified its effectiveness through these field trials.
Keywords: network, edge node, distributed processing

1. Introduction
At the early stage of the field trials conducted by
Toyota Motor Corporation and the NTT Group, alarge datacenter collected data from vehicles, andmany servers were installed at the datacenter to process the data in a distributed manner. However, thisarchitecture was not able to achieve the target scalability (handling 30 million vehicles) and processing
time (sending a response within 7 seconds). To overcome this limitation, edge nodes were placed between
vehicles and the datacenter, and some data processing
was delegated to these edge nodes. Usually, the term“edge node” refers to a terminal device, for example,
the computing resource in a vehicle in the case of aconnected vehicle. With our new approach, however,edge nodes are also geographically distributed but
located between vehicles and the datacenter.
Challenges in using edge nodes include not only theneed to establish the appropriate application architecture but also a network-related need to transport dataappropriately so that load balancing can be achieved
between the servers and distributed edge nodes. To
NTT Technical Review

Vol. 20 No. 7 July 2022

address the latter issue, we developed an architecturein which processing systems executing multiple network functions are located between a vehicle and theapplication group. In the field trials, we verified
whether this architecture is effective for solving thisissue. To distinguish these processing systems fromapplications installed at terminals (edges), we refer tothese processing systems as Network Edge (Fig. 1).
Network Edge acts as a gateway when vehicles
upload data and a gateway when applications in the
datacenter send notifications to vehicles. One of thegoals with Network Edge is to hide the complexity of
the network or infrastructure so that application
developers can concentrate on developing application
functions.
2. Target use cases
To avoid narrowing down target use cases too
much, we selected two fundamental use cases toverify the performance of Network Edge: vehicle
movement and wide-area rerouting (Fig. 2).
In the use case of vehicle movement, the application48

Feature Articles

Monitoring site

Edge site 1

Edge site 2

Network Edge

Applications

Applications

Annotation platform

Scout platform

Annotation platform

Scout platform

State
management
Network Edge

Metrics
collection

Notification tovehiclesSessionmanagement

Visualization

Network Edge

Inter-site offloading

Inter-site offloading

State-management
coordination

State-management
coordination

Inter-site proxy

Inter-site proxy

Deliverymanagement

Notification tovehiclesSessionmanagement
Deliverymanagement

Queuing function

Queuing function

BrokerBroker

Proxy

Proxy

Routing-control function

Routing-control function

TLS termination

TLS termination

Load balancing

Load balancing

BGP Anycast

BGP Anycast

Mobile network (EPC)

Mobile network (EPC)

BGP: Border Gateway Protocol
EPC: Evolved Packet Core
TLS: Transport Layer Security

Fig. 1. Overview of Network Edge.

Vehicle movement

Wide-area rerouting

Application
serverEnterprise network
pGW

FailureLoadincrease

pGW
Mobile network (EPC)

Move

Enterprise network
pGW

Study traffic control at
the time of network
rerouting due tovehicle movement

Application
serverpGW

Mobile network (EPC)

Study traffic control at
the time of network
rerouting due to
network-device failure
or load concentration

pGW: Packet Data Network Gateway

Fig. 2. Fundamental use cases.

NTT Technical Review

Vol. 20 No. 7 July 2022

49

Feature Articles

The field trials using actual vehicles were conducted in the Odaiba area in Tokyo.

LoadingsystemVehicle data for 5 million vehicles

5G field-trial area

01010

01010

01010

01010

010

010

010

010

1010101

1040101

Experimental servers at
Osaka Data Center

1010101

1010101

Experimental servers at
Tokyo Data Center

5G: the 5th-generation mobile communication system

Fig. 3. Overview of the environment for the field trials.

facility and server to which a vehicle is connected areswitched as the vehicle moves. In principle, data areprocessed at the edge node closest to the vehicle concerned, and the results are sent from there to neighboring vehicles. This can be described as local production for local consumption of data. As the vehicle
moves, it is necessary to switch not only the edge
node to which the vehicle is connected but also theapplication executing the server. We verified whether
this switching requirement can be supported by network technology.
In the use case of wide-area rerouting, the application facility to which a vehicle is connected isswitched when a failure occurs at the facility or when
it is necessary to execute load balancing. As edge
nodes are distributed, a failure or overload of an edge
node makes it necessary to transfer the processing
load from the edge node concerned to neighboringedge nodes. We verified whether, in the proposedarchitecture, failures of edge nodes can be detected
NTT Technical Review

Vol. 20 No. 7 July 2022

and transfer of processing loads between edge nodes
can be supported.
3. Network Edge functions andverification items
Network Edge performs three main functions: control of routing to application servers at datacenters,load balancing based on metrics, and message queuing distributed over a wide area. If we put geographic
distribution aside for the moment, data from a vehicle
reaches an edge node via a wireless access network,
such as a mobile network. The control functions mentioned above send data to applications on the edge
node or, in some cases, to applications in the datacenter site.
In the field trials, we implemented an environment
with multiple edge nodes using datacenters in Tokyo
and Osaka to confirm the two use cases and verified
the effectiveness of the three functions (Fig. 3). Since
50

Feature Articles

the use of Transport Layer Security (TLS) is assumed
to encrypt communication with vehicles for increased
security, terminating TLS at Network Edge may
become a processing bottleneck. Therefore, we verified the performance of TLS termination. Details ofthese verifications are described below.

For example, when the application load exceeds athreshold during an experiment, 70% of the load atCenter 1 is processed at Center 1 and the remaining
30% is offloaded to Center 2.

4. Routing control

If the only response to a server or datacenter failure
is to reroute data to a server in another datacenter,data transmission from vehicles would fail during thererouting time. If this occurs, depending on the situation, it is necessary either to retransmit data or give up
sending data. The purpose of the wide-area distributed message-queuing function is to prevent this fromoccurring.
As the term “message queue” indicates, the message-queuing function receives data only temporarily.
A message queue resides at each edge node; therefore, message queues at different nodes work together
to deliver data to the appropriate servers when
offloading processing loads.
Since the edge node to which a vehicle is connected
is switched as the vehicle moves, the datacenter thatwants to send a notification to a vehicle needs toknow the edge node to which the vehicle is currently
connected. We aimed to satisfy this need using thisqueuing function. When the datacenter receives messages from applications, it retains them so that it cansend them to the vehicle concerned even if the vehicle
has moved to an area covered by a different edge
node. In the field trials, we examined whether thisoccurs reliably.
If Apache Kafka or similar software is used to
execute these processes, problems arising in that data
are exchanged unnecessarily between datacenters;
thus, it is impossible to manage notification delivery.
To avoid this problem, we combined basic messagequeuing software with a proxy program we developed. (We used NATS, which is open-source software, for the basic message-queuing function.) The
properties required for data upload and data download (sending notifications) are different. Therefore,
for the former, we used a mechanism with which thetarget data are first selected then sent. For the latter,
we adopted a mechanism with which metadata areshared and there is a logical queue that spans different
datacenters.

We investigated two methods of controlling routing
data from a vehicle to an application server: thedomain name system (DNS) method and load balancer (LB) method. We confirmed the basic functions
and performance of the two methods and investigated
how they behave when a vehicle moves or wide-area
rerouting takes place as a result of a facility failure.
(1)	DNS method: Border Gateway Protocol(BGP) Anycast connects a vehicle to the nearest DNS server, and the DNS server selects themost appropriate datacenter and application
server and notifies the vehicle of this information.
(2)	LB method: BGP Anycast connects a vehicle
to the nearest LB server, and the LB forwards
requests from vehicles to the most appropriate
server at the most appropriate datacenter.
With the LB method, vehicles always communicate
with the application via the LB. This lowers performance but makes finer-granularity control possible.
We compared the two methods in the early stage ofthe field trials and found that both were applicable tothe target use cases. Therefore, from then on, the fieldtrials were based on the LB method.
5. Load balancing using metrics
On the assumption that applications were located in
multiple servers in multiple datacenters and processed in a distributed manner, we verified load balancing in which the datacenter and server to use were
determined on the basis of metrics information, andin which the routing control described above wasused.
We examined whether extreme performance degradation could be avoided by detecting server or datacenter failures, software process failures, or overloads from received information about response
times from application servers, facility information,and metrics of the infrastructure resources, and by
conducting load balancing on the basis of the detected situation. We also verified whether the load can be
distributed over a wide area in specified proportions.
NTT Technical Review

Vol. 20 No. 7 July 2022

6. Wide-area distributed message queuing

7. Verification of TLS performance
Vehicles communicate using TLS to ensure security.
Applications communicate using HTTPS (Hypertext
51

Feature Articles

Transfer Protocol Secure) or MQTTS (Message
Queuing Telemetry Transport Secure) because theyuse HTTP or MQTT. Since we chose to use the LB
method for routing control in this verification, theLayer 7 LB software needs to handle a large numberof transactions. Normal tuning alone would result ina central processing unit bottleneck. Therefore, TLS
processing was offloaded to another piece of hardware. We verified whether this would improve performance and resource-usage efficiency. Two types of
hardware, TLS Accelerator and SmartNIC, both ofwhich can be installed in a standard AMD64 server,
were used for this verification. Although we have notyet been able to confirm the effectiveness of thismethod when the dominant traffic is data uploaded
from vehicles, we will continue to verify this method
since, for a connected vehicle platform designed fortens of millions of vehicles, it is important to minimize the number of required servers by using
resources efficiently.

8. Future outlook
In these field trials, we implemented an architecturein which Network Edge, which perform multiple
functions, is placed between vehicles and applications [1] and were able to confirm the effectiveness of
this architecture.
On the basis of this architecture, we will further
develop technologies and attempt to combine themwith other technologies to develop a more advanced
platform. For example, we are considering the application of artificial intelligence to metrics-based loadbalancing and coordination with priority control ofapplication servers. We will continue our efforts toachieve a world in which intelligent network technology supports real-time communication of tens of millions of fast-moving vehicles and in which engineers
can concentrate on developing applications withoutworrying about complex infrastructure conditions.
Reference[1]

NTT Technical Review

Vol. 20 No. 7 July 2022

Press release issued by NTT, “Tech Document on ICT Platform forConnected Vehicles Released by NTT and Toyota,” Nov. 29, 2021.
https://group.ntt/en/topics/2021/11/29/techdoc_rel_ntt_toyota.html

52

Feature Articles

Ryosuke Noji

Senior Manager, Smart Mobility, Smart World
Business, Business Solution Division, NTT
Communications Corporation.
He started his career as a solution engineer at
NTT Communications in 2007. After several
projects, he joined the research and development
department. As a data platform architect, hedesigned and developed data platforms for thecompany’s IPTV Service and the Data Science
Lab, which is a data science Center of Excellence
at NTT Communications, and analyzed anddeveloped algorithms for several services. He has
been engaged in solution development for the
automotive industry since 2018 and has led aconnected-vehicle project for developing network and computing services or solutions,
including edge computing, for connected-vehicle
big data.

Takayuki Kamei

Engineer, Dept. Technology, Innovation Center, NTT Communications Corporation.
He started his career as a network engineer at
NTT Communications in 2013. He has continued
to research and develop network functions virtualization, software-defined wide area network,
application-layer overlay networks, and content
delivery network. He has been engaged in thedevelopment of edge computing and traffic optimization since 2019 and led the architecture
design of distributed platforms over a WAN
including connected-vehicle use cases.

Daisuke Murata

Engineer, Innovation Center, NTT Communications Corporation.
He received an M.E. in information sciencefrom Nara Institute of Science and Technology in2014. In 2018, he joined NTT Communications
and has been engaged in the development/
research of computer systems and computer networking technologies, especially in the area of
TLS acceleration since 2020.

Yuken Kanemaru

Engineer, Dept. Technology, Innovation Center, NTT Communications Corporation.
He started his career as a network engineer at
NTT Communications in 2019. He has continued
to research and develop application-layer overlay
networks and content delivery network. He has
been engaged in the development of edge computing, traffic optimization, and distributed platforms over a WAN including connected-vehicle
use cases.

Kaito FunabikiEngineer, Smart Mobility, Smart World Business, Business Solution Division, NTT Communications Corporation.
He started his career as a network engineer at
NTT Communications in 2017. After working on
the operation and development of an internet
exchange service, he has been engaged in a connected-vehicle project for developing networkand computing services or solutions, includingedge computing, for connected-vehicle big data
since 2020.

NTT Technical Review

Vol. 20 No. 7 July 2022

53

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Real-time Spatiotemporal
Data-management Technology(AxispotTM)
Atsushi Isomura, Naoko Shigematsu, Isoo Ueno,
Nobuhiro Oki, and Yutaka Arakawa
Abstract

To enable the obstacle-detection use case, in which vehicles approaching an obstacle (e.g., falling
object) on the road are immediately notified of the obstacle, and the lane-specific congestion detection
use case, in which the number of vehicles in each lane on the road is determined, it is necessary to store
data sent simultaneously from a large number of connected vehicles, search the data in real time forvehicles present within a certain area (mesh area, road, parking lot, etc.) at any specific time, and determine the number of these vehicles. This article describes the real-time spatiotemporal data-management
technology (AxispotTM) that we are developing to meet these requirements.
Keywords: digital twin, spatiotemporal data, AxispotTM

1. Introduction
Internet of Things (IoT) technology makes it possible to collect a range of information about people,
objects, and the natural environment in real space andcentrally manage these pieces of information on thecloud. IoT is becoming essential for next-generation
services that require management of moving objects,
such as people and vehicles. Such services include
lane-specific congestion detection, route search to
avoid congested routes, and aerial delivery of goods bydrones. In a collaboration between NTT and Toyota
Motor Corporation, we are studying an obstacledetection use case, in which vehicles approaching an
obstacle on the road are immediately notified of the
obstacle. To enable this use case, it is necessary tostore a large amount of data in real time, search thedata in real time for vehicles present within a certain
area at any specific time, and determine the numberof these vehicles.
NTT Human Informatics Laboratories is developing a real-time spatiotemporal data-management
system, AxispotTM*1 [1, 2], which stores data sent
NTT Technical Review

Vol. 20 No. 7 July 2022

simultaneously from a large number of moving
objects and searches the data in real time for moving
objects present within a certain area at any specifictime (Fig. 1). The following sections present four
technical issues facing current technologies andintroduce the four core technologies that are includedin AxispotTM, which are intended to overcome thesetechnical issues. We also evaluate the improvement in
performance achieved with these technologies. Finally, the issues being addressed to further advance ourresearch and development efforts, as well as their
future outlook, are described.
2. Technical issues
A spatiotemporal database is a database for efficiently storing, searching for, and extracting groupsof data related to both spatial information, such as
latitude and longitude, and temporal information, suchas time and period [3, 4]. Searching a spatiotemporal*1 AxispotTM: A spatiotemporal data-management technology that
identifies (spots) data from multi-dimensional axes in real time ora system using the technology.

54

Feature Articles

Cloud applications (congestion detection, vehicle-dispatch planning, anomaly detection, etc.)
AxispotTM
High-speed mesh/polygon aggregation
Cloud
environment

High-speed polygon search
Real-time storage and mesh search technology
Data reception and distribution platform
NTT network lines (5G network, etc.)
Surrounding
vehicles in
the rearPrecedingvehicleSupport for safe drive(Obstacle notification)

High-speed, high-precisioncounting of vehicles(Lane-specific congestion detection
and road pricing)

Real-time spatial-information
sharing by humans and vehicles(AR navigation)

AR: augmented reality
5G: fifth-generation mobile communications

Fig. 1. Image of using AxispotTM.

database for objects present within a certain area of acertain shape (road, parking lot, etc.) using spatial
information is called a polygon search. To enable the
obstacle-detection use case, it is necessary to store
data generated by a huge number of vehicles throughout the country in a spatiotemporal database and use
polygon searches to extract only specific vehicles.
However, the following four technical issues stand in
the way of achieving our target performance.
•	Issue 1: In an ordinary spatiotemporal database,indexes are generated on the basis of a searchtree*2, which is commonly used in relational databases. However, when data are to be stored ordeleted in such a database, an update to the searchtree occurs. If the database is handling data fromtens of millions of vehicles, such search tree
updates significantly reduce the efficiency ofstoring, updating, or deleting data, making it difficult to store data in real time.
•	Issue 2: When it is necessary to handle a large
amount of data, a distributed database consistingof multiple servers is used. In such a case, eachserver is usually assigned to the management ofdata for a specific region. However, the number ofvehicles on the road differs from region to region
and varies with the time of day. Therefore, theNTT Technical Review

Vol. 20 No. 7 July 2022

amount of data handled by a server varies greatly
among the servers that make up a distributed
database. As a result, the load may be concentrated on a few servers, thereby significantly
reducing processing performance and making
real-time storage and search difficult.
•	Issue 3: With a polygon search, the more vertices
the target polygon has, the slower the search
becomes [5]. Unfortunately, the polygons provided by high-precision maps *3, which havebecome popular, have a large number of vertices.
Therefore, polygon searches that use polygons
provided by a high-precision map are slow, falling far short of the speed required for real-timesearch.
•	Issue 4: When aggregating the number of vehicles
on each road around the country on the basis ofspatiotemporal data in a database, it is necessary
to assign the number of meshes or polygons
within the aggregation area to computers in amanner that takes into account the number ofrunning vehicles in each mesh or polygon. In*2 Search tree: Indexing technology for data indexes that have a tree
structure such as an r-tree.
*3 High-precision map: High-precision map data that include information on each lane of each road.

55

Feature Articles

2D Morton curve
1

0

1

11
10

1001

3D extension

100Time (30 bits)

Latitude (30 bits)

Longitude (30 bits)

t1 ... t30

x1 ... x30

y1 ... y30

1D bit sequencing & key-value management

01

00

100

Spatiotemporal index (90 bits)

10 Range searchby prefix-match isavailable

100101

010

011

Value (backward 51 bits,
forward match search)t1x1y1 ... t13x13y13

x14y14t14 ... x30y30t30

Index
No treerequired

Subdivision

64 years
× half a circleof the earth
× one circleof the earth

100...1
100...0011...0

Key(forward 39 bits, exact match search)

011...1

Approx. 1 hour
× 2.5 Km× 5.0 Km30 ms
× 3 cm× 3 cmFig. 2. Spatiotemporal index.

most services, the number of meshes or polygons
is manually assigned to computers in advance.
However, as mentioned earlier, the number ofrunning vehicles varies depending on the region
and time of day. Consequently, the amount ofjobs *4 allocated to each computer is highly
uneven, making aggregation computation inefficient.
3. Proposed technologies
This section describes the four technologies proposed by NTT Human Informatics Laboratories toaddress technical issues (1) through (4) mentioned
above then introduces the overall architecture of
AxispotTM.
3.1 Spatiotemporal indexing
This technology does not use a tree structure. It is
an indexing technology that uses the characteristics
of a space-filling curve called a Morton curve. The
two-dimensional (2D) Morton curve shown on theleft of Fig. 2 is a process of repeatedly dividing theworld map into four parts and assigning either bit [0]or [1] to each partitioned area. The longer this 1D bit
sequence becomes, the narrower the area it represents. A forward match search can be executed on thisbit sequence to search a certain range of space. For
example, a search of the forward bit [1001...] is asearch of the yellow box area in the world map shownin Fig. 2. We added a time dimension to the Morton
NTT Technical Review

Vol. 20 No. 7 July 2022

curve, thereby extending it to a 3D Morton curve.
Furthermore, the forward and backward bits of thegenerated bit sequence are respectively stored as Key
and Value in a database called key-value store, as
shown on the right of Fig. 2. As a result, a spatiotemporal range search can be executed on the basis of an
exact match of Key. This spatiotemporal indexing
technology makes both high-speed data storage andreal-time spatiotemporal range search possible without using a tree structure.
3.2 Limited node selection
This technology is a new data-distribution technology that determines multiple combinations of servers
on the basis of location and time and distributes data
to one of the combinations. This technology provides
the three effects shown in Fig. 3. (1) Load distribution
can be equalized by selecting multiple servers insteadof selecting one specific server for a specific region.
(2) Since the selected servers included in a combination change with the passage of time, continuous loadon specific servers can be prevented. (3) At the timeof the search, a unique combination of servers can be
identified on the basis of time and space. Therefore,
unnecessary searches involving all servers can beprevented. This limited node-selection technology
prevents load concentration on specific servers andenables high-speed storage and search.

*4 Job: A group of processing operations.

56

Feature Articles(1) Load balancing duringstorage(2) Automatic switching ofstorage location(3) Preventing search accesstowards all servers

7:00
7:01
Query

7:02
7:03

7:00–7:01

Time(1) Equalize load
distribution(2) Prevent continuous load ona specific server(3) Acquire data by only
accessing to necessary nodes

Fig. 3. Limited node selection.

Polygon (proposed technology)

Vertex reduction
algorithm

Polygon (raw data)

Fig. 4. Fast polygon search with vertex reduction.

3.3 High-speed polygon searchThis technology applies the Douglas-Peucker algorithm [6] for simplifying polygonal lines forming
polygonal shapes to significantly reduce the numberof vertices while maintaining the original polygonal
shape and to speed up polygon search. By applying
the Douglas-Peucker algorithm to the polygon (rawdata) in Fig. 4, a polygon with a significantly reduced
number of vertices (proposed technology) is generated. After applying this conversion, the high-speed
polygon search technology efficiently manages polygons in a database. This technology prevents any
NTT Technical Review

Vol. 20 No. 7 July 2022

reduction in polygon search speed by using polygons
generated on the basis of a high-precision map, making real-time search possible.
3.4 High-speed spatiotemporal data aggregation
This technology automatically allocates aggregation jobs within the target aggregation range of spatiotemporal data to computers by using a distributed
processing platform. As shown in Fig. 5, when data
for a spatial area are to be aggregated using meshes,jobs are allocated to computers in such a way that theamount of jobs allocated to each computer is equal
57Feature Articles

Distributed processing infrastructure

Targetaggregationarea(iii) (2) 4) 4) (i) (2) 4) (iii) (i) 4) (2) (iii)(iii) (i) 4) (iii) 4) (2) 4) (i) (2) (i) (i) (i)4) (2) (i) (iii) (2) (iii) (iii) (i) (2) 4) (i) 4)(2) (iii) (i) 4) 4) (2) (i) (iii) (i) (iii) 4) (iii)(iii) (iii) 4) (2) 4) (i) (i) (i) (iii) (i) (i) (iii)(iii) (2) (iii) (iii) (2) (iii) 4) (2) (i) 4) (2) (i)(i) 4) (i) 4) (2) (i) (i) 4) (2) (2) (i) (iii)(i) (i) (iii) (iii) (iii) 4) 4) (2) (i) (iii) (iii) 4)(i) (2) (i) (2) (i) (iii) (2) (i) 4) (iii) 4) (iii)(iii) 4) (i) (2) (i) (iii) (i) 4) (iii) (i) 4) (i)(i) (2) (2) (i) 4) (i) 4) (iii) 4) (2) (i) (2)4) (i) (i) (iii) (iii) (2) (2) 4) (iii) (2) 4) (2)

(2)
(2)

(i)
(i)

(iii)
(iii)

4)
4)

Aggregation
results

Image of this technology applied to Odaiba in Tokyo Bay

Fig. 5. High-speed mesh-based aggregation.

even though the meshes allocated to each computer
may be spatially scattered. This high-speed spatiotemporal data aggregation technology prevents the
allocation of uneven amounts of jobs across computers and reduces the aggregation response time.
AxispotTM includes these four technologies and has
the architecture shown in Fig. 6. The spatiotemporal
database shown in the figure can use Redis [7] andApache Ignite [8] to store spatiotemporal data. The
static polygon database can use open-source software, such as PostgreSQL [9], to store polygon information.
4. Evaluation
This section shows a comparison between whenboth spatiotemporal indexing technology and limited
node-selection technology are applied to a database
and when neither technology is applied (i.e., only
conventional technologies are used) to the same database. We collected data for several million vehicles,
the number of vehicles constantly running in the
Kanto region surrounding Tokyo, and used a datasetthat was designed to search for vehicles that exist in a
specific space and time. As shown in Fig. 7(a), we
confirmed that storage performance improved by a

NTT Technical Review

Vol. 20 No. 7 July 2022

factor of 3.1 and search performance by a factor of246. As shown in Fig. 7(b), we also confirmed thatthese technologies prevent load concentration
because they evenly allocate data to the servers thatmake up the distributed database. By using thesetechnologies in the collaborative field trials with
Toyota Motor Corporation, we were able to achieve
real-time vehicle data storage and search.
5. Future outlook
NTT Human Informatics Laboratories is developing a technology called Digital Twin Computing [10],which copies all relevant objects in the real world intoa virtual space in real time. To enable this, it is essential to store and update, in real time, data that aretransmitted simultaneously from all moving objects,
including not only humans and vehicles but also
drones, ships, and satellites, and to search an area ofany shape, such as a city (buildings, roads, etc.) andenvironments (rivers, etc.), for the necessary data.
Therefore, we are studying the extension of indexing
in the height dimension, which is currently not supported by AxispotTM, as well as the searching of 3D
polygons.

58

Feature Articles

Application

High-precision 3D map

Aggregation infrastructure (Apache Spark)
Aggregation by mesh
Map conversion API

Road coordinate data

Mesh search(Search using
spatiotemporal index)

Polygon database(PostgreSQL)

Polygon search

Spatiotemporal database(Redis, Apache Ignite)(1) Spatiotemporal indexing technology(2) Limited node-selection technology

Extraction results

Lane 4

Road polygon +vehicle data

Lane 3

Vehicle data
(dynamic data)

Lane 2

Search range

Lane 1

Road A

Polygon after vertex
reduction

Road B

Road polygons(static map)

Search range

Search API

Road polygon generation(Vertex reduction algorithm)
Original polygon

Aggregation by polygon(Determine whether inside oroutside a polygon usingmesh search results)(3) High-speed polygon search technology

Spatiotemporal data-storing module(Spatiotemporal indexing & load balancing that takes into accountuneven geographical distribution)(4) Mesh and polygon aggregation technology

Storage API
Dynamic data
(Latitude, longitude, time, data)

API: application programming interface

Fig. 6. AxispotTM architecture.

(a)

(b)
Distribution of storage load
(number of data items on each server)

Storage throughput (10,000 req/s)
20
Load balanced

3.1-fold

10

100%

NTT
Technology
Server 5

0
NTT
Technology

Conventional
technology

Conventional
technology
Server 5
Loadconcentrated

80%Server 4

Server 4

60%

Search throughput (10,000 req/s)
4Server 3

3

Load
concentrated

40%2Server 3

Server 1

Server 2
Server 1

20%

246-fold

0.02
0.01
0

Server 2

0%
NTT
Technology

Conventional
technology

Fig. 7. Evaluation results.

NTT Technical Review

Vol. 20 No. 7 July 2022

59

Feature Articles

References
[1]

[2]

[3]
[4]

A. Isomura, I. Naito, Y. Iida, and T. Nakamura, “Axispot: A Distributed Spatiotemporal Data Management System for Digital Twins ofMoving Objects,” IEEE Software, Vol. 39, No. 2, pp. 33–38, 2022.
https://doi.org/10.1109/MS.2021.3132899
A. Isomura, K. Miyahara, I. Naito, and M. Hanadate, “Real-time
Spatial-temporal Database for Geographic Polygon Data Using HDmap,” Proc. of the 4th International Conference on Geoinformatics
and Data Analysis (ICGDA 2021), pp. 7–13, April 2021. https://doi.
org/10.1145/3465222.3465230
T. Abraham and J. F. Roddick, “Survey of Spatio-temporal Databases,” GeoInformatica, Vol. 3, pp. 61–99, 1999. https://doi.
org/10.1023/A:1009800916313
N. Pant, M. Fouladgar, R. Elmasri, and K. Jitkajornwanich, “A Survey

Atsushi Isomura

Researcher, NTT Digital Twin Computing
Research Center.
He received a Master’s degree in information
science and technology from Aichi-Prefectural
University in 2016. He joined NTT Software
Innovation Center in 2016 and helped develop
AxispotTM, a distributed spatiotemporal databasemanagement technology. Currently, he is working on digital-twin data-management technology
at Digital Twin Computing Research Center. He
received the Institute of Electronics, Information
and Communication Engineers (IEICE) Tokai
Section Student Award in 2015 and the bestposter award in 12th International Conference on
Ubiquitous Healthcare. He presented at RedisConf19 held in San Francisco.

Naoko Shigematsu

Research Engineer, NTT Digital Twin Computing Research Center.
She received a B.S. and M.S. in geophysics
from Tohoku University, Miyagi, in 1993 and1995. In 1995, she joined NTT Telecommunication Networks Laboratories and in 1999 moved
on to NTT EAST R&D Center, where she
engaged in the research of asynchronous transfer
mode network operation systems. In 2000, she
moved on to NTT Information Sharing Platform
Laboratories, where she engaged in research ofstorage area networks. Since 2020, she has beenin charge of research for Digital Twin Computing
at NTT Digital Twin Computing Research Center.

of Spatio-temporal Database Research,” Lecture Notes in Computer
Science (LNCS), Vol. 10752, pp. 115–126, 2018.
[5] I. E. Sutherland, R. F. Sproull, and R. A. Schumacker, “A Characterization of Ten Hidden-surface Algorithms,” ACM Computing Surveys, Vol. 6, No. 1, pp. 1–55, 1974.
[6] M. Visvalingam and J. D. Whyatt, “The Douglas-Peucker Algorithm
for Line Simplification: Re-evaluation through Visualization,” Computer Graphics Forum, Vol. 9, No. 3, pp. 213–225, 1990. https://doi.
org/10.1111/j.1467-8659.1990.tb00398.x[7] Redis, https://redis.io/[8] Apache Ignite, https://ignite.apache.org/[9] PostgreSQL, https://www.postgresql.org/[10] Press release issued by NTT, “NTT Announces New R&D Projects ofDigital Twin Computing,” Nov. 13, 2020. https://group.ntt/en/
newsrelease/2020/11/13/201113c.html

Nobuhiro Oki

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a B.E. and M.E. in chemistry from
Tokyo Institute of Technology in 1996 and 1996.
He joined NTT in 1996 and launched the Internet
telegram system (D-Mail). In 2008, he joined
NTT Communications and launched various
cloud services, including Smart Data Platform(formerly called Enterprise Cloud). In 2018, hejoined NTT laboratories and helped develop
AxispotTM. He is currently working on digitaltwin data-management technology.

Yutaka Arakawa

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a B.E. and M.E. in electronicengineering from the University of Tokyo in2001 and 2003 and Ph.D. in engineering from the
University of Electro-Communications, Tokyo,
in 2014. He joined NTT in 2003. His current
research interests include distributed systems anddata structure design. He is a member of IEICE.

Isoo Ueno

Research Engineer, NTT Digital Twin Computing Research Center.
He received a B.S. and M.S. in earth sciencefrom Kobe University, Hyogo, in 1990 and 1992.
He joined NTT in 1992 and engaged in basic
research on complex systems. In 1998, helaunched internet businesses in Hong Kong and
UK. He also managed the development projectsof OCN web hosting, Biz authentication, and theglobal billing system of NTT Communications.
In 2021, he helped develop the polygon-based
data-aggregation technology of AxispotTM. He iscurrently working on Digital Twin Computing.

NTT Technical Review

Vol. 20 No. 7 July 2022

60

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Selective Vehicle-data-collection
Algorithm
Masaru Takagi, Kazuya Matsuo, Ryota Nakada,and Koya Mori
Abstract

In an obstacle-detection use case, in which a monitoring system keeps track of an obstacle on the road,the system needs to continuously collect the latest images of the obstacle captured using onboard cameras. Although image-recognition technology can be used to accurately select relevant images (images
that capture the obstacle in question), computational resources available in a vehicle are too limited to
execute this task. In addition, transferring all images via a mobile network to the cloud incurs considerable communication costs. To solve these problems, we devised a technology that estimates the range ofthe area that can be captured with each camera (hereafter, visible range) and selectively collects only therelevant images. The visible range is estimated on the basis of meta-information, such as vehicle position, direction of movement, and camera angle of view.
Keywords: connected vehicle, data collection, priority control

1. Potential of connected vehicles
Connected vehicles are expected to become key
players in using information and communication
technology in road transportation. They also holdgreat potential as a sensing platform. Since connectedvehicles are equipped with various high-end sensors
and communication modules and move around a citywith virtually no risk of running out of battery, they
are ideal as mobile sensor nodes. If camera images
and LiDAR (light detection and ranging)* pointcloud data from tens of millions of connected vehicles could be collected, it would be possible to continuously scan an entire city and build a digital twinof that city.
2. Assumed use case
Unfortunately, the total amount of sensor data generated by tens of millions of connected vehicles
ranges from 10 to 100 Tbit/s. This is too enormous forcommunication networks, computers, and storage
units to handle. This means that it is impractical tocollect all available sensor data. It should be noted
NTT Technical Review

Vol. 20 No. 7 July 2022

that sensor data include not only data that should becollected immediately but also data for which somecollection delay is tolerable, data for which periodic
collection is sufficient, and data that are of no value.
Therefore, it is important to selectively collect important sensor data on a priority basis. By adjusting thepace and timing of sensor-data collection in accordance with the amount of load on the communication
network and computers, it is also possible to equalize
the load fluctuation over time, thus improve facilityutilization efficiency.
The collaboration projects between Toyota Motor
Corporation and the NTT Group include several use
cases that require the selective collection of sensor
data. This article focuses on the obstacle-detection
use case and introduces the technical challenges,
implementation details, and our efforts to improve theperformance of selective collection. In this use case,obstacles on the road are assumed to be moved by thewind or removed by the road administrator. Therefore,*

LiDAR: A type of remote sensing technology using light. It measures the distance to an object on the basis of the reflection timeof a laser light emitted in all directions.

61

Feature Articles: Obstacle
Found an obstacle.
First detection

Cloud
Notify surrounding vehicles.
Is there any obstacle there?

5 minutes later

Cloud
Looks like the obstacle was moved a little.
Is there any obstacle there?

10 minutes later

Cloud
It’s already gone.

Fig. 1. Overview of the obstacle-detection use case.

the monitoring system keeps track of an obstacle on
the road by collecting images from vehicles passingin the vicinity of the obstacle (Fig. 1).
3. Approach based on image recognition
At first glance, the use of image recognition seems
to be the obvious approach to select images that capture the obstacle to be monitored. This approach canbe implemented in two different configurations, butboth present significant problems.
Configuration 1: Image recognition is executed
within connected vehicles
The simplest configuration is to mount a computeron a connected vehicle to execute image recognition.
However, this configuration has three problems
unique to vehicles.
First, the temperature inside a vehicle under theburning sun in mid-summer can reach close to 80°C
[1]. At such a high temperature, it is not easy to keep
the computer operating stably. Second, it is said thatthe average usage rate of passenger cars is only 5%.
If the computer stays idle most of the time, having acomputer in a passenger car is not cost-effective.
Third, the product life cycles of vehicles and computers differ greatly. The computer will reach the end ofits life cycle or become obsolete in terms of performance much earlier than the vehicle on which it is
NTT Technical Review

Vol. 20 No. 7 July 2022

mounted. Considering these problems, it is not realistic to install a computer capable of image recognition
in a connected vehicle.
Configuration 2: Image recognition is executed in
the datacenter
If there is a connected-vehicle platform in place, allimages can be transferred to the datacenter so thatimage recognition can be executed there. Although
this configuration prevents the three problems mentioned above, the cost of image transfer over themobile network is very high. The load on the mobile
network is also enormous because images are sentfrom tens of millions of vehicles simultaneously.
Even if sensor-data traffic from a connected vehicle isjust 1 Mbit/s, the total traffic from 10 million vehicles
can reach 10 Tbit/s. This is 15 times the current volume of mobile communication traffic in Japan [2].
4. Approach based on location information
Faced with the problems mentioned above, we reexamined the requirements for the obstacle-detection
use case. There is no need to detect an obstacle withhigh precision during image selection because thecollected images are analyzed in detail at a later stageof processing. The number of images to be analyzed
should be kept as small as possible because the processing for image analysis requires a large amount of62

Feature Articles(1)(2) Maintain and search thevehicle-information database(3) Determine the distance
OK

OK

NG
(4) Determine the angle of viewOK(5) Detect shielding vehicles
OKNG

Map source:
Geospatial Information
Authority of Japan

Fig. 2. Process of image selection based on meta-information.

computation. Since the location of the obstacle to bemonitored is already known, there is also no reason
not to use this information as a clue.
In light of the above, we devised a technology that
selectively collects images without relying on image
recognition. Specifically, it estimates the visible
range of the onboard camera of each vehicle on thebasis of meta-information, such as the locations ofsurrounding vehicles, direction of the vehicle’s
movement, the vehicle’s size, the angle of view of its
camera, and the camera’s resolution. Images that arelikely to have captured the obstacle are then selectedon the basis of the positional relationship between the
obstacle and each vehicle. This image-selection process consists of the following five steps (Fig. 2).
Process 1: Maintain a vehicle-information database
Vehicle meta-information, which provides important clues for image selection, is routinely collectedfrom connected vehicles and stored in the real-time
spatiotemporal database AxispotTM.
Process 2: Search the vehicle-information database
When a request for obstacle monitoring arrives, thesystem searches the vehicle-information databaseusing the spatiotemporal index associated with the
obstacle location and the current time and selects the
NTT Technical Review

Vol. 20 No. 7 July 2022

search hits as candidate vehicles.
Process 3: Determine the distance
Next, the distance between each candidate vehicle
and the obstacle is calculated, and any vehicle with adistance from the obstacle greater than the visible
range of its onboard camera, which is calculated fromthe resolution of the onboard camera, is eliminatedfrom the list of candidates.
Process 4: Determine the angle of view
Next, the direction of the obstacle from each candidate vehicle is calculated, and any vehicle that is
estimated not to have captured the obstacle within theangle of view of its onboard camera is eliminatedfrom the list of candidates.
Process 5: Detect shielding vehicles
Finally, the positional relationship between surrounding vehicles and the obstacle on the road is
examined, and any vehicle that is found to be unable
to detect the obstacle because its view of the obstacleis blocked (shielded) by other vehicles is eliminatedfrom the list of candidates.
5. Reducing complexity for shielding
vehicle detection
Of these five steps, the detection of shielding vehicles demands the most complex computation and63

Feature Articles(1) Calculate the visible range of each vehicle(2) Check for presence/absence of a shielding vehicle

NG

OK

NG

OK

OK

OK

Fig. 3. Example of checking for presence/absence of a shielding vehicle (before algorithm improvement).

accounts for the bulk of the computation time. Therefore, during the field trial period, we attempted to
substantially improve the algorithm for shielding
vehicle detection. This section presents the algorithm
before and after this improvement, in this order.
The most intuitive method of detecting a shielding
vehicle is to calculate the visible range of each candidate vehicle and determine whether there is any
vehicle within this range that blocks the view of the
obstacle. For example, suppose there are six vehicles
in the vicinity of an obstacle, as shown in Fig. 3. A
fan-shaped field of view is drawn for each of the six
vehicles on the basis of the angle of view and visible
range of its camera. The system then checks whether
a vehicle’s view of the obstacle is blocked (shielded)by any of the other five vehicles and eliminates any
vehicle that is unable to detect the obstacle. Since thismethod requires a brute-force examination of any
shielding vehicle, the computational complexity isO(n2), where n is the number of vehicles. This means
that, as the number of vehicles in the vicinity of the
obstacle increases, the processing time increases rapidly. Therefore, in situations where many vehicles aredensely packed around the obstacle due to traffic
congestion, the processing time can become very
long.
We, therefore, took on the challenge of improving
the algorithm to reduce the amount of computation.
The key point of this improvement effort was to abandon the conventional approach of calculating the visible range of each onboard camera. We reversed thethinking: from examining whether a vehicle can
NTT Technical Review

Vol. 20 No. 7 July 2022

detect the obstacle to examining whether the obstacle
can detect the vehicle (i.e., whether the view of thevehicle from the obstacle is blocked by anothervehicle). The system calculates the visible range ofthe obstacle instead of the visible range of the vehicle
to determine whether any vehicle blocks the view ofthe obstacle from each vehicle (Fig. 4). With thismethod, the system examines each vehicle, one byone, to check whether it blocks the view of othervehicles. This eliminates the need for a brute-force
calculation and reduces the computational complexity to O(n). Also, by approximating the fields of viewas a set of fan shapes, the process of determining
whether a vehicle blocks the view of others can be
implemented in a very simple manner.
6. Performance evaluation
We evaluated the processing time before and after
the algorithm improvement for various vehicle location patterns (Fig. 5). It was assumed that the visible
range of the onboard cameras was 100 m. The number of vehicles posited to be present within a 100-m
radius of the obstacle was varied. To evaluate theworst case, it was important to set a sufficiently highupper limit for the number of vehicles. We set theupper limit at 2000 vehicles, assuming that the vehicles were packed as densely as in a parking lot. Javawas used as the language for implementing the selection algorithm. A geometric calculation library called
JTS Topology Suite [3] was used with the algorithm
before the improvement, but not after the improvement.
64

Feature Articles(1) Calculate the visible range of the obstacle(2) Detect shielding vehicles
OK
NG

NG

OK

OK

OK

Fig. 4. Example of shielding determination (after algorithm improvement).

Pattern (1)Pattern (2)Range: OK
Angle: NG

Range: OK
Angle: OK
Shield: OK

Case of 50 vehicles

Case of 50 vehicles

Pattern (3)[Legend]
Obstacle location

Range: OK
Angle: OK/NG
Shield: OK/NG

100-m radius
Vehicles in the vicinity
Map source: Geospatial Information Authority of Japan

Case of 100 vehicles

Fig. 5. Vehicle-location patterns in performance evaluation.

Figure 6 shows the processing time for each vehicle-location pattern with varying the number ofvehicles. Before the algorithm improvement, theprocessing time for detecting shielding vehicles
increased rapidly in proportion to the square of thenumber of vehicles. After the algorithm improvement, the processing time was proportional to thenumber of vehicles. In the worst-case scenario with
2000 vehicles, the processing time for detecting
shielding vehicles was 25 ms, which is 1/60 of theprocessing time before the algorithm improvement.

NTT Technical Review

Vol. 20 No. 7 July 2022

7. Future outlook
In these field trials, we optimized the shieldingvehicle-detection algorithm for the requirements of
the obstacle-detection use case and achieved a processing speed 60 times faster than that for when thealgorithm was implemented in a conventional mannerusing a geometric calculation library. In the future,
we will investigate the possibility of using our study
results for applications other than connected vehicles.
For example, when creating a digital twin of a city,
it is necessary to ensure that it reflects the latest stateof the city by periodically re-sensing every area.
65

Feature Articles

Pattern (1)
(ms)
125
100
75
50
25
0

(ms)
125

Other
ShieldingAngle of view
Visible range
DB search
100

200

100
75
50

500

1000

No shielding
detection2000 (vehicles)

25
0

100

200

500

1000

2000 (vh.)

100

200

500

1000

2000 (vh.)

100

200

500

1000

2000 (vh.)

Pattern (2)
(ms)
1000
800
600
400
200
0

100

200

500

1000

2000 (vh.)

Pattern (3)
(ms)
1500
1200
900
600
300
0

100

200

500

1000

2000 (vh.)

Shieldingdetection
5.8 times
fasterShieldingdetection
60.8 times
faster

(ms)
1000
800
600
400
200
0

(ms)
1500
1200
900
600
300
0

Fig. 6. Vehicle-location patterns vs. processing times.

Some parts of the cityscape change more frequently
than others. Therefore, it is more efficient to systematically re-sense only key areas frequently than touniformly re-sense the entire city. In such situations,the algorithm presented in this article could be used
to efficiently select data to be collected. We will also
study application-related issues, such as issues that
may arise when the monitoring targets are widely
spread or when position information contains errors.

NTT Technical Review

Vol. 20 No. 7 July 2022

References
[1]
[2]

[3]

Japan Automobile Federation, “Temperature Inside a Vehicle in Midsummer” (in Japanese), https://jaf.or.jp/common/safety-drive/carlearning/user-test/temperature/summer
Ministry of Internal Affairs and Communications, “Aggregation andProvisional Calculation of Internet Traffic in Japan—Announcementof Aggregate Results as of November 2021,” Feb. 2022. https://www.
soumu.go.jp/main_sosiki/joho_tsusin/eng/pressrelease/2022/2/04_02.html
The JTS Topology Suite, https://locationtech.github.io/jts/

66Feature Articles

Masaru Takagi

Ryota Nakada

Researcher, NTT Digital Twin Computing
Research Center.
He received a B.E. in engineering and M.E. and
Ph.D. in information science and technology
from the University of Tokyo in 2013, 2015, and2018. He joined NTT in 2018 and was engaged
in a collaboration project with Toyota Motor
Corporation. He is currently studying multiagent mobility simulation as part of Digital Twin
Computing. He is a member of the Institute ofElectronics, Information and Communication
Engineers (IEICE) and Information Processing
Society of Japan (IPSJ).

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a B.E. and M.E. in mathematics
from Tokyo University of Science in 2008 and2010. He joined NTT Network Innovation Laboratories in 2010 and researched a high-resolution
image-transport system. He moved on to NTT
WEST in 2013, where he studied an indoorlocation information system. He rejoined NTT
Network Innovation Laboratories in 2016 andhas been researching edge computing. Since
2020, he has been in charge of research for Digital Twin Computing at NTT Digital Twin Computing Research Center.

Kazuya Matsuo

Koya Mori

Researcher, NTT Digital Twin Computing
Research Center.
He received a Ph.D. in information sciencefrom Osaka University in 2017 and joined NTT
the same year. While at university (2012–2017),
he studied the collection of data in mobile sensor
networks. He studied modeling of vertically distributed applications in NTT until 2019. Since
2018, he has been involved in collaborative work
between Toyota and NTT. Since 2020, he has
also been working on a Digital Twin Computing
project. He received a Young Researcher’s Award
from IEICE in 2018. He is a member of IPSJ.

NTT Technical Review

Vol. 20 No. 7 July 2022

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
Since joining NTT in 2004, he has been
involved in the research and development of anInternet-of-Things application based on software
technologies, such as OSGi, OpenStack, andEdge Computing. Since 2020, he has been in
charge of research for Digital Twin Computing at
NTT Digital Twin Computing Research Center.

67

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Vertically Distributed Computing
Technology
Kazuya Matsuo, Masaru Takagi, Ryota Nakada,and Koya Mori
Abstract

We developed a technology that quickly shares information found by a connected vehicle with othervehicles. For example, when a connected vehicle finds an obstacle on the road, the technology can transmit information about it quickly to other vehicles. This quick notification is achieved by offloading partof the processing of the collected obstacle information to network-edge nodes and transmitting interim
results to vehicles near the obstacle. However, in urban areas, the delay in notification may become large
because a large number of vehicles connect to a small number of network-edge nodes; thus, the volumeof data received from these vehicles may overwhelm the computing resources of these nodes. To solve
this problem, we also developed a technology that dynamically selects which computer will execute any
particular notification processing on the basis of the states of the connected vehicles. Using this technology, obstacle information can be transmitted quickly even when a large number of vehicles are connected to a small number of network-edge nodes.
Keywords: distributed processing, edge computing, load balancing

1. Roles and challenges with an ICT platform
for connected vehicles
The role of an information and communication
technology (ICT) platform for connected vehicles is
to collect videos captured with onboard cameras and
CAN (controller area network) data from connectedvehicles, analyze them, and send the analysis results
to the other vehicles to provide these vehicles withmore information than they can obtain from their sensors alone. This enables drivers to obtain information
about their blind spots, preventing traffic accidents
from occurring.
When connected vehicles are widely used, a large
amount of data will be collected from a large numberof such vehicles and may overwhelm the ICT platform. Even in such situations, the ICT platform must
be able to quickly notify vehicles of urgent information, such as that relating to obstacles.
In the collaboration between the NTT Group andToyota Motor Corporation, we conducted field trials
for several use cases in which the ICT platform notiNTT Technical Review

Vol. 20 No. 7 July 2022

fies relevant vehicles about the processing results forthe data it has collected.
The vertically distributed computing introduced inthis article consists of the vertically distributed application architecture and technology for dynamicallyselecting processing nodes we developed. The following sections describe the challenges each of thesetechnologies addresses, solutions each provides, andverification results of the field trials.
2. Vertically distributed computing
2.1 Vertically distributed application architecture(1)	Obstacle detection and notification in the previous ICT platform
The previous ICT platform used in our collaboration with Toyota was built on the lambda architecture[1, 2], in which stream (real-time) data processing
and batch data processing are executed in parallel,allowing both quick results from stream processing
and detailed results from batch processing to be madeavailable (Fig. 1). To be able to execute obstacle
68Feature Articles

All data

Batch-processing layer

Processing-resultprovision layer

Batch-processing
group

Batchprocessing
results

Inquiry aboutprocessing results

Collected
dataReal-time-processing layer
Real-timeprocessing
results

Real-time-processing
group

Fig. 1. Lambda architecture.

detection and notification within 7 s, the platform
estimated the type and location of any obstacle fromeach frame (image) by stream processing. However,the lambda architecture does not take into account therequirements for processing time. Therefore, a system built on this architecture does not necessarily
satisfy the requirement to execute obstacle detection
and notification within 7 s. The actual processing
time has been about 15 s.
(2) Vertically distributed application architecture
To solve this problem, we proposed an architecturethat allows a processing-time threshold to be set in
the lambda architecture. This new architecture iscalled a vertically distributed application architecture(Fig. 2). In this architecture, groups of processes are
extracted from a series of processes in the order of
their processing in such a way that the total processing time does not exceed the threshold. The results ofthese extracted processes can be obtained by the client. We revised the ICT platform to run on this architecture. In the revised ICT platform, object recognition is extracted from image-data processing, which
consists of object recognition and object-location
estimation. By doing so, connected vehicles canobtain information about the type of obstacle before
the ICT platform completes the estimation of the
obstacle location. To further increase the processing
speed, the revised ICT platform offloads*1 object
recognition to network-edge nodes *2, which arelocated closer to the relevant connected vehicles than
NTT Technical Review

Vol. 20 No. 7 July 2022

the datacenter servers are. These nodes send obstacle
information directly to the connected vehicles. In thisarchitecture, obstacle detection and notification are
executed in the following two stages (Fig. 3):(i)	First notification: Information about the typeof obstacle and the rough obstacle location(actually, the location of the connected vehicle
that took the image of the obstacle) is sent toconnected vehicles near the obstacle.
(ii)	Notification of details: The type of obstacle
and more precise information about its location, as obtained from the obstacle location
estimation, are sent to connected vehicles near
the obstacle.
(3) Field trial
In a field trial, the vehicle that took the image of the
obstacle and the vehicle that received the obstacle
information were the same. We measured the time
between when the onboard camera captured the
obstacle and when the vehicle received the obstacle
information from the ICT platform. The results areshown in Table 1. The first notification was sent
within 7 s. As mentioned above, the first notification
only contained information about the approximate
location of the obstacle because the precise location
had not yet been calculated at this point. Since this*1 Offloading: A mechanism by which the processing load of acomputer is reduced by passing it to another computer.
*2 Network-edge node: A set of computers that are located at NTT’sbase stations and central offices.

69

Feature Articles

Batch-processing layer

Processing-resultprovision layer

Batchprocessing
group

Batchprocessing
results

All data

Collected
data
Processing layer for
quick response

Real-time-processing layer

Real-timeprocessing group

Real-timeprocessing group

Inquiry
about processing
results
Real-timeprocessing
results
Temporaryprocessing-resultprovision layer
Interim realtime-processing
results

Fig. 2. Vertically distributed application architecture.

ICT platform using vertically
distributed application architecture

Previous ICT platform
Datacenter

Datacenter

Image-data processing
Location
estimation

Objectrecognition

Notification

Image-dataprocessing

Notificationof details

Location
estimation

Objectrecognition

Vehicle in
the rear

Obstacle

Vehicle ahead

Notification delay: large
Location accuracy: highObstacle

Notification
Edge selection/transmission
request

Edge

First
notification

Notificationof details

Vehicle in
the rear

Notification delay: large
Location accuracy: highNotification delay: small
Vehicle Location accuracy: low
ahead

Fig. 3. A before-and-after comparison for the introduction of vertically distributed application architecture.

approximate location was the location of the vehicle
that took the image of the obstacle, it was several toten meters away from the real obstacle location.
NTT Technical Review

Vol. 20 No. 7 July 2022

However, since vehicles can move more than 10 m/s,
we believe that providing information about theapproximate obstacle location more than 10 s earlier
70

Feature Articles

Table 1. R esults of the field trial of vertically distributed
application architecture.
Processing time
From obstacle detection
to first notification

From obstacle detection
to notification of details

Average (s)Fastest (s)Average (s)Fastest (s)

4.574

4.092

9.772

9.212

than had been possible, even at the cost of location
accuracy, would be extremely valuable for assisting
safe driving.
2.2 Dynamic selection of processing nodes(1)	Issues confronting the ICT platform after the
adoption of the vertically distributed applicationarchitecture
The field trial showed that the vertically distributed
application architecture made quick notification possible. However, the trial was conducted under conditions such that the computing resources of the network-edge nodes were not exhausted. When connected vehicles are widespread, a large number ofthem may be connected to a small number of network-edge nodes. The concentration of the processing load for object recognition on these edge nodes
may overwhelm their computing resources, making it
impossible to send notifications quickly.
One way to distribute a load that would otherwise
concentrate on a small number of computers is tooffload the processing load to the most appropriatecomputers on the basis of the acceptable processing
delay and remaining computer resources. Many suchtechnologies have been proposed (e.g., [3]). However, adopting only these conventional technologies
cannot solve the above problem because vehicles canmove more than 10 m/s, causing the surrounding
conditions to change quickly from moment tomoment, and the acceptable processing delay varies
depending on these surrounding conditions. For
example, if an obstacle is located on a blind curve, therisk of an accident is very high, and it is urgent to alert
the vehicles near the obstacle. In contrast, if an obstacle is on a road with good visibility, the risk of an
accident is low, and it is less urgent to send obstacle
notifications to the vehicles near the obstacle. Conventional technologies do not take into account theseclient situations (i.e., the road situations mentioned
above). Therefore, if the ICT platform’s load balancing is based on conventional technologies, obstacle
notification may not be sent quickly even in situations
NTT Technical Review

Vol. 20 No. 7 July 2022

where quick notification is absolutely needed, or it
may be sent quickly even when there is no urgent
need for it to be.
(2)	Overview of technology for dynamically selecting processing nodes
To address the issue mentioned above, we developed a technology for dynamically selecting processing nodes. It adds to the conventional load balancing
technology the ability to take into account the surrounding conditions of the vehicles to be notified.
This technology consists of the following three processes:(i)	Determine data-processing priority on thebasis of the surrounding conditions of thevehicles to be notified (the part added in the
proposed technology)(ii)	Select the computers to which the target processing is offloaded (conventional technology)(iii)	Execute the above offloading (conventional
technology)
Various algorithms for Process (i) can be conceived
for each use case. In the field trial, we adopted analgorithm designed for the obstacle detection andnotification use case and verified its effectiveness.
(3)	Application of the technology for dynamicallyselecting processing nodes to the ICT platform
In the field trial, we adopted an algorithm that takes
into account the requirement that connected vehicles
moving at high speed without being able to detect the
obstacle should be urgently notified. Specifically, theplatform searches for connected vehicles in the vicinity of the vehicle that transmitted the relevant onboard
camera image. These are the targets for obstacle notification. The platform then determines whether it isurgent to notify these connected vehicles on the basisof the following two conditions (Fig. 4):(i)	Whether the average speed of the target connected vehicle exceeds the threshold.
(ii)	Whether the obstacle is visible from the driver’s seat (to be more precise, whether thedriver can see the location of the connectedvehicle that sent the relevant obstacle-containing image when the object is first detected, orwhether the driver can see the estimated location of the obstacle when the obstacle has
already been precisely detected and is being
monitored).
To implement Condition (ii), the platform uses analgorithm similar to the selective vehicle-data-collection algorithm to determine whether the view of the
obstacle is blocked. If quick notification is required,71

Feature Articles

Condition (i): Average moving speed

Condition (ii): Shielding

Whether the speed of thesurrounding vehicles exceeds
the threshold.

Whether there are vehicles
that cannot detect the
obstacle.
Cannot detect
Can detect

Fig. 4. Determining whether there are connected vehicles requiring quick notification.

Table 2. S etting of processing priority for images fromonboard cameras.
(i) Average
moving speed(ii) Shielding

High
Low

Priority

Yes

High

No

MediumYes

MediumNo

Low

the processing priority is set to high. If quick notification is not required, the processing priority is set tolow. For in-between cases, the processing priority is
set to medium. This priority setting is summarized inTable 2.
The next step is to determine, based on the priority
setting just described, which computer should execute the object recognition. As with the conventional
technology, our technology for dynamically selecting
processing nodes monitors the remaining resourcesof each computer in the system and selects the appropriate computer on the basis of the remaining
resources of each computer and the processing priority, as shown in Table 3. Finally, the platform offloads
the object-recognition process to the selected computer and the latter executes the process.
Figure 5 illustrates an example computer configuration in the ICT platform and how the computers towhich the target process is offloaded are selected. We
assume that the network-edge node receives theonboard camera videos in the sequence of videos 1 to
6 in the figure and that the object-recognition priority
for each video is set as shown in the figure. Object
recognition 4, which processes video 4, has low priNTT Technical Review

Vol. 20 No. 7 July 2022

Table 3. S election of the processing computer based onpriority.
Priority

Processing computerHigh

Nearby network-edge nodewith remaining resources

MediumNearby computer with remaining
resources (network-edge nodeor datacenter server)

Low

Datacenter serverority thus offloaded to a datacenter server. Object
recognition 5 has medium priority. It is executed by
the network-edge node because the above offloading
has freed up its resources. This exhausts the computing resources at the network-edge node. Therefore,
object recognition 6, which has medium priority, isoffloaded to a datacenter server.
(4) Field trial
In the field trial, we examined how the number of
onboard camera videos sent to the network-edge nodeaffects the image-processing time. The results areshown in Fig. 6. The horizontal axis shows the number of videos received per second. The number ofhigh-priority videos was varied between 1 and 3, andthe number of medium-priority videos was varied
between 1 and 9. The vertical axis shows the increasein processing time from a condition under which
computing resources are sufficiently available. As
can be seen from this figure, when no dynamic selection of processing nodes was used, the processing
time at the network-edge node increased when morethan three videos were received per second. In contrast, when dynamic selection of processing nodes
72

Feature Articles

Computing resources
Datacenter servers

Computing resources

Object recognition 6

Object recognition 6

Object recognition 4

Object recognition 5
Video 6

Object recognition 4

Video 4

Network-edge
node

Object recognition 3
Object recognition 2
Object recognition 1

Video 5

Video 1
Video 3
Video 2

Priority: highVideo 4

Video 6

Priority: medium
Priority: low

Increase in processing time (s)Fig. 5. Example of the dynamic selection of processing nodes.

No dynamic selection used:
Processing time of network-edge
node increases.

420
320
220

Dynamic selection used:
Both network-edge node and datacentercan send notifications quickly.

No dynamic selection used:
Processing time increases
when more than 3 videos arereceived per second.

120
20
0

1

2

3

4

5

Number of videos received per second

No dynamic
selection

Dynamic selection(network-edge node selected)

6

7

8

9

Dynamic selection(datacenter selected)

Dynamic selection used:
Even if the number of medium priority videos increases,
network-edge node can send notifications quickly becausesome of its processing load is offloaded to datacenter.

Fig. 6. Results of the field trial of dynamically selecting processing nodes.

NTT Technical Review

Vol. 20 No. 7 July 2022

73

Feature Articles

was used, the processing time at the network-edge
node did not increase until nine *3 videos werereceived per second. This was because the networkedge node processed only high-priority processes andother processes were offloaded to the datacenter.
Because of this offloading, the processing time at the
datacenter began to increase when six videos werereceived per second.
3. Future outlook
Since computer technology is advancing daily, thecomputers in vehicles have become powerful enoughto process the video from their onboard cameras.
However, it is not assumed with the current vertically
distributed computing technology that connectedvehicles can process onboard camera videos. To
make more effective use of the computing resources
at both network-edge nodes and datacenters, we will
study the extended use of the computing resources inconnected vehicles.
In the obstacle detection and notification use case,an additional type of processing is required: when theplatform detects an obstacle in the image from a connected vehicle, it needs to determine whether the
obstacle is being identified for the first time or has
already been found in the image from another vehicle. This is because, if the obstacle has already beenfound, the connected vehicles near the obstacle will
have already been notified of the obstacle, and so
there is less urgency. We are also studying technology
for determining whether the objects detected by multiple sensors are identical [4].
References
[1]
[2]
[3]
[4]

N. Marz, “Big Data Lambda Architecture,” 2012. http://www.
databasetube.com/database/big-data-lambda-architecture/
K. Dutta and M. Jayapal, “Big Data Analytics for Real Time Systems,” Big Data Analytics Seminar, pp. 1–13, 2015.
Y. Dai, D. Xu, S. Maharjan, and Y. Zhang, “Joint Load Balancing andOffloading in Vehicular Edge Computing and Networks,” IEEE Internet of Things Journal, Vol. 6, No. 3, pp. 4377–4387, June 2019.
K. Matsuo, M. Takaki, R. Nakata, and K. Mori, “A Study on a
Dynamic Object Detection and Identification System That Uses Point
Cloud Alignment for Creating a High-precision Dynamic Map,” SIG
Technical Reports on Human-Computer Interaction, Vol. 2021, No. 6,
pp. 1–8, 2021 (in Japanese).

*3 This load is very small for a world in which connected vehicles
are widely used. This small value was derived because the fieldtrials were intended to confirm the effectiveness of each technology studied in this collaboration, thus were conducted using theminimum computer configuration. When the ICT platform isbuilt with a large-scale computer configuration to confirm practical feasibility in the future, a larger load will be applied whenverifying the effectiveness of each technology.

NTT Technical Review

Vol. 20 No. 7 July 2022

74

Feature Articles

Kazuya Matsuo

Researcher, NTT Digital Twin Computing
Research Center.
He received a Ph.D. in information sciencefrom Osaka University in 2017 and joined NTT
the same year. While at university (2012–2017),
he studied the collection of data in mobile sensor
networks. He studied modeling of vertically distributed applications in NTT until 2019. Since
2018, he has been involved in collaborative work
between Toyota and NTT. Since 2020, he has
also been working on a Digital Twin Computing
project. He received a Young Researcher’s Award
from the Institute of Electronics, Information andCommunication Engineers (IEICE) in 2018. He
is a member of the Information Processing Society of Japan (IPSJ).

Ryota Nakada

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
He received a B.E. and M.E. in mathematics
from Tokyo University of Science in 2008 and2010. He joined NTT Network Innovation Laboratories in 2010 and researched a high-resolution
image-transport system. He moved on to NTT
WEST in 2013, where he studied an indoorlocation information system. He rejoined NTT
Network Innovation Laboratories in 2016 andhas been researching edge computing. Since
2020, he has been in charge of research for Digital Twin Computing at NTT Digital Twin Computing Research Center.

Koya Mori

Masaru Takagi

Researcher, NTT Digital Twin Computing
Research Center.
He received a B.E. in engineering and M.E. and
Ph.D. in information science and technology
from the University of Tokyo in 2013, 2015, and2018. He joined NTT in 2018 and was engaged
in a collaboration project with Toyota Motor
Corporation. He is currently studying multiagent mobility simulation as part of Digital Twin
Computing. He is a member of IEICE and IPSJ.

NTT Technical Review

Vol. 20 No. 7 July 2022

Senior Research Engineer, NTT Digital Twin
Computing Research Center.
Since joining NTT in 2004, he has been
involved in the research and development of anInternet-of-Things application based on software
technologies, such as OSGi, OpenStack, andEdge Computing. Since 2020, he has been in
charge of research for Digital Twin Computing at
NTT Digital Twin Computing Research Center.

75

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and ToyotaLane-specific Traffic-jam-detection
Technology
Kouhei Mori, Yuki Yokohata, Aki Hayashi,
Takahiro Hata, and Masato Kamiya
Abstract

Dynamic maps are being constructed to support automated driving and advanced navigation. They
combine high-precision map information with traffic-related information, such as that about traffic controls and jams. We at NTT believe that detecting lane-specific traffic jams, such as those caused by aqueue of vehicles waiting to enter the parking lot of a commercial facility or by vehicles parked on thestreet, and providing information on these jams will enable unprecedentedly advanced navigation. This
article describes a technology for detecting lane-specific traffic jams on the basis of information that canbe collected from connected vehicles.
Keywords: lane-specific traffic jam, connected vehicle, traffic flow optimization

1. Introduction
Dynamic maps [1] are being constructed to support
automated driving and advanced navigation. They
combine high-precision map information with trafficrelated information such as that about traffic controls
and jams. Such traffic-related information is mappedon a dynamic map in real time, enabling map systems
to detect road-traffic conditions and provide services,such as navigation, to optimize traffic flow. To construct a dynamic map, an approach is needed to collect, in real time, traffic-related information on roads
across the country. Examples of such an approach are
millimeter-wave radar [2] and traffic counters [3],which can be used to measure the volume of the flowof passing vehicles at fixed points, thus detect traffic
jams. However, they must be installed at all locations
where traffic jams are expected, so cannot cover awide area. Another approach [4] involves using thesmartphones of drivers to measure their vehicle
speeds to detect traffic jams. This approach can covera wide area but does not provide lane-specific resolution. In real-world traffic, lane-specific queues occur
at various locations such as at the entrance to theparking lot of a commercial facility, at the exits ofhighways, or in turn lanes at intersections. When
NTT Technical Review

Vol. 20 No. 7 July 2022

drivers who are new to an area encounter such aqueue, they find it difficult to judge whether theyneed to stay in the queue to reach their destination.
They may wait in a queue unnecessarily or avoid thequeue when they actually need to join it, with theresult that they must take a roundabout route to reach
their destinations.
If lane-specific queues (traffic jams) and specificsections within a queue can be detected, it will bepossible to provide unprecedented, advanced driving
assistance, such as lane-specific navigation, that
determines, on the basis of the driver’s destination,
whether the driver should join or avoid a queue in aparticular lane, enabling drivers to reach their destinations without queuing up unnecessarily. This article describes a technology that detects lane-specifictraffic jams on the basis of video data and vehicle
speeds obtained from connected vehicles [5].
2. Lane-specific traffic-jam-detection technology
2.1 Definition
Taking a cue from the definition of a traffic jam
applied to current services on general roads [6], we
defined a lane-specific traffic jam as a line of traffic
moving at 10 km/h or less. We also defined the76

Feature Articles

Lane-specific traffic jam

Driver’s perspective (onboard camera image)Part of a lane-specific traffic jam
VehiclesDirection of movement

Fig. 1. Driver’s perspective when encountering a lane-specific traffic jam.

minimum length of a traffic jam as 42 m, which is
1/10 the length defined in current services. In thisarticle, the connected vehicle from which data arecollected is referred to as an observation vehicle, andvehicles other than the observation vehicle thatappear in images from the onboard camera mounted
on the observation vehicle are referred to as othervehicles.

2.3 Processing flow
The front camera of the observation vehicle takes avideo of the direction in which the vehicle is moving.
NTT Technical Review

Vol. 20 No. 7 July 2022

IoU = |X Y| .(1)
|X Y|
⊂ ⊃

2.2 Processing overview
Figure 1 illustrates road conditions and an onboard
camera image when a lane-specific traffic jam is
encountered. From the onboard camera video takenwith the observation vehicle, our technology counts
other vehicles overtaken by the observation vehicle
and that of other vehicles that passed in the opposite
lane. The density of other vehicles can then be calculated from the speed of the observation vehicle andnumber of vehicles counted in each lane. A lanespecific traffic jam can be detected from this density
calculation. When a lane-specific traffic jam is
detected, the timeframe used to evaluate the traffic
jam is shifted, and the evaluation is repeated to estimate the section where the lane-specific traffic jam islocated and identify the vehicles at the front and backof the queue.

From that video, the following processes are executed, as shown in Fig. 2.
(1) Detecting vehicles
The collected video is divided into frames, and an
object-detection technology [7] is used to detect othervehicles in each frame. A detected vehicle is represented as a rectangle (detection rectangle) and labeled
as a car, truck, or bus on the basis of the vehicle type(Fig. 3).
(2) Tracking
Since the relative positions between the observation
vehicle and other vehicles change over time, themovements of other vehicles between frames aretracked and numbered. Next, a number is assigned toeach of the other vehicles in each frame. The degreeof similarity between the detected rectangle X in thecurrent frame N and detected rectangle Y in the f-th
frame back from N (i.e., Frame N – f) is evaluatedusing a similarity measure between two sets. Thismeasure is called the intersection over union (IoU):

A pair of rectangles with the largest IoU, i.e., therectangles with the most common pixels, are judged
to be the same vehicle. This makes it possible to track
the trajectory of this vehicle.
(3) Counting overtaken vehicles
The same reference lines are drawn on each frame(Fig. 4). The trajectory of the other vehicle obtained77Feature Articles(1)
Vehicledetection(2) Tracking(3) Counting
overtaken
vehicles(5) Lanespecific trafficjam detection

Output:Detection time andlocation information

Input:
Front camera image(4) Vehicle-speed
information andtravel-distancecalculation
Input:
Vehicle speed and location information

Fig. 2. Processing flow.

Car

Car

Truck

Car

Fig. 3. Detection of vehicles from a frame image (car, truck, or bus).

Frame N − f

Referenceline (left)

Frame N

Referenceline (right)

Relativemovement
vectorFig. 4. Counting overtaken vehicles using a reference line.

by tracking is expressed as a relative movement vector.
When the relative movement vector (of one of theother vehicles) from the road-perspective vanishingpoint to the outside of the frame intersects a referenceline, it is determined that the observation vehicle has
overtaken this other vehicle. In this manner, overtaken vehicles are counted.
(4) Calculating the travel distanceThe distance traveled by the observation vehicle iscalculated from the vehicle-speed information.
(5) Detecting a lane-specific traffic jam
NTT Technical Review

Vol. 20 No. 7 July 2022

Unfortunately, the threshold for the number ofovertaken vehicles per evaluation timeframe is notclear. Therefore, we instead used the density of theovertaken vehicles (vehicle density) as a temporary
solution because it is reasonable to think that the distance between vehicles is shorter in a traffic jam thanin normally flowing traffic.
The threshold for the distance occupied by thevehicles in the adjacent lane (called the occupied
distance threshold, Lθ) due to a traffic jam is calculated from the number of overtaken vehicles, the78

Feature Articles

length, li, of an overtaken vehicle i, and the distance,
τi, between vehicle i and the vehicle immediately infront of vehicle i. The occupied distance threshold isobtained as
Lθ = ∑(li + τi). (2)i

Although vehicle length varies from vehicle tovehicle, we used the following vehicle lengths in ourlearning model for simplicity. The length of a carlcar = 4.8 m, that of a truck ltruck = 12.0 m, and that ofa bus lbus = 12.0 m. The distance between two vehicles, τ, is set to 5.0 m regardless of the vehicle typebecause at such a distance it is reasonable to thinkthat the vehicles are packed together but that eachvehicle is far enough from the vehicle in front of it to
be able to avoid a collision by braking suddenly. Notethat the upper limit of traffic speed for the definitionof a traffic jam is 10 km/h (2.778 m/s). At this speed,
there is a sufficient stopping distance (brake reaction
distance + braking distance) between vehicles. The
upper limit of the traffic speed for the definition ofminor congestion is 20 km/h. At this speed, there is
not a sufficient stopping distance between vehicles.
The reaction time was set to 0.75 s, which is considered a typical value, and the coefficient of friction
was set to 0.7 assuming a dry road surface.
Brake reaction distance: 2.778 m/s × 0.75 s= 2.08 m
Braking distance: ( 10 × 10) ÷ (254 × 0.7)
0.56 mLet x, y, and z respectively denote the number of cars,
number of trucks, and number of buses that were
overtaken. Then, the occupied length isLθ = xlcar + yltruck + zlbus + (x + y + z)τ.(3)If this occupied distance is longer than the observation vehicle’s travel distance D, i.e., ifLθ ≥ D, (4)

the distance between vehicles is shorter than thedefined threshold, it is determined to be a lane-specific traffic jam.
2.4 Other new features
To improve detection accuracy, it is important toprevent object-detection failures caused by backlighting or reflections. For this purpose, a Kalman filter isused to predict where the other vehicles are heading.
To determine whether other vehicles are running inthe same direction as the observation vehicle or running in the opposite lane, our technology examines
whether the video shows the front or back of a vehiNTT Technical Review

Vol. 20 No. 7 July 2022

cle. If there are many vehicles facing towards thecamera, the system determines that the other vehicles
are in the opposite lane and reverses the front and lastvehicles.
3. Field trials
3.1 Overview of the evaluation
The accuracy of our lane-specific traffic-jam-detection technology was evaluated using a test vehicle—
running in the Odaiba district, Minato Ward, Tokyo—
as the observation vehicle. The vehicle ran on the tworoutes indicated with the solid red line and dottedblue line in Fig. 5. Preliminary investigations indicated that there is a high probability of naturally
occurring lane-specific traffic jams on these routes. A
camera was installed in the upper front of the observation vehicle. The resolution of the video from thiscamera was 1920 × 1080 pixels. The video was taken
at 10 fps and saved. The location information
expressed in latitude and longitude was obtained atone-second intervals using GPS (Global Positioning
System). The vehicle’s speed was determined on thebasis of the change in the vehicle’s location over time.
3.2 E
 valuation of accuracy in detecting lane-specific traffic jams
This section describes the accuracy evaluation conducted using 48 videos (each 60 s long) for situations
in which the presence of a lane-specific traffic jam
was visually confirmed.
(1) Visual checkWe decided that our technology must be able todetect any lane-specific traffic jam with a length of 42m or longer. We manually extracted video scenes inwhich the observation vehicle continuously overtook
a series of vehicles, such as three cars and one truck.
The data in these visually verified scenes were treatedas the correct data.
(2) Accuracy evaluation
Table 1 shows a confusion matrix comparing theresults of the visual check and the application output.
Precision (true positive (TP)/(TP+ false positive(FP))) was 89.7%. The main factor for an FP was theeffect of vehicles in parking lots or other premises.
This applied to three cases. The main factor for a false
negative (FN) was object-detection errors caused
when some vehicles in the opposite lane were hidden
behind guardrails or other objects, even though alane-specific traffic jam was present in that lane. Thisapplied to four cases. Factors common to both FP and
FN included object-detection errors, errors caused
79

Feature Articles

Tokyo Big Sight
Odaiba Marine Park

ine

iL

a
ink

L
Tokyo Teleport
Daiba

Aomi

Tokyo Cruise Terminal

Telecom Center

Fig. 5. Travel routes.

Table 1. Evaluation of accuracy in detecting lane-specific traffic jams.
Application output

Visual check Lane-specifictraffic jam
No lane-specifictraffic jam

because we fixed the occupied distance of othervehicles, and inaccurate travel distance of the observation vehicle due to errors in positional information.
3.3 E
 valuation of accuracy in estimating the frontvehicle in a queue
We conducted an additional evaluation for 70 TP
scenes, scenes for which both the visual check and
NTT Technical Review

Vol. 20 No. 7 July 2022

N = 98Lane-specifictraffic jam

No lane-specifictraffic jam

TP: 70

FN: 8

FP: 8

TN: 12application output indicated the presence of a lanespecific traffic jam. We compared the frame in which
the visual check determined that the observation
vehicle overtook the front vehicle with the frame inwhich the application did the same. The comparison
results are shown in Fig. 6. The average difference intime between the two frames was 0.57 s, the median
was 0 s, and the maximum was 4.1 s. The speed limit
80Feature Articles

60
Number of cases

50
40
30
20
10
0

0
0.1–1.0 1.1–2.0 2.1–3.0 3.1–4.0 4.1–5.0
Error between visual check result and application output (s)Fig. 6. Accuracy in estimating the front vehicle.

on these routes is 50 km/h. With the distance between
cars taken into account, the time required to overtakea car is approximately 0.7 s. Therefore, the average
difference of 0.57 s is acceptable. We examined 14
cases in which the error was 1.6 s or more and identified two main factors that caused the large error. The
first was that there were errors in the positional information and, as a result, the observation vehicle’s
travel distance was not calculated accurately (thedistance calculation process was either continued orinterrupted). This applied to nine cases. The second
factor was that cars were incorrectly recognized astrucks or vice versa. This applied to three cases.

the front of a queue were evaluated on the basis of thevideo from a front-facing onboard camera and positional information. We plan to improve the accuracyof our technology and study methods for identifying
the causes of lane-specific traffic jams.
References
[1]

[2]
[3]

4. Conclusion
We developed a lane-specific traffic-jam-detection
technology. It uses video data, vehicle speed, andlocation information that can be collected from asingle connected vehicle to determine if there is atraffic jam in each lane and, if there is, identify thefront and the last vehicles in the traffic queue. We
conducted field trials, in which the accuracy of traffic-jam detection and the estimation of the vehicle at

NTT Technical Review

Vol. 20 No. 7 July 2022

[4]

[5]

[6]
[7]

H. Koyama and Y. Shibata, “Dynamic Map Maintenance in Automated Driving,” Journal of the Institute of Systems, Control andInformation Engineers, Vol. 60, No. 11, pp. 463–468, 2016 (in Japanese).
Website of Central Nippon Highway Engineering Tokyo Company
Limited, “Millimeter Wave Radar Type Traffic Monitoring System,”https://www.c-nexco-het.jp/tech/english/detail/0048.php
Website of Hitachi Industry & Control Solutions, Ltd., “Hitachi
Image Traffic Counter,” https://info.hitachi-ics.co.jp/eng/products/
gazoutraffics/index.html
D. Watanabe, F. Kanazawa, and J. Sawa, “Development of a Probe
Information Collection System Using Smartphones,” Proc. of theTechnical Study Group, the Ministry of Land, Infrastructure, Transport and Tourism, pp. 221–226, 2013 (in Japanese).
K. Mori, Y. Yokohata, A. Hayashi, T. Hata, and K. Obana, “Traffic Jam
Detection in Each Lane and Its Evaluation in Field Trials,” Journal of
the Information Processing Society of Japan, Vol. 63, No. 1, pp.
163–171, 2022 (in Japanese).
Website of Japan Road Traffic Information Center, FAQ (in Japanese),https://www.jartic.or.jp/s/attention/inquiry/#map 8
Darknet on GitHub, https://github.com/pjreddie/darknet

81

Feature Articles

Kouhei Mori

Research Engineer, NTT Smart Data Science
Center.
He received an M.E. from University of
Yamanashi in 2009 and joined NTT the same
year. He studied ubiquitous services and edge
computing at NTT Network Innovation Laboratories. From 2015 to 2018, he was involved in thedevelopment of the e-Learning service for corporations and schools at NTT DOCOMO. Since
2018, he has been active in the analysis of datacollected from vehicles for driving support at
NTT Smart Data Science Center.

Yuki Yokohata

Senior Research Engineer, NTT Smart Data
Science Center.
She received an M.S. in earth and planetaryscience from Nagoya University in 2004 andjoined NTT the same year. She studied ubiquitous services at NTT Network Service Systems
Laboratories. From 2008 to 2015, she wasinvolved in the development of public wireless
local area network services at NTT Communications. Since 2016, she has been active in thedevelopment of smart city applications andanalysis of data collected from vehicles for driving support at NTT Smart Data Science Center.

Takahiro Hata

Senior Research Engineer, NTT Smart Data
Science Center.
He received an M.Inf. from Kyoto University
in 2006 and joined NTT the same year. He studied ubiquitous services at NTT Network Innovation Laboratories. From 2010 to 2013, he wasinvolved in the development of public wireless
local area network services at NTT WEST. Since
2013, he has been active in the development ofsmart city applications and analysis of pedestrianflow and driving data of vehicles at NTT Smart
Data Science Center.

Masato Kamiya

Senior Research Engineer, Supervisor, NTT
Smart Data Science Center.
He received an M.E. from Nagoya University
in 1999 and joined NTT the same year. He studied quality of network services at NTT Network
Innovation Laboratories. From 2008 to 2011, he
was involved in the development of a next-generation network system at NTT Communications.
Since 2011, he has been active in the development of smart city applications and analysis ofdata collected from vehicles and vessels at NTT
Smart Data Science Center.

Aki Hayashi

Research Engineer, NTT Smart Data Science
Center.
She received a B.S. and M.S. in information
science from Ochanomizu University, Tokyo, in2010 and 2012 and joined NTT in 2012. She
researched information visualization and data
mining at NTT Service Evolution Laboratories.
She then received a Ph.D. in science from Ochanomizu University in 2015. From 2016 to 2019,
she was involved in the development of voiceagent services at NTT DOCOMO. Since 2019,
she has been active in the development of smart
city applications and analysis of data collectedfrom vehicles for driving support at NTT Smart
Data Science Center.

NTT Technical Review

Vol. 20 No. 7 July 2022

82

Feature Articles: ICT Platform for Connected Vehicles
Created by NTT and Toyota

Technology for Calculating
Suddenness Index for Aggregated
ValuesAki Hayashi, Yuki Yokohata, Takahiro Hata,
Kouhei Mori, and Masato Kamiya
Abstract

We developed a technology for calculating the suddenness index for aggregated values to reduce theamount of computation and communication for video processing needed to detect lane-specific traffic
jams. This technology aggregates the number of connected vehicles for each mesh on a map and quantifies the degree of deviation from the ordinary state. This article gives an overview of this technology anddescribes the value it provides and future issues based on the verification conducted for this lane-specific traffic-jam use case in field trials.
Keywords: statistical analysis, CAN, traffic-jam detection

1. Issues regarding detection of lane-specifictraffic jams
NTT Smart Data Science Center aims to provide
advanced driving support and optimize traffic flow by
detecting lane-specific traffic jams using video analysis, thus providing more detailed traffic-jam information faster than conventional services. To identify theexact location and length of a lane-specific trafficjam, dashcam videos of connected vehicles runningin the lane adjacent to the congested lane are collected into the cloud where image processing [1] is
executed. This process presents a problem in that alarge amount of communication is required to collectthe video data and a large amount of computation isrequired to analyze the data. To solve this problem,
NTT Smart Data Science Center developed a technology for calculating the degree of suddenness of theincrease or decrease in the latest number of vehicles
compared with the number of vehicles in the ordinary
state, which is quantified and called the suddenness
index for aggregation values, or just suddenness
index, SI [2]. It uses controller area network (CAN)
data, which feature a small data size, and identifies
NTT Technical Review

Vol. 20 No. 7 July 2022

roads (or meshes* on a map), the videos of which
need to be collected and analyzed with high priority.
By collecting and analyzing only high-priority videos, it is possible to reduce the amount of both communication and computation.
2. Which roads have high priority forvideo analysis?
Compared with traffic jams in areas where traffic
volumes are predictable, such as areas with littlechange in traffic volume or where traffic jams occur
periodically, lane-specific traffic jams that occur
unexpectedly and suddenly are more likely to trigger
new accidents. Therefore, it is important to provide
advanced driving assistance in areas that are prone tosuch sudden lane-specific traffic jams.
Traffic jams in the former category include thosethat occur on arterial roads and right/left-turn lanes,
which become congested during the morning andevening rush hours, and at entries to commercial*

Mesh: A specific square area on a map. In the field trials, a
150-m2 area was used as a mesh.

83

Feature Articles

facilities that are crowded on weekends and holidays.
Such traffic jams can be predicted on the basis ofhistorical data, and the video-analysis results of pasttraffic jams have already been accumulated. Therefore, the analysis of videos of such areas has low
priority, especially when remaining server resources
are scarce.
Lane-specific traffic jams in the latter category
include those caused by an extreme reduction in traffic capacity on the road due to an accident or vehicle
breakdown, those caused by the opening of a newcommercial facility, and those caused by changes inpeople’s daily habits, such as a sudden rise in demandfor drive-through services due to the COVID-19 pandemic. Such traffic jams are difficult to predict from
historical data. Therefore, it is a matter of high priority to understand the traffic situation in such cases
through video analysis. By giving high priority tolane-specific traffic jams that occur unexpectedly, it
is possible to reduce the amount of both communication and computation required.
3. Issues with detecting sudden traffic jams
The issues with detecting sudden traffic jams are
summarized below.
First, it is necessary to use, as input data, information that is readily available and can be analyzed
without incurring a high cost in communication andcomputation. The proposed technology collects CAN
data from connected vehicles and uses the number ofvehicles in each mesh as input data. It is necessary totake note of the facts that the reliability of data on thenumber of vehicles is low and will remain so untilconnected vehicles become widespread and that it isnecessary to reduce the amount of computationneeded for detecting a sudden traffic jam to improvereal-time performance.
To be able to identify roads encountering sudden
traffic jams, the number of vehicles in the ordinary
state (normal traffic without sudden congestion)
should be learned from data on the number of vehicles in each mesh in each hour segment and the timeseries of that data. In doing so, it is important to take
into account the facts that the ordinary state varies
from road to road and that the number of vehicles
changes periodically, for example, depending on the
day of the week, hour of the day, and combination of
the two.

NTT Technical Review

Vol. 20 No. 7 July 2022

4. Technology for calculating suddenness index
Our technology can quickly narrow down the spot
where a sudden traffic jam is occurring. The input
data it uses are the number of vehicles in each mesh,which is collected with AxispotTM [3] based on CAN
data. This technology can detect the moment when
the number of vehicles suddenly increases in comparison with the number of vehicles in the ordinary
state.
The flow for calculating SI is shown in Fig. 1. To
take multiple types of periodicity into account, thetechnology calculates the ordinary state (hereafter,
referred to as statistical information) of each meshfrom various data: the average, standard deviationand time-scale reliability of the number of vehicles ineach mesh, which are aggregated for the entire period, by the hour of the day, day of the week, and combination of the two (these are collectively called time
granularity). Time-scale reliability is also calculated
to determine, from among the results for different
time granularities, which should be given high priority. This time-scale reliability is designed in such a
way that it is low when the density of connectedvehicles is low; thus, the number of vehicles on which
information is obtained is excessively small relative
to the number of lanes. Specifically, it is calculated on
the basis of the number of times that the number ofvehicles on which information is obtained increases
above a defined threshold. When the latest number ofvehicles is input, SI is calculated. The input data are
the latest number of vehicles (by mesh and by time)and statistical information calculated in advance.
First, the suddenness index by time granularity (SIT)
is calculated by comparing the latest number ofvehicles with the statistical information for each time
granularity. The SIT is calculated using a method
often used in outlier tests [4], as shown in Fig. 1,
where C(t, m) is the number of vehicles passing
through mesh m at time t, μ(T, t, m) is the mean number of vehicles at t, at m for time granularity T, andσ (T, t, m) is the standard deviation of the same. The
SIT is positive if the latest number of vehicles isgreater than the mean and is negative if it is smaller.
The larger the standard deviation of the population,
the smaller SIT is. The SIT is high when a large numberof vehicles suddenly accumulate in a place where thenumber of vehicles is normally small and varies little.
Next, the SIT for each time granularity is multipliedby the time-scale reliability. The weighted sum of the
SIT values for all time granularities is the final output,
called the SI. If the value is positive, the number of84

Feature Articles

OutputInput

Calculate SI for each time granularity

Latest number of vehicles
SIT(T, t, m) =

 (T, t, m)

 (T, t, m)

SI(t, m) = (T, t, m)SI (T, t, m)T

T

Learned statisticalinformationµ (mean number of vehicles), (standard deviation of thenumber of vehicles), (time-scale reliability)

By hour

0

0

Index
value
0

0

Mesh

Mesh
By day of theweek x hour

Mesh

Entire period

C(t, m)By day of
the week

Number ofvehiclesMesh, time, numberof vehicles

SI

C(t, m) − µ (T, t, m)Mesh

Suddentraffic jam
Mesh

Surplus server resources:0

Sufficient:
Mesh

Adopt

Insufficient: Adopt

Eliminate
Eliminate

Fig. 1. Calculation method for SI.

vehicles is higher than usual at many high-reliability
time granularities. This enables flexible selection of
the mesh to study. For example, if the server that collects and processes video has sufficient surplus
resources, the center and left-most meshes of thechart in the right-hand column in Fig. 1 can be selected. If it does not, only the left-most mesh, which has
the highest SI, can be selected.
To observe deviations from the ordinary state, it isnecessary to know the ordinary state. It has been common to use a complex model, such as Auto Encoder[5], to learn the ordinary state. We adopted a different
approach for the lane-specific traffic-jam-detection
use case. To minimize the amount of computation andcomputing time required for prioritizing video-processing-based traffic-jam detection, multiple aggregation time granularities are used, and whether thenumber of vehicles is greater than in the ordinary
state is quantified for each aggregation granularity. A
benefit of this approach is that it minimizes the computation time while minimizing the negative effects
when the reliability of the data is low.
5. Evaluation of accuracy and performance
Since connected vehicles are still in the earlyadoption stage, we believed that a reasonable method
for evaluating the accuracy and performance of the
proposed technology with real data was to use Global
NTT Technical Review

Vol. 20 No. 7 July 2022

Positioning System (GPS) logs. We calculated SI
using GPS data collected from 10 taxis every 10 seconds because the occurrence of a traffic jam would
reduce vehicle speeds, thus increase the number ofGPS logs per unit of time.
A summary of the data used is as follows:• Mesh size: 110 m2• Number of GPS logs: 2,757,003• Number of meshes: 47,146• Number of time frames: 9,258•	Total number of all meshes in the aggregation:
1,016,024•	Period: November 27, 2017–January 31, 2018
Figure 2 shows an example of the SI calculation fora specific mesh for which the highest SI was produced. The SI values are shown in red and the numberof logs in blue. When the number of logs (in blue)rose suddenly at (1), the SI produced with the proposed technology (in red) also increased at (1). This
indicates that the technology can be used to detect a
sudden traffic jam. When traffic jams occurred
repeatedly, as shown with the blue dots at (2) and (3),the SI values shown with the red dots at (2) and (3),decreased. This indicates that the use of SI graduallyeliminates recurring traffic jams of similar magnitude
from the target for traffic-jam detection.
Next, we evaluated whether SI is a reliable indicatorfor detecting sudden traffic jams. We visually detected 121 lane-specific traffic jams in videos taken by85

Feature Articles

Number of aggregated logs

SI for aggregate values

65

20

SI for aggregated values

15

(1)
55
(2)
(1)

(3)

(2)

10

45
(3)
35
25

5

15
0

Number of aggregated GPS logs

Top
index
value

5
−5

−5
Time frame
11/27/2017

1/31/2018

Fig. 2. Example of calculating SI for aggregate values.

Table 1. Identification rate of sudden traffic jams.
Effect ofaggregation on
cost saving(Odaiba)

Effect ofaggregation on
cost saving(Overall)

82.2% reduction(1,187/1,444)

72.6% reduction(344,019/473,723)

Rate ofidentifying
meshes withtraffic jams
(Odaiba)
69.2% identified(9/13)

taxis, labeled each of them as either sudden or not,and checked whether SI is effective for identifying
sudden traffic jams. From the taxi data collected overa period of 66 days, we used the data from the first 33
days as training data. We then checked whether SI
could identify the 7 sudden traffic jams out of the 14
traffic jams that occurred during the remaining 33
days. The results are listed in Table 1. The aggregation time at which SI became greater than 0 was considered the moment when a sudden traffic jam
occurred. The table also shows that, even if 82.2% of
the aggregation data taken around Odaiba (the district
where the demonstration experiment was conducted)
were removed, sudden traffic jams were still identified with a probability of 100%. In contrast, SI identified only 42.9% of chronic traffic jams, which are nottargeted traffic jams in this use case.
This technology was implemented on the platform
NTT Technical Review

Vol. 20 No. 7 July 2022

Rate ofidentifying
sudden trafficjams(Odaiba)
100% identified(7/7)

Rate ofidentifying
chronic trafficjams(Odaiba)
42.9% identified(3/7)

used in the field trials, which is presented in anotherarticle in this issue. We evaluated the proposed technology in this setting. We examined what reduction incomputation time can be achieved if the number ofprocessed meshes (meshes for which video is processed) is reduced by 80% using SI. We examined
this effect with different numbers of meshes. The
results are shown in Fig. 3. As the number of meshes
increased, the greater the effect of decreasing thenumber of processed meshes on reducing computing
time. When the number of meshes was 400 or 800,
the computing time decreased by 70% when using SI.
However, the time required to compute SI remained
as short as 0.5 seconds even with 800 meshes.
6. Future outlook
We developed a technology for calculating the86Feature Articles

0.5

100

0.4

80

0.3

60

70%
reduction

0.2
40%
reduction

0.1

40

70%
reduction

20

0

0
200

400

600

Computing time for the entire platform (s)Time for calculating SI for aggregate values (s)Computing time on the platform

800

Number of meshes
Computing time for calculating SI (red)Overall processing time without prioritization using SI (light blue)
Overall processing time with prioritization using SI (blue)

Fig. 3. Effect of using SI for aggregate values on reducing computing time on the platform.

suddenness index for aggregated values, which canquickly identify sudden increases in time-series datawith a small amount of computation. We plan to use
this technology to detect or predict sudden traffic
jams on expressways caused by accidents on the basisof information about previous traffic jams on expressways.

[3]

References
[4]
[1]

[2]

K. Mori, Y. Yokohata, A. Hayashi, T. Hata, and K. Obana, “Traffic Jam
Detection in Each Lane and Its Evaluation in Field Trials,” Journal of
the Information Processing Society of Japan, Vol. 63, No. 1, pp.
163–171, 2022 (in Japanese).
A. Hayashi, Y. Yokohata, T. Hata, K. Mori, and K. Obana, “Prioritiza-

NTT Technical Review

Vol. 20 No. 7 July 2022

[5]

tion of Lane-based Traffic Jam Detection for Automotive Navigation
System Utilizing Suddenness Index Calculation Method for Aggregated Values,” 60th Annual Conference of the Society of Instrument
and Control Engineers of Japan (SICE2021), Tokyo, Japan, pp. 874–
880, Sept. 2021.
M. Hanadate, T. Kimura, N. Oki, N. Shigematsu, I. Ueno, I. Naito, T.
Kubo, K. Miyahara, and A. Isomura, “Introduction to AxispotTM, Realtime Spatio-temporal Data-management System, and Its High-speed
Spatio-temporal Data-search Technology,” NTT Technical Review,Vol. 18, No. 1, pp. 22–29, 2020.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr202001fa3.html
V. J. Hodge and J. Austin, “A Survey of Outlier Detection Methodologies,” Artificial Intelligence Review, Vol. 22, pp. 85–126, 2004.
S. Nakatsuka, H. Aizawa, and K. Kato, “Defective Products Detection
Using Adversarial AutoEncoder,” Proc. of International Workshop on
Advanced Image Technology (IWAIT) 2019, Vol. 11049, Singapore,Jan. 2019.

87

Feature Articles

Aki Hayashi

Research Engineer, NTT Smart Data Science
Center.
She received a B.S. and M.S. in information
science from Ochanomizu University, Tokyo, in2010 and 2012 and joined NTT in 2012. She
researched information visualization and data
mining at NTT Service Evolution Laboratories.
She then received a Ph.D. in science from Ochanomizu University in 2015. From 2016 to 2019,
she was involved in the development of voiceagent services at NTT DOCOMO. Since 2019,
she has been active in the development of smart
city applications and analysis of data collectedfrom vehicles for driving support at NTT Smart
Data Science Center.

Yuki Yokohata

Senior Research Engineer, NTT Smart Data
Science Center.
She received an M.S. in earth and planetaryscience from Nagoya University in 2004 andjoined NTT the same year. She studied ubiquitous services at NTT Network Service Systems
Laboratories. From 2008 to 2015, she wasinvolved in the development of public wireless
local area network services at NTT Communications. Since 2016, she has been active in thedevelopment of smart city applications andanalysis of data collected from vehicles for driving support at NTT Smart Data Science Center.

Kouhei Mori

Research Engineer, NTT Smart Data Science
Center.
He received an M.E. from University of
Yamanashi in 2009 and joined NTT the same
year. He studied ubiquitous services and edge
computing at NTT Network Innovation Laboratories. From 2015 to 2018, he was involved in thedevelopment of the e-Learning service for corporations and schools at NTT DOCOMO. Since
2018, he has been active in the analysis of datacollected from vehicles for driving support at
NTT Smart Data Science Center.

Masato Kamiya

Senior Research Engineer, Supervisor, NTT
Smart Data Science Center.
He received an M.E. from Nagoya University
in 1999 and joined NTT the same year. He studied quality of network services at NTT Network
Innovation Laboratories. From 2008 to 2011, he
was involved in the development of a next-generation network system at NTT Communications.
Since 2011, he has been active in the development of smart city applications and analysis ofdata collected from vehicles and vessels at NTT
Smart Data Science Center.

Takahiro Hata

Senior Research Engineer, NTT Smart Data
Science Center.
He received an M.Inf. from Kyoto University
in 2006 and joined NTT the same year. He studied ubiquitous services at NTT Network Innovation Laboratories. From 2010 to 2013, he wasinvolved in the development of public wireless
local area network services at NTT WEST. Since
2013, he has been active in the development ofsmart city applications and analysis of pedestrianflow and driving data of vehicles at NTT Smart
Data Science Center.

NTT Technical Review

Vol. 20 No. 7 July 2022

88

Regular Articles

Unsupervised Depth and Bokeh
Learning from Natural Images Using
Aperture Rendering Generative
Adversarial Networks
Takuhiro Kaneko
Abstract

Humans can estimate the depth and bokeh effects from a two-dimensional (2D) image on the basis of
their experience and knowledge. However, computers have difficulty in doing this because they logicallycannot have such experience and expertise. To overcome this limitation, a novel deep generative model
called aperture rendering generative adversarial network (AR-GAN) is discussed. AR-GAN makes it
possible to control the bokeh effects on the basis of the predicted depth by incorporating an opticalconstraint of a camera aperture into a GAN. During training, AR-GAN requires only standard 2D images(such as those on the web) and does not require 3D data such as depth and bokeh information. Therefore,
it can alleviate the application boundaries that come from the difficulty in collecting 3D data. This
technology is expected to enable the exploration of new possibilities in studies on 3D understanding.
Keywords: generative adversarial networks, unsupervised learning, depth and bokeh

1. Introduction
Humans can estimate the depth and bokeh (shallow
depth-of-field (DoF)) effects from a two-dimensional(2D) image on the basis of their experience andknowledge. However, computers have difficulty indoing this because they logically cannot have such
experience and expertise. However, considering that,in the future, robots will be able to move around us
and the real and virtual worlds will be integrated, it
will be necessary to create computers that can act orpresent information on the basis of 3D data such as
depth and bokeh information. Considering that aphoto is one of the most frequently used forms of datafor recording or saving information, understanding
3D information from 2D images will be valuable forvarious 3D-based applications to reduce installation
cost because it enables using easily available 2D
images as input.
Three-dimensional understanding from 2D images
NTT Technical Review

Vol. 20 No. 7 July 2022

has been actively studied in computer vision andmachine learning. A successful approach is to learn
the 3D predictor using direct or photometric-driven
supervision after collecting pairs of 2D and 3D data
[1] or sets of multi-view images [2]. This approachdemonstrates good prediction accuracy due to theease of training. However, collecting pairs of 2D and
3D data or sets of multi-view images is not always
easy or practical because they require special devices
such as a depth sensor or stereo camera.
To reduce the data-collection costs, our team isinvestigating a fully unsupervised approach for learning 3D representations only from images without any
additional supervision. In the study published in the
34th IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR 2021) [3], I introduced a
new deep generative model called aperture rendering
generative adversarial network (AR-GAN), which
can learn depth and bokeh effects from standard 2D
images such as those on the web. Focus cues that are
89Regular Articles

Generated
image

Latentvariablez

Generator
G(z)xgDiscriminatorD(x)Training datap r(x)~

Real or fake

xr

Real
image

Fig. 1. Architecture of GAN.

inherent in photos but had not been actively studied inprevious deep generative models were considered.
On the basis of this consideration, our team developed AR-GAN to incorporate aperture rendering(particularly light field aperture rendering [4]) into a
GAN [5] (a variant of deep generative models). Thisconfiguration allows synthesizing a bokeh image on
the basis of the predicted depth and all-in-focus (deep
DoF) image using a camera with an optical constraint
on the light field.
The rest of this article is organized as follows. In
Section 2, I first review two previous studies on
which AR-GAN is based: GAN [5] and light field
aperture rendering [4]. In Section 3, I explain ARGAN, which is the main topic of this article. In Section 4, I discuss the experiments on the effectivenessof AR-GAN. In Section 5, I present concluding
remarks and areas for future research.
2. Preliminaries
2.1 GANGANs [5] can mimic training data without defining
their distribution explicitly. This property enables
GANs to be applied to various tasks and applications
in diverse fields.
As shown in Fig. 1, a GAN is composed of twoneural networks: a generator G(z) and discriminatorD(x). These two networks are optimized through a
two-player min-max game using an objective function LGAN:
LGAN = 𝔼xr~pr(x)[logD(xr)]+ 𝔼z~p(z)[log(1 − D(G(z)))],

where, given a latent variable z~p(z), a G(z) attempts
to generate an image xg = G(z) that can deceive a D(x)NTT Technical Review

Vol. 20 No. 7 July 2022

by minimizing LGAN. By contrast, the D(x) attempts
to distinguish a generated image xg from a real image
xr~pr(x) by maximizing LGAN. Superscripts r and gdenote the real and generated data, respectively.
Through this adversarial training, a generative distribution pg(x) reaches close to a real distribution pr(x).
2.2 Light field aperture rendering
Light field aperture rendering [4] is a module thatsimulates an optical phenomenon (particularly
bokeh) on a camera aperture in a differentiable manner. Note that such a differentiable property is necessary for deep neural networks (DNNs), such as a G(z)(discussed in Section 2.1), to update the parameters
through the backpropagation commonly used for
DNNs.
More concretely, as shown in Fig. 2, the rendering
provides an aperture renderer R(xd, d) that synthesizes a bokeh image xs(r) from an all-in-focus image
xd(r) and depth map d(r). Here, r indicates the spatial
coordinates of the light field on the image plane.
I explain the details in a step-by-step manner. First,
a depth map d(r) is expanded into a depth map foreach view in the light field, i.e., m(r, u), using a neural network T:
m(r, u) = T(d(r)),
where u indicates the angular coordinates of the lightfield on the aperture plane. Subsequently, an all-infocus image xd(r) is warped into an image for eachview of the light field, i.e., l(r, u), using the predictedm(r, u):l(r, u) = xd(r + um(r, u)).
From this formulation, the left-side images in thelight field (5 × 5 images in Fig. 2) represent images
90Regular Articles

All-in-focus
image Warpxd (r )
T

Depth mapd (r )

Light fieldl(r, u)Aperture
a (u)
u

Bokeh
image
xs(r )

Corresponding
depth mapm (r, u)Fig. 2. Processing flow of light field aperture rendering.

Image
generator
Gx (z)

Latentvariablez

Depth
generator
Gd (z)

All-in-focus
image Warpx gd(r)T

Light fieldl g(r, u)

Aperture
a (u)

u

Bokeh
image
x gs(r )Depth mapd g(r)
Aperture renderer
R (x gd,d g)

Fig. 3. Processing flow of AR-GAN generator.

3. AR-GAN

only from images without additional supervision. In
this setting, it is not easy to construct a conditional
generator that directly predicts the depth or bokeh
effects from an image due to the absence of pairs of
2D and 3D data or sets of multi-view images. Therefore, as an alternative, the aim is to learn an unconditional generator that can generate a tuple of an all-infocus image xdg, depth map dg, and bokeh image xsg
from a latent variable z.
AR-GAN uses focus cues as a clue for addressing
this challenge. When the training images are highlybiased in terms of bokeh effects (e.g., all training
images are all-in-focus), it is difficult to gain focus
cues from the images. Therefore, it is assumed with
AR-GAN that the training dataset includes various
bokeh images. Note that this assumption does notmean that the training dataset contains sets of different bokeh images for each instance. Under thisassumption, AR-GAN learns the generator in a wisdom of crowds approach.

3.1 Problem statement
The problem statement is clarified before explaining the details of AR-GAN. As described in Section
1, AR-GAN is used to learn depth and bokeh effects

3.2 Model architecture
The processing flow of the AR-GAN generator ispresented in Fig. 3. Given a latent variable z, the ARGAN generator generates an all-in-focus image

when viewing objects from the left side, and theright-side images represent vice versa.
Finally, the l(r, u) is integrated using an aperturea(u) (an indicator that represents the disk-shaped
camera aperture and takes ones for views within theaperture (indicated with white in Fig. 2) and zeroes
otherwise (indicated with black in Fig. 2)) to render abokeh image xs(r):xs(r) = ∑ua(u) l(r, u).
When an object is on the focal plane, the object’s
position is consistent regardless of the l(r, u). Therefore, no bokeh occurs when the l(r, u) is integrated by
the above equation. By contrast, when an object isseparate from the focal plane, the object’s position
varies depending on the l(r, u). Thus, bokeh occurs inthis case. Hereafter, r and u are omitted for simplicity
except in necessary cases.

NTT Technical Review

Vol. 20 No. 7 July 2022

91

Regular Articles

Real images

x rd
?
z

G

?

x rd

x rs

Real images

x rs

All-in-focus
image

?
xg

Bokeh
image

Gx

x gd

Gd

dg

x gs

R

z
s
(a) Standard GAN learning(b) DoF mixture learning

Fig. 4. Comparison between standard GAN learning and DoF mixture learning.

xdg = Gx(z) and depth map dg = Gd(z) using an all-infocus image generator Gx(z) and a depth generator
Gd(z), respectively. Subsequently, the aperture renderer R(xd, d) (explained in Section 2.2) synthesizes
a bokeh image xsg = R(xdg, dg). Using this configuration, AR-GAN makes it possible to generate a tupleof an all-in-focus image xdg, depth map dg, and bokeh
image xsg using a camera with an optical constraint on
the light field.
3.3 Training method
As shown in Fig. 1, a typical GAN applies a D(x) to
the final output of the G(z) (i.e., xsg in the case of the
AR-GAN generator). However, in the AR-GAN generator, three modules, i.e., Gx(z), Gd(z), and R(xdg, dg)
are trainable. Therefore, they compete for roles if
there is no constraint. For example, they can fall into
an extreme solution (e.g., R(xdg, dg) learns strong
bokeh effects and Gx(z) learns over-blurred images).
To alleviate this problem, AR-GAN is trained using
DoF mixture learning. Figure 4 illustrates the comparison between the standard GAN learning and DoF
mixture learning. In the standard GAN learning
shown in Fig. 4(a), the G(z) attempts to cover theoverall real image distribution using generated images without any constraint. Consequently, it cannot
determine to make a generated image xg close to areal all-in-focus image xdr or a real bokeh image xrs(indicated with question marks “?” in Fig. 4(a)).
By contrast, as shown in Fig. 4(b), in DoF mixture
learning, the AR-GAN generator attempts to represent the real image distribution using generated
images, the bokeh degrees of which are adjusted by ascale factor s. More concretely, the GAN objectivefunction presented in Section 2.1 is rewritten as follows:
NTT Technical Review

Vol. 20 No. 7 July 2022

LAR−GAN = 𝔼xr~pr(x)[logD(xr)] + 𝔼z~p(z),s~p(s)[log 1 − D(R(Gx(z), sGd(z))) ],()

where s ∈ [0, 1]; when s = 0, an all-in-focus image xdg
is generated, whereas when s = 1, a bokeh image xsg isrendered. Intuitively, the aperture renderer R(xdg, dg),
which has an optical constraint on the light field,
functions as a bokeh image prior. This prior encourages a generated all-in-focus image xdg to approximate a real all-in-focus image xdr (indicated by the“All-in-focus image” in Fig. 4(b)) and promotes a
generated bokeh image xsg to mimic a real bokeh
image xsr (indicated by the “Bokeh image” in Fig.
4(b)). Consequently, dg, which connects xdg and xsg, isalso optimized. In this manner, the DoF mixture
learning allows optimizing Gx(z), Gd(z), and R(xdg, dg)
together under an optical constraint.
A remaining challenge specific to unsupervised
depth and bokeh learning is the difficulty in distinguishing whether blur occurs ahead of or behind the
focal plane. For this challenge, on the basis of theobservation that the focused image tends to be placedat the center of a photo, AR-GAN uses the center
focus prior, which encourages the center to befocused while promoting the surroundings to bebehind the focal plane. In practice, this prior is onlyused at the beginning of training to determine thelearning direction.
4. Experiments
4.1 Image and depth synthesis
The previous AR-GAN study [3] demonstrated theutility of AR-GAN using various natural image datasets, including flower (Oxford Flowers [6]), bird
(CUB-200-2011 [7]), and face (FFHQ [8]) datasets.
92

Regular Articles(a) Trained with flower images+ Bokeh effectDepth+ Bokeh effect

Depth

Depth

+ Bokeh effect(b) Trained with bird images(c) Trained with face images

Fig. 5. Examples of generated images and depth maps.

(a) Input(all-in-focus image)(b) Output 1(bokeh image)(c) Output 2(depth map)

Fig. 6. Examples of bokeh rendering and depth prediction.

The implementation details are omitted because of
space limitations. See that AR-GAN study [3] if
interested in the implementation details.
Figure 5 shows examples of generated images anddepth maps. AR-GAN succeeds in generating a tupleof an all-in-focus image (upper left), bokeh image(upper right), and depth map (lower left) in every setting. For example, in Fig. 5(a), the background isblurred while the foreground is unchanged in bokeh
conversion (the conversion from the upper left toupper right). In depth prediction (the transformationfrom the upper left to lower left), the depth map(lower left) corresponding to the image (upper left) is
successfully predicted. A light color indicates the foreground while a dark color indicates the background.
Recall that the training data are only 2D images, anddepth and bokeh effects are not provided as supervision. In this manner, learning depth and bokeh effects
only from 2D images is the main strength of AR-GAN.
NTT Technical Review

Vol. 20 No. 7 July 2022

4.2 Application to bokeh rendering and depth
prediction
As discussed in Section 3.1, AR-GAN learns an
unconditional generator that generates a tuple of an
all-in-focus image xdg, depth map dg, and bokeh image
xsg from a latent variable z. Therefore, it cannot bedirectly used to convert a given image to the bokeh
image or depth. However, AR-GAN can generate setsof all-in-focus and bokeh images or sets of all-infocus images and depth maps artificially and abundantly by randomly changing the latent variable. By
using these data, we can learn a bokeh renderer (i.e.,a converter that converts an all-in-focus image to abokeh image) and depth predictor (i.e., a predictor
that predicts a depth map from an image) in a supervised manner.
Figure 6 shows example results obtained with thebokeh renderer and depth predictor mentioned above.
A photo I took was used as an input (Fig. 6(a)). The
93

Regular Articles

bokeh renderer synthesizes a bokeh image (Fig. 6(b)),
and the depth predictor predicts a depth map from the
input image (Fig. 6(c)). Similar to the results in
Fig. 5, the background is blurred while the foreground remains unchanged in the bokeh conversion(the conversion from (a) to (b)), and the depth map
corresponding to the input image is predicted in thedepth prediction (the transformation from (a) to (c)).
Note that the data required for training the bokeh
renderer and depth predictor are only the data generated by AR-GAN, and no additional data are needed.
That is to say, in this setting, we can learn a bokeh
renderer and depth predictor in a fully unsupervisedmanner, similar to AR-GAN. This is a strength of an
AR-GAN-based approach.

tions for social media).
Future work includes further improvement of depth
and bokeh accuracy since unsupervised learning ofdepth and bokeh effects is an ill-posed problem, andthere is room for improvement. Our team is tackling
this challenge, and my latest paper [9] has been
accepted to CVPR 2022. Due to space limitations,details of this are omitted. Please check my latest
paper [9] if interested in the details.
References
[1]

[2]

5. Conclusion and future work
This article explained AR-GAN, which is a new
deep generative model enabling the unsupervised
learning of depth and bokeh effects only from natural
images. Since we live in the 3D world, human-oriented computers are expected to understand the 3D
world. For this challenge, AR-GAN is effective
because it can eliminate the requirement of 3D data
during training. AR-GAN is expected to enable theexploration of new possibilities in studies on 3D
understanding.
AR-GAN will also be useful for many applications
in various fields such as environmental understandingin robotics, content creation in advertisements, andphoto editing in entertainment. For example, ARGAN can learn a data-driven model from collected
images. Using this strength, a data-driven bokeh renderer reflecting a famous photographer can be constructed if we can collect his/her photos. Thus, ARGAN can be used to obtain more natural and impactful bokeh images and enrich the functionality ofphoto-editing applications (e.g., smartphone applica-

NTT Technical Review

Vol. 20 No. 7 July 2022

[3]

[4]
[5]

[6]

[7]
[8]
[9]

D. Eigen, C. Puhrsch, and R. Fergus, “Depth Map Prediction from aSingle Image Using a Multi-scale Deep Network,” Proc. of the 27th
International Conference on Neural Information Processing Systems(NIPS 2014), pp. 2366–2374, Montreal, Canada, Dec. 2014.
R. Garg, V. Kumar B. G., G. Carneiro, and I. Reid, “Unsupervised
CNN for Single View Depth Estimation: Geometry to the Rescue,”
Proc. of the 14th European Conference on Computer Vision (ECCV2016), pp. 740–756, Amsterdam, The Netherlands, Oct. 2016.
T. Kaneko, “Unsupervised Learning of Depth and Depth-of-field
Effect from Natural Images with Aperture Rendering Generative
Adversarial Networks,” Proc. of the 34th IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR 2021), pp. 15679–
15688, Virtual, June 2021.
P. P. Srinivasan, R. Garg, N. Wadhwa, R. Ng, and J. T. Barron,“Aperture Supervision for Monocular Depth Estimation,” Proc. of
CVPR 2018, pp. 6393–6401, Salt Lake City, USA, June 2018.
I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial
Nets,” Proc. of NIPS 2014, pp. 2672–2680, Montreal, Canada, Dec.
2014.
M. Nilsback and A. Zisserman, “Automated Flower Classification
over a Large Number of Classes,” Proc. of the Sixth Indian Conference
on Computer Vision, Graphics & Image Processing (ICVGIP 2008),
pp. 722–729, Bhubaneswar, India, Dec. 2008.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie, “TheCaltech-UCSD Birds-200-2011 Dataset,” Technical Report CNSTR-2011-001, California Institute of Technology, 2011.
T. Karras, S. Laine, and T. Aila, “A Style-Based Generator
Architecture for Generative Adversarial Networks,” Proc. of CVPR
2019, pp. 4401–4410, Long Beach, USA, June 2019.
T. Kaneko, “AR-NeRF: Unsupervised Learning of Depth and Defocus
Effects from Natural Images with Aperture Rendering Neural
Radiance Fields,” CVPR 2022, pp. 18387–18397, New Orleans,
USA, June 2022.

94Regular Articles

Takuhiro Kaneko

Distinguished Researcher, Media Information
Laboratory, NTT Communication Science Laboratories.
He received a B.E., M.S., and Ph.D. from the
University of Tokyo in 2012, 2014, and 2020. He
joined NTT Communication Science Laboratories in 2014 and has been a distinguishedresearcher since 2020. His research interests
include computer vision, signal processing, andmachine learning. He is currently working on
image synthesis, speech synthesis, and voice
conversion using deep generative models such as
GANs. He received the Hatakeyama Award fromthe Japan Society of Mechanical Engineers, the
ICPR 2012 Best Student Paper Award in the 21stInternational Conference on Pattern Recognition(ICPR 2012), and the Dean’s Award for Best
Doctoral Thesis from the University of Tokyo.
He is a member of the Institute of Electronics,
Information and Communication Engineers(IEICE) of Japan, the Information Processing
Society of Japan (IPSJ), and the Acoustic Societyof Japan (ASJ).

NTT Technical Review

Vol. 20 No. 7 July 2022

95

Global Standardization Activities

Report of the 9th ITU-T TSAG
(Telecommunication Standardization
Advisory Group) Meeting
Noriyuki Araki
Abstract

The 9th meeting of the Telecommunication Standardization Advisory Group (TSAG) of the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T), the final meeting of the study period (2017–2021) for the World Telecommunication Standardization Assembly(WTSA-20), was held in an online conference from January 10 to 17, 2022. The 4th Inter-regional Meeting, a preparatory meeting for WTSA-20, was also held on January 6 before the TSAG meeting. This
article describes the main results of the 9th TSAG meeting.
Keywords: ITU, TSAG, standardization

1. Introduction
From January 10 to 17, 2022, the 9th meeting of theTelecommunication Standardization Advisory Group(TSAG) of the International Telecommunication
Union - Telecommunication Standardization Sector(ITU-T) was held remotely with 206 participantsfrom 49 countries. From Japan, the Ministry of Internal Affairs and Communications served as the Headof Delegation for Japan, with 15 representatives fromJapanese companies and organizations (National
Institute of Standards and Technology (NICT), NTT,
KDDI, NEC, The ITU Association of Japan, Hitachi,
Mitsubishi, and the Telecommunication Technology
Committee (TTC)).
2. Outline of the TSAG meeting
The 9th TSAG meeting consisted of plenary sessions for overall deliberation and Rapporteur Group(RG) meetings for detailed examination of eachissue, with two online sessions per day. As this was
the final TSAG meeting of the study period (2017–2021) for the World Telecommunication Standardization Assembly (WTSA-20), the closing plenary was
allocated for two days on January 14 and 17.
NTT Technical Review

Vol. 20 No. 7 July 2022

The plenary sessions were led by TSAG Chair
Bruce Gracie (Ericsson, Canada), and the RG meetings were led by four Rapporteurs: RG on Work Programme (RG-WP), RG on Working Methods (RGWM), RG on Strengthening Collaboration (RG-SC),and RG on Review of Resolutions (RG-RR). RG on
Standardization Strategy (RG-StdsStrat) had completed the discussion of major issues, and there wasno discussion at this meeting. From Japan, Ms. Miho
Naganuma (NEC) participated as an RG-WP Rapporteur.
The online conference was held at 13:00–16:00
Geneva time (21:00–24:00 Japan time), with 1 session of 90 minutes and 2 sessions a day as the coretime, taking into account the time difference between
participants.
We used Zoom for this meeting as a remote meeting
tool. The ITU has its own remote meeting tool,
MyMeetings, which is built on open source, but forlarge meetings with more than 200 participants,
Zoom will be used for the time being. From the viewpoint of improving accessibility, captioning in English was provided at all meetings and simultaneousinterpretation in six official languages of the United
Nations was provided at plenary sessions in core
time.
96

Global Standardization Activities

3. Inter-regional Meeting for WTSA
On January 6, the week before the 9th TSAG meeting, the Inter-regional Meeting (IRM), a preparatory
meeting for WTSA, was held by representatives of
six regional telecommunications organizations (theAsia-Pacific Telecommunity (APT), League of ArabStates/Arab Standardization Team, African Telecommunications Union (ATU), European Conference ofPostal and Telecommunications Administrations(CEPT), Inter-American Telecommunication Commission (CITEL), and Reginal Commonwealth in the
Field of Communications (RCC)), and a summary
report was given to the TSAG plenary. Sixty-one proposals for revision or withdraw of existing WTSA
resolutions, 12 proposals for new resolutions, and 17
proposals for revision of ITU-T A-Series Recommendations were submitted as their common proposals to
WTSA-20 for discussion.
The purpose of the IRM is to exchange informationin advance to promote mutual understanding of the
proposals, including a comparative analysis of the
proposals made by regional organizations for resolutions and recommendations of the WTSA.
From APT, Mr. Yoichi Maeda of the TTC presented
a progress report as the chair of the APT WTSA Preparatory Meeting and provided the latest informationon the 29 focal points of the APT Co-Proposals fordeliberation toward the unification of the draft resolutions at the WTSA.
4. Discussions toward WTSA-20
WTSA-20, the highest decision-making meetingfor the ITU-T’s operational policies, was scheduled
to be held in Hyderabad, India in November 2020, but
was postponed twice due to the COVID-19 pandemic. The meeting in India was finally cancelled, and theWTSA was held from March 1 to 9 in 2022 in Geneva, where the ITU headquarters are located. The
venues were the Geneva International ConferenceCenter adjacent to the ITU headquarters and theMontbrillant building of the ITU headquarters.
On February 28, just before WTSA-20, the Global
Standards Symposium (GSS) was held. The theme ofthis GSS was “International standards for enabling
digital transformation and achieving sustainable
development goals,” and WTSA-20 attendees were
expected to participate in the GSS.
Under the current WTSA rules, votes at the WTSA20 plenary session can only be taken by physicalparticipants representing each country, and remote
NTT Technical Review

Vol. 20 No. 7 July 2022

participants cannot participate in the vote. However,in consideration of the impact of the COVID-19 pandemic, participants were able to participate in discussions at all WTSA-20 meetings in a remote online
format and were able to participate in monitoring ofplenary sessions.
5. Meetings of Focus Group, Joint Coordination
Activities, and Ad-Hoc GroupThe activities of the Focus Group (FG), Joint Coordination Activities (JCA), and Ad-Hoc Group (AHG)
were reported. The results of the FG’s deliberations
will be transferred to the relevant Study Groups(SGs), which will be considered as new issues handled by the ITU-T. This will have an impact on the
structure of Questions for each SG in the next study
period and will be important to understand the trendsof future deliberations.
(1)	FG on Quantum Information Technology for
Networks (FG-QIT4N)
The final activity report of FG-QIT4N on quantum
information and communication technology was presented, and the report was approved. This FG wasestablished in September 2019 under the leadershipof TSAG and worked until December 2021, producing nine outcome documents. To promote the specificstandardization of the outcome document, FG members proposed to plan a briefing session for each SG
to facilitate the transition of the outcome document to
the related SG and agreed to send a liaison document
to all SGs to that effect.
The outcome documents will be compiled into nine
documents, including glossary terms, use cases, thequantum key distribution network protocol, and network technology. They will be transferred to SG13,
SG17, SG11, and SG15, which are closely related toeach other, for further study. This FG was managed
by three co-chairs from Russia, the United States, andChina.
(2)	JCA on Digital COVID-19 Certificates (JCADCC)
Regarding the JCA-DCC whose establishment wasagreed at the previous TSAG meeting, it was agreed
to issue a liaison document to publicize the establishment of the JCA, the purpose of its activities, and thedeadline for applications to participate in the JCA ofthe end of February. The scope of the JCA is to coordinate the work of standardizing digital COVID-19
certificates among the relevant ITU-T SGs, externalorganizations, and forums; promote compatible data97

Global Standardization Activities

architectures for data sharing; and promote interoperability, agility, and security for all parties involved
with users. The JCA-DCC is chaired by Mr. Heung
Youl Youm (Korea, Chairman of SG17), and the firstmeeting will be held electronically in May 2022.
(3)	FG on Testbeds Federations for IMT-2020 andbeyond (FG-TBFxG)
The establishment of an FG-TBFxG was reported,with SG11, which handles signaling protocols andtest specifications, as its parent. This FG functions as
a platform to harmonize testbed specifications
between standards developing organizations/Fora,develops an application programming interface (API)along the testbed federation reference model definedin Recommendation ITU-T Q.4068 developed in collaboration with European Telecommunications Standards Institute (ETSI) Technical Committee on Core
Network and Interoperability Testing, and defines a
set of federated testbeds and API use cases.
6. RG meeting
The following is a summary of the main issues discussed in the RG established at TSAG. The report ofeach RG is approved at the closing plenary of TSAG
and reflected as part of the TSAG meeting report at
WTSA.
(1) RG-WP
The RG-WP is a group that deals with issues related
to SG restructuring, and was given a two-session timeframe for deliberation. This RG reviewed all SG
activity reports, sought endorsement of the proposed
agenda in plenary, and compiled discussions on SG
restructuring toward WTSA-20.
It was assumed that the SG configuration would bemaintained at the current configuration with 11 SGs
at the WTSA-20. Full-scale reorganization will be
discussed in the lead-up to WTSA-24. To accelerate
this discussion, the Correspondence Group (CG) wasestablished under the RG-WP to analyze and study
the optimal distribution of SG configuration. Mr.
Philip Rushton (Department for Digital, Culture,
Media & Sport, United Kingdom), Chairman of CG,
reported the action plan of CG and agreed on theaction plan for analysis of SG restructuring.
The action plan for the SG restructuring analysis
was discussed during the meeting with two additional
editing sessions. This action plan aims to thoroughly
review the potential restructuring options of the
ITU-T based on empirical analysis, with a view to

NTT Technical Review

Vol. 20 No. 7 July 2022

approving the SG restructuring plan at WTSA-24. In
advancing the action plan, it was agreed that the definition of the key performance indicators (KPIs)/metrics to be collected would be clarified, the prioritiesof the various KPIs/metrics to be collected and thetiming of implementation of KPIs/statistics would beclarified, and consideration would be given to thefunds to be considered.
(2) RG-WM
The RG-WM reviews the WTSA Resolution 1 and
A-Series Recommendations (Resolution 32, Recommendation A.1, Recommendation A.7, Recommendation A.8, etc.), which stipulate various ITU-T work
procedures and rules. There were many issues todiscuss at this meeting, and the RG-WM was allocated a three-session time frame. Regarding the proposal to revise Chapter 5.3 of the JCA proposed by
Korea in ITU-T Recommendation A.1, “Working
Method of ITU Telecommunication Standardization
Sector,” the necessity was not fully understood basedon the discussions at the RG-SC, and the proposal torevise JCA was not agreed upon.
(3) RG-SCThe RG-SC is studying ways to strengthen cooperate with other standards bodies and measures. At thismeeting, we discussed liaison activities to strengthen
cooperation between sectors within the ITU and otherstandardization organizations such as International
Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Joint TechnicalCommittee 1 and oneM2M. It was agreed in plenary
on the revised draft recommendation of A.5 “Generalprocedure for including references to other organizations’ documents in ITU-T Recommendations” as a
WTSA-related issue after 1 drafting session, and it
was decided to propose the revised draft recommendation to WTSA-20. It was also agreed on a draft
recommendation to revise ITU-T Recommendation
A. 23, “Cooperation with the International Organization for Standardization on Information Technology(ISO) and the International Electrotechnical Commission (IEC) - Appendix II: Best Practices” andproposed it to WTSA-20.
7. Schedule of future meetings
The first meeting of TSAG for the new study period(2022–2024) is scheduled to be held in Geneva on
December 12–16, 2022.

98

Global Standardization Activities

Noriyuki Araki

Director, Standardization Office, Research andDevelopment Planning Department, NTT.
He received a B.E. and M.E. in electrical andelectronic engineering from Sophia University,
Tokyo, in 1993 and 1995. He joined NTT AccessNetwork Service Systems Laboratories in 1995,
where he researched and developed operationand maintenance systems for optical fiber cable
networks. He has been contributing to standardization efforts in ITU-T SG6 since 2006. He was
the rapporteur of Question 6 of ITU-T SG6 from2006 to 2008 and the rapporteur of Question 17
of ITU-T SG15 from 2008 to 2012. He also
served as the chairman of the ITU-T Focus
Group on Disaster Relief Systems and Network
Resilience and Recovery. He was the vice-chairman of ITU-T SG15 from 2013 to 2022. He also
contributes to the activities of IEC TechnicalCommittee 86 (fiber optic systems). He received
the ITU-AJ award from the ITU Association ofJapan in 2017. He is a member of the Institute ofElectronics, Information and Communication
Engineers (IEICE).

NTT Technical Review

Vol. 20 No. 7 July 2022

99

External Awards
IPSJ Outstanding Paper Award
Winners: Takashi Koide, NTT Security Japan; Daiki Chiba, NTT
Security Japan; Mitsuaki Akiyama, NTT Social Informatics Laboratories; Katsunari Yoshioka, Yokohama National University; Tsutomu
Matsumoto, Yokohama National University
Date: March 30, 2022
Organization: Information Processing Society of Japan (IPSJ)
For “Understanding the Fake Removal Information Advertisement
Sites.”
Published as: T. Koide, D. Chiba, M. Akiyama, K. Yoshioka, and T.
Matsumoto, “Understanding the Fake Removal Information Advertisement Sites,” Journal of Information Processing, Vol. 29, pp.
392–405, 2021.
IEEE Senior Member
Winner: Takayuki Ogasawara, NTT Basic Research Laboratories
Date: April 30, 2022
Organization: The Institute of Electrical and Electronics Engineers(IEEE)IEEE Senior Membership is an honor bestowed only to those whohave made significant contributions to the profession.
Honorable Mention Award
Winners: Jack Jamieson, NTT Communication Science Laboratories; Daniel A. Epstein, University of California Irvine; Yunan Chen,
University of California Irvine; Naomi Yamashita, NTT Communica-

NTT Technical Review

Vol. 20 No. 7 July 2022

tion Science Laboratories
Date: May 5, 2022
Organization: ACM Conference on Human Factors in Computing
Systems (CHI) 2022
For “Unpacking Intention and Behavior: Explaining Contact Tracing App Adoption and Hesitancy in the United States.”
Published as: J. Jamieson, D. Epstein, Y. Chen, and N. Yamashita,“Unpacking Intention and Behavior: Explaining Contact Tracing
App Adoption and Hesitancy in the United States,” Proc. of CHI
2022, New Orleans, USA, Apr./May 2022.
Best Paper Award
Winners: Koji Yamamoto, Kyoto University; Takayuki Nishio,
Kyoto University; Masahiro Morikura, Kyoto University; Hirantha
Abeysekera, NTT Access Network Service Systems Laboratories
Date: May 7, 2022
Organization: The Institute of Electronics, Information and Communication Engineers (IEICE) Communications Society
For “Stochastic Geometry Analysis of Inversely Proportional Carrier Sense Threshold and Transmission Power for WLAN Spatial
Reuse.”
Published as: K. Yamamoto, T. Nishio, M. Morikura, and H. Abeysekera, “Stochastic Geometry Analysis of Inversely Proportional
Carrier Sense Threshold and Transmission Power for WLAN Spatial
Reuse,” IEICE Trans. Commun., Vol. E104.B, No. 10, pp. 1345–
1353, 2021.

100