NTT Technical Review

December 2016 Vol. 14 No. 12
View from the TopHiroyasu Asami, Senior Executive Vice President, NTT DOCOMO

Feature Articles: 2020 Showcase2020 Showcase—Providing Japan and the Rest of the World with State-of-the-art Technology
for 2020

2020 Airport/Station—Hospitality for Foreign Visitors at Airports and Train Stations2020 Town—Web Design Converter for Providing Guidance Assistance to Individuals in
Cities2020 Town—Developing MACHINAKA Service, a Device Integration Service that Utilizes
Artificial Intelligence Technology2020 Public Viewing—Kirari! Immersive Telepresence Technology2020 Entertainment—A New Form of Hospitality Achieved with Entertainment × ICT2020 MICE—New Hospitality through Exhibitions × ICTPromotion of Co-innovation through Collaboration with Different Business Sectors

Regular Articles

Milagro Multi-Factor Authentication

Path Loss Models for Wireless Access Network Systems Using High Frequency Bands

Global Standardization Activities


Report on Apache Big Data North America 2016 and Spark Summit 2016

Short Reports

Deployment of NTT’s Open Source Software GoBGP in INTERNET MULTIFEED’s JPNAP
Service Achieves Automation of Operation and Vast Improvement in Efficiency

New NTT Colleagues


Nefos

External Awards/Papers Published in Technical Journals and Conference ProceedingsExternal Awards/Papers Published in Technical Journals and Conference Proceedings

View from the Top

Creating Extra Value with a New
Medium-term Plan toward 2020—
Refining Intuition throughExperience, Feeling, and Belief
Hiroyasu Asami, Senior Executive Vice
President, NTT DOCOMO
Overview

Feature phones and smartphones have become an indispensable part ofour lives. How can NTT DOCOMO, a company that has a strong connection with its customers, seize the opportunities to leap forward into thesociety of the future? We sat down with Hiroyasu Asami, Senior Executive
Vice President, to learn about the present role of NTT DOCOMO and its
strategies for business expansion, and to hear about a work philosophy thatis essential for making predictions about the future.
Keywords: co-creation, 5G, artificial intelligence

Formulating a new medium-term plan toward2020 after assessing the limits of business growth—Mr. Asami, in light of the initiatives you have personally been involved with to date, please tell us
about the current business conditions surrounding
NTT DOCOMO.
After moving to NTT DOCOMO from the NTT
holding company in 2003, I oversaw the planning anddevelopment of services and content for over ten
years. This is a very rare background for vice presidents of technical operations at NTT DOCOMO.
I was always attracted to the unique characteristics
of NTT DOCOMO from my vantage point in the
NTT holding company. While the target of fixed-line(landline) telephone services is households and businesses, the target of mobile phone services is even1

finer, that is, the individual. Since coming to NTT
DOCOMO, I have placed importance on this featureof having a direct channel to each and every customer, and I have strived to make it a strength in our business operations.
Actually, I came to NTT DOCOMO during the
rapid growth period of the i-mode mobile Internet
service. It was a time when the company was expanding its business model of delivering the content provided by content providers (other companies) to ourcustomers and generating revenue through commissions. However, with the coming of smartphones, it
was inevitable that this business model would undergo a major change. This is because in the smartphone
world, smartphone OS (operating system) makers
such as Google were beginning to construct platforms
that would provide customers with applications andcontent in much the same way as the i-mode business
NTT Technical Review

View from the Top

model.
Sensing the limits to our i-mode business model,
we at NTT DOCOMO thought we needed to take up
a new business challenge. The decision was made thatNTT DOCOMO itself would provide services andcontent, and we launched “dmarket” in 2011. The
NTT DOCOMO dmarket provides a variety of services such as dvideo (now called dTV), dhits, anddmagazine, all of which are presently being enjoyedby about 15 million customers, reflecting genuine
growth.
To digress a bit, in 1980 when I was a new employee, I was involved in the development of a video service in which we attempted to use what was then thelatest technology for a mechanism for video on
demand. This was a truly analog mechanism consisting of 12 video players and featuring remote operation of a mechanical arm for inserting videotapeswith movies and other content into these players. Up
to 12 people could simultaneously view whatever
they wanted to watch. This mechanism has sinceevolved into a digital and software-based platform,
thanks to technological advances, and it has growninto a DOCOMO service with about 5 million subscribers. To see this cherished idea of video services
come to fruition after 30 years of being in the company is truly gratifying for me.
About five years have passed since NTT DOCOMO
set out to provide dmarket services. However, as
General Manager of the Corporate Strategy andPlanning Department, a few years ago I was beginning to think that the things we could accomplish on
our own were approaching a limit. With this in mind,
last year we took up a variety of important themes
such as how to leverage research and development(R&D) results, how to approach corporate sales business, and how to create services for the mass market.
As a result, we hammered out a new vision under the
keywords co-creation and +d (plus d) in which NTT
DOCOMO would partner with other enterprises andleverage our own business assets with the aim of creating new services. Since then, we have begun +d
initiatives with over 150 partners including Lawson(convenient store), McDonald’s Japan, andTakashimaya (department store), and we have begun
to see results (Fig. 1).
—Please tell us about the company’s medium-term
initiatives and the centerpiece of business expansion
toward 2020.
We endeavored to change our business model toVol. 14 No. 12 Dec. 2016

keep up with the times, but we suffered a significant
drop in revenue with the introduction of a new billingplan in 2014. Consequently, we announced three-year
medium-term targets for the period from fiscal year
(FY) 2015 to 2017 (April 1, 2015 to March 31, 2018)
toward profit recovery at the time of the release of the
FY2014 first-half results. We succeeded in reducing
the costs by improving efficiency and making ourdmarket strategy work, and as a result, we will meet
our targets at the two-year point in FY2016.
We are now in the process of drawing up a new
medium-term plan for the period FY2017 to 2020.
We consider this plan to be the centerpiece of NTT
DOCOMO’s new management team under Kazuhiro
Yoshizawa, our new president, who took office inJune 2016. This is a plan that everyone can look forward to.
Two important keywords in the new medium-termplan are “2020” and “5G,” which refer to what wewant to do for the major international event to be heldin 2020 and how we will provide services as a carrier
via 5th generation mobile communications (5G),
scheduled to be launched around that time. Here, ouraim is not to create services for 2020 but rather tohave them already in use by 2020.
I would also like to change the image that customers have of NTT DOCOMO. The business of NTT
DOCOMO has traditionally been centered on theprovision of mobile phone network services, but weare now providing “docomo Hikari” as our own
fixed-line broadband service made possible throughwholesale purchasing of FTTH (fiber to the home)
services from NTT EAST and NTT WEST. We arealso providing dmarket services such as dTV to customers who are not using NTT DOCOMO mobile
phones. Similarly, customers can use NTT
2

View from the Top

Cocreation

Collaborate
with partners.

Provideservices.

ServiceproviderAdd
partners.

Capitaltie-ups

Platform
providerProvidefunctions.

5G

Mobilecarrier

Feature phone era

Smartphone era

Fulfillmission.

ApproachFig. 1. Business model evolution.

DOCOMO’s “d point” loyalty program regardless ofwhether they have an NTT DOCOMO service contract. The traditional customer management method
based on mobile phone subscriptions is reaching its
limits.
Capturing the opportunities brought by this change,
we would like to treat all customers of the variety ofservices provided by NTT DOCOMO as members
who are tied to a unified system of d point, regardlessof whether they have a mobile phone contract with us.
In other words, when we carry out our medium-term
plan, I want to change the awareness internally toredefine customers as “members” using DOCOMO

3

services, who in the past were just “subscribers.” Asa member, a person can continue to be an NTT
DOCOMO customer even if he or she no longer hasa mobile phone contract. Even those residing abroad
can use NTT DOCOMO services as long as they are
members. This is one example of global expansion. In
short, I would like to stress the benefits of DOCOMO
so people will say, “Wow, becoming a DOCOMO
member is really convenient and brings good value.”
Meanwhile, an important element of implementing
our medium-term plan is R&D. We are focusing ourefforts on artificial intelligence (AI) and 5G. For
example, DOCOMO services are used by a very large
number of people, so we envision the use of
DOCOMO AI technology to analyze the big data
generated and thereby deliver more value to customers to enrich their lives. In addition, as 2020 is expected to coincide with a large number of foreign visitors
to Japan, we can envision the use of our AI technology to achieve various types of language processing
such as automatic translation, speech recognition,and speech synthesis to help overcome language barriers. The term “5G” indicates evolution of the network, which is the core technology of DOCOMO,
and initiatives are needed that leverage its features
such as large capacity, high speed, and low latency.
People have smartphones with them 24/7, and we
can expect them to become more important in theyears to come. Compared with feature phones of the
NTT Technical Review

View from the Top

Hardware

Small, light, sturdy
Extended battery life (low power consumption)
Low price
High definition (video), high quality (audio)
Userinterfaces

Ultimate
personal
agent

NetworkSensors

High speed
Large capacity
Low latency
Positioning, geofencing
Drone control

Speech/image recognition, speech synthesis
Natural dialog, machine translation
High resolution, high audio quality
AR, VR, 3D
Security (biometric authentication)

AI
Cloud
Recommendations, big data
Processing
Security (payments, safety)
Edge computing
Weather: temperature, humidity, pressure
Positioning: latitude, longitude, geomagnetism
Dynamic systems: acceleration, gravity
Biological systems: iris, fingerprints, pulse, body temperature, blood pressure, breath, sweat

Fig. 2. Making the smartphone into the ultimate personal agent.

past, today’s smartphones are much like a supercomputer in the palm of your hand, but if they evolve evenfurther, we can expect the service domain to broaden
even more. This, however, will require the development of various types of technology. First, from a
hardware perspective, handsets must be lighter, battery capacity must be greater, and displays must be
high definition. There will also be a need for virtual
reality (VR), augmented reality (AR), and speech
recognition in user interfaces as well as support for ahigh-speed network, built-in AI capabilities, andadvanced sensors. I believe that this sort of evolution
will turn the smartphone into a “personal agent.” Of
course, this means the ultimate personal agent thatcan be used not only in the home, but outside wherever the user goes as well (Fig. 2).
For future initiatives, we cannot forget the importance of making social contributions using information and communication technology (ICT) in
response to all sorts of issues. For example, ICT
could be used to improve Japan’s competitiveness at
the national level, revitalize local communities anddeal with the issues of the low-birth rate and the aging
society at the regional level, make operations more
efficient at the corporate level, and add value toeveryday life at the personal level. As we work to
fulfill the +d concept, we won’t be smug; we intend
Vol. 14 No. 12 Dec. 2016

to maximize the use of DOCOMO’s business assets
and co-create new services with a diverse range ofpartners. Initiatives making use of R&D results andcollaborative activities with local governments are
already underway (Fig. 3).
“Intuition” is the sum total of experience. The“right answer” is found from experience.
—Using new technologies and mechanisms to create
the future sounds exciting. What kind of approach isimportant in uncovering these technologies andmechanisms?
Experience, feeling, and belief are important. For
example, when a new service is launched, it is notenough to just study a comparison chart. It is also
important to have a “hands-on” experience. And
while it would be time-consuming and unrealistic to
try out all aspects of the new service, trying out atleast its key points would enable one to exchange
opinions and make a sound decision. I myself recently tried out Pokémon Go on my own. By actuallyplaying the game myself, I was able to get a feel forthe idea behind it and sense the merits of the service
and points where it could be improved. I was able toget the information necessary for making a decision4View from the Top

Country

Improve national competitiveness.

Society

Solve social problems.

Region

Revitalize local communities.

Industry

Carry out structural reform.

Business

Achieve corporate efficiency.

Individual

Add value to everyday life.

Fig. 3. Expectations of ICT.

by myself.
Looking for the right answer alone will simply leadto a dead end. And no one knows what things will belike in five years’ time. Making a judgment solely on
what you heard from others is also dangerous. That’s
why I try to interact with as many people as possible.
Through communication, one can get hints for creating new ideas and build up a storehouse of knowledge, which in turn will have the effect of improving
decision-making skills. In contrast, a book can only
include material on what has occurred up to the timeof its writing. For this reason, I only use it to confirm
what has already occurred.
When I was involved in service development in thepast, I made it a point to communicate with partners
who had no direct relationship with a potentially newservice. They had an understanding of the way theworld works from their own perspective. Of course,
their forecasts were sometimes off, but they had
information far more accurate than what we had, andthey were able to make predictions. From these experiences, I believe that interacting with other people
will naturally help me see the right answers, so it isimportant to have a network in which we can consider what the right answer is.
In this way, we assessed the limits of i-mode-based
growth and made a major decision to go in the direc5

tion of a smartphone-based business within a newframework called “dmarket.” For NTT DOCOMO, acarrier that had not traditionally provided content
services on its own, that decision was a major turning
point. Furthermore, by providing these services evento customers without DOCOMO subscriptions, there
was a need to redefine the “customer.” For this reason, and with future business expansion in mind, we
made a major decision to shift from the concept of“subscriber” to “member” as a new definition of customer. To put it briefly, the key is to make a soundjudgment on what you are thinking about based onyour experience, feeling, and belief. And make sure
you execute actions appropriately. However, making
such a major transformation requires power.
Learning something about “people” throughexperience, feeling, and belief—What would you like to say to everyone in R&D?
Please learn about “people.” I say this because
people do not necessarily behave in a logical manner.
For example, when in public, people’s actions do notalways match their thinking. Also, when presented
with a service menu having three pricing levels, people tend to choose the level in the middle. In this way,
NTT Technical Review

View from the Top

do. In addition, keep in mind the importance of experience, feeling, and belief, and practice it in your daily
life. As I touched upon earlier, pursuing a majortransformation might mean coming up against a wall.
In this regard, I would like to end by introducing toeveryone one of my favorite sayings: “Ryu-kan
Go-dou,” meaning if you hit a wall, work hard andsweat it out; the way will be revealed.

please pursue R&D themes based on the assumptionthat people are not always logical in their actions, andstrive to create services and supporting technologies
for people who can be somewhat ambiguous in their
behavior.
—Mr. Asami, please leave us with a message for all
NTT DOCOMO employees.
For our younger employees, always think in firstperson terms; what is it that makes you happy andcontent? Apply this to daily work. Furthermore, do
not hesitate to voice what you yourself would like toVol. 14 No. 12 Dec. 2016

Interviewee profile
Career highlights
Hiroyasu Asami entered Nippon Telegraph andTelephone Public Corporation (now NTT) in
1980 and moved to NTT DOCOMO in 2003. He
has served as Senior Vice President, General
Manager of the Smart Communication Service
Department; Executive Vice President, Executive
General Manager of the Smart-life Business
Division; and Member of the Board of Directors,
Executive Vice President, General Manager of
the Corporate Strategy and Planning Department.
He took up his current post as Senior Executive
Vice President, Chief Information Officer, Chief
Information Security Officer, Chief Privacy
Officer, and Member of the Board of Directors inJune 2016.

6

Feature Articles: 2020 Showcase

2020 Showcase—Providing Japanand the Rest of the World with
State-of-the-art Technology for 2020
Ryuji Kubozono
Abstract

The international sporting events that will be held in Tokyo in the year 2020 will be seen as a great
opportunity to showcase our new innovations to Japan and the rest of the world. The Feature Articles inthis issue focus on the 2020 Showcase—a trial held by NTT Service Evolution Laboratories to introduce
the new technologies we are developing in preparation for 2020 and to accelerate the introduction ofthese technologies into society. We will also introduce some of the business aspects of this initiative.
Keywords: 2020 Showcase, epoch-making service, social innovation

1. Introduction
The major sporting events to be held in Tokyo in2020 are expected to provide a major boost to Japan’s
economic recovery. These events will give us achance to showcase Japan’s information and communication technology (ICT) to the rest of the world,and by enhancing our ICT services and infrastructure,
we hope to impress visitors from around the world
with our technological prowess. They will also create
a diverse legacy including an improved social infrastructure and the creation of new business, which willcontinue to create new opportunities in the future.
The Cabinet Office is also actively involved in preparations for the games and is undertaking various studies by creating a task force concerned with scienceand technology innovation [1].
At NTT’s laboratories, we are researching anddeveloping a wide range of technologies for 2020,
including new information display technology basedon innovative user interface and media technology,
and stress-free mobility assistance technology basedon the analysis of big data. In particular, at NTT Service Evolution Laboratories, we are conducting
research and development (R&D) to promote themes
such as shared excitement, optimal navigation, supported growth, and user-friendly service design with1

the aim of providing visitors to Japan in 2020 with an
unparalleled level of hospitality and service [2].
In addition to our R&D efforts, we are also working
to verify the results of our R&D and promote thespread of this technology by collaborating with businesses in other fields to create and develop new value.
The Feature Articles in this issue introduce the 2020
Showcase, an initiative aimed at holding public demonstrations where people can experience this newtechnology for themselves in venues such as airports,
railway stations, stadiums, museums, art galleries,and exhibition halls. Some of the current businesstrends are also discussed.
2. 2020 Showcase
For the 2020 Showcase, we aim to hold practicalfield tests with potential business partners, and we
also aim to construct case and role models. The fields
in which we will conduct demonstration tests arebeing strategically chosen based on a consideration of
the flow of tourists visiting Japan, their preferences/interests, and their consumption habits (Fig. 1). This
article discusses the characteristics of each field andthe technologies that will be needed for 2020 andintroduces the work that we are doing to address theseneeds.
NTT Technical Review

Feature Articles

Town
Airport/Station

Stadium

Hall/Convention center

Station/Airport

Fig. 1. 2020 Showcase demonstration test fields.

2.1 Airports and stations
The first place people see when visiting Japan is the
airport. They may also frequent train stations during
their travels. As gateways to Japan, airports and train
stations require technology that makes it easy forforeigners to access information despite the language
barrier and the cultural differences associated withactivities in unfamiliar places such as traveling andeating out.
To address these issues, we are developing technology that can provide information via an augmented
reality interface by using technology that can recognize images and other objects and technology that canpresent information such as maps and routes clearlyand concisely, both indoors and outdoors, with clear
voice guidance and signs to guide the flow of crowds.
We are also conducting demonstration tests at key
locations in Japan, including Haneda Airport andTokyo Station. For further details, including details ofa test we are conducting in collaboration with NTT
DATA at Namba Station, see the article “2020 Airport/Station—Hospitality for Foreign Visitors at Airports and Train Stations” [3] in this issue.
2.2 Towns
Eating Japanese cuisine, sightseeing, and shopping
are some of the top activities that foreign visitors
want to experience in Japan. Food and drink outlets inurban areas need to be able to display information in
a way that is accessible not only to a growing numberof foreign tourists but also to Japanese people withparticular needs such as the elderly. There is consequently a need for technology that can transform
existing content into a representation that can be easVol. 14 No. 12 Dec. 2016

ily understood by customers, not only according to
the aims of these services but also to the age and otherpersonal attributes of their customers.
To address these issues, we are developing technology that automatically transforms the designs ofexisting web content according to the attributes ofcustomers, and technology that automatically createssupplemental information. Further details can befound in the article “2020 Town—Web Design Converter for Providing Guidance Assistance to Individuals in Cities” [4].
To develop stronger customer interaction in tourist
areas and local shopping malls, there is a need fortechnologies and systems that can stimulate communication and provide support for suitable guidanceand customer actions after an ad hoc appraisal of thecustomer’s situation and requirements. To addressthese issues, we are developing technologies thatfacilitate collaboration via the cloud between various
devices such as sensors and communication robots.
We are also promoting demonstration tests in cities
and other places where people gather in large numbers. Further details, including a discussion of thetests we are conducting jointly with NTT Communications and NTT EAST, can be found in the article“2020 Town—Developing MACHINAKA Service, aDevice Integration Service that Utilizes Artificial
Intelligence Technology” [5].
2.3 Stadiums
At international sporting events, large numbers ofvisitors including foreign visitors and first-time spectators make their way to a stadium. There is therefore
a need for technology that can help people move2

Feature Articles

around efficiently and safely even at crowded, largescale events. Also, for spectators in remote locations,
it is important to convey the atmosphere inside thestadium and to bring a sense of excitement and unity
to them that is similar to what the spectators at thestadium are experiencing. This calls for the development of ultra-realistic services that allow people indifferent locations to feel as if they were actuallypresent in the stadium.
We have been developing techniques for predicting
and guiding the flow of crowds of people, and immersive telepresence technology for the realization ofreal-time ultra-realistic broadcasting. We are also
conducting experiments to demonstrate how the flowof crowds at event venues can be predicted and controlled [6] and immersive telepresence experimentsthat simulate live broadcasts. Details of our immersive telepresence technology can be found in thearticle “2020 Public Viewing—Kirari! Immersive
Telepresence Technology” [7] in this issue.
2.4 Halls and convention centers
Since foreign visitors to Japan will want to makethe most of their time in this country, it is assumed
that they will also make their way to event halls andconference centers in order to participate in various
cultural events and exhibitions. At the venues of
events and exhibitions, there is a need for technologies and systems to support trouble-free viewings andvisits, and technologies to enliven the events themselves with various types of stagecraft. There will
also be a need for technologies and systems that support event operators such as displaying in real time
the usage status of networks and applications.
With these issues in mind, we have worked at
expanding the enjoyment of kabuki theater and providing completely new kinds of kabuki performances
by fusing the latest ICT with traditional performing
arts. We have also worked to enable people to experience the latest technology through official applications at NTT R&D Forum 2016 held in February
2016. Further details can be found in the article “2020Entertainment—A New Form of Hospitality Achieved
with Entertainment × ICT” [8], and “2020 MICE—
New Hospitality through Exhibitions × ICT” [9].
3. Future development
A key point in promoting the 2020 Showcase has
been collaborating with other businesses in different

3

fields. For more details, see the article “Promotion of
Co-innovation through Collaboration with Different
Business Sectors” [10].
To create epoch-making services to support theevents of 2020, the NTT Group will continue with
R&D in new collaborations with different industries,
promote the incorporation of these services in society
in general, and create a legacy that will continue into
the future.
References[1]

Cabinet Office, Government of Japan, “Task Force on Science, Technology, and Innovation for the 2020 Tokyo Olympics and Paralympics” (in Japanese).
http://www8.cao.go.jp/cstp/tyousakai/olyparatf/index.html[2] H. Jozawa, S. Nishioka, and K. Hidaka, “Creating Epoch-making
Services that Excite Users,” NTT Technical Review, Vol. 13, No. 7,
2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201507fa1.html[3] Y. Ichikawa, Y. Nakamura, T. Nakamura, H. Tezuka, H. Seshimo, S.
Fukada, and H. Mitsui, “2020 Airport/Station—Hospitality for Foreign Visitors at Airports and Train Stations,” NTT Technical Review,Vol. 14, No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa2.html[4] K. Nakahama, Y. Morinishi, M. Watanabe, S. Teranaka, and R.
Mochizuki, “2020 Town—Web Design Converter for Providing Guidance Assistance to Individuals in Cities,” NTT Technical Review, Vol.
14, No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa3.html[5] T. Mochizuki, M. Norota, and K. Suga, “2020 Town—Developing
MACHINAKA Service, a Device Integration Service that Utilizes
Artificial Intelligence Technology,” NTT Technical Review, Vol. 14,
No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa4.html[6] Press release issued by NTT on January 20, 2016 (in Japanese).
http://www.ntt.co.jp/news2016/1601/160120a.html[7] A. Akutsu, A. Ono, H. Takada, Y. Tonomura, and M. Imoto, “2020
Public Viewing—Kirari! Immersive Telepresence Technology,” NTT
Technical Review, Vol. 14, No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa5.html[8] T. Nishitani, A. Ono, T. Kanekiyo, T. Yamaguchi, A. Kameda, S.
Nishida, J. Nakagawa, and M. Makiguchi, “2020 Entertainment—A
New Form of Hospitality Achieved with Entertainment × ICT,” NTT
Technical Review, Vol. 14, No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa6.html[9] M. Motegi, K. Hashiguchi, H. Kataoka, T. Kusabuka, Y. Tanaka, U.
Shibusawa, T. Yamazaki, and T. Ogawa, “2020 MICE—New Hospitality through Exhibitions × ICT,” NTT Technical Review, Vol. 14,
No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa7.html[10] T. Ohashi and S. Honda, “Promotion of Co-innovation through Collaboration with Different Business Sectors,” NTT Technical Review,Vol. 14, No. 12, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201612fa8.html

NTT Technical Review

Feature Articles

Ryuji Kubozono

Vice President, NTT Service Evolution Laboratories.
He received a B.S. and M.S. in physics from
Kagoshima University in 1987 and 1989. He
joined NTT Human Interface Laboratories in
1989. He was with NTT WEST from 1999 to2003 and from 2006 to 2008, and with NTT
Smartconnect from 2012 to 2016.

Vol. 14 No. 12 Dec. 2016

4

Feature Articles: 2020 Showcase

2020 Airport/Station—Hospitality for
Foreign Visitors at Airports and Train
Stations
Yusuke Ichikawa, Yukihiro Nakamura, Taiji Nakamura,
Hirohisa Tezuka, Hitoshi Seshimo, Satoshi Fukada,and Hideki Mitsui
Abstract

The number of foreign visitors to Japan is rapidly increasing, and determining how to provide information adapted to the needs of various nationalities in various languages has therefore become a criticalissue. Accordingly, airports—the gateways of Japan—and train stations—key points along visitors’ transit routes—will not only require technologies that enable easy access to information for foreign visitors
but will also become places that showcase Japan’s cutting-edge technologies around the world. In thisarticle, we describe our collaborative efforts with various businesses to conduct demonstration tests inorder to improve the way information is provided at airports and train stations.
Keywords: airport, station, showcase

1. Introduction
For visitors to Japan, differences in language andculture in matters such as eating and traveling in
unfamiliar places cause a great deal of stress. Visitors
may also have difficulty with transit routes, including
the airports at which they just arrived and the connecting train stations to their target destinations, andthus, many problems remain to be solved. For example, the guidance information on most signs is limited
to major languages such as Japanese and English. In
addition, visitors cannot quickly think of their nextmeans of transport and find out how to use it.
At NTT, we are targeting the year 2020 as we aim
to solve the problems experienced by visitors at airports, train stations, sports stadiums, and other facilities. Namely, we hope to accommodate differences incultures and achieve multilingualization of guidance
information. We also hope to optimize guidance
methods to ensure that visitors can transit smoothly inrecognition of the fact that not only physical mea1

sures such as conventional human responses andsignboard but also measures that utilize ICT (information and communication technology) are required.
Moreover, we aim to provide innovative hospitality
through user interface/user experience (UI/UX)
design by utilizing cutting-edge technologies such asimage analysis and big-data analysis. We are thus
actively researching and developing such technologies in collaboration with various transport-terminal
facility operators, related agencies of government
ministries, and equipment manufacturers.
With airports, train stations, and their surrounding
environments as target areas, demonstration tests
were done in 2015 and 2016 in three locations: Haneda Airport International Passenger Terminal, the areaaround Tokyo Station, and commercial facilities
around Namba Station in Osaka. The effectiveness ofthe technologies was evaluated, and problems
involved in introducing the technologies were clarified.

NTT Technical Review

Feature Articles

Point-and-search guidance

Projection signage

Intelligent sound sign

Fig. 1. Collaborative tests at Haneda Airport International Passenger Terminal.

2. Demonstration tests at Haneda Airport
We carried out demonstrations of an upgraded universal design for providing information at Haneda
Airport. We describe the tests and their results here.
2.1 Overview of collaborative tests
Haneda Airport International Passenger Terminal(international flights terminal) is based on the concept of universal design that involves user participation from the design stage. The airport combines hardfeatures such as buildings and facilities and soft features such as human services and is one of the world’s
leading airports that achieves universal design withhigh dimensionality.
However, the environment is rapidly changing andexceeding expectations; examples of this include therecent rapid increase in visitors to Japan and the hugeincrease in the number of international flights in andout of Haneda Airport. Consequently, it is forecast
that the existing hard and soft facilities will become
insufficient in the near future, and new efforts aimed
at reducing congestion and further accommodating
multilingualization will be needed. It is difficult toaddress these issues with conventional fixed guidance
signs and face-to-face guidance, so new methods ofcomputerization that will help further improve accessibility and the customer experience of various airport users will be required.
In view of the above-described circumstances, NTT
has been collaborating with Tokyo International Air
Terminal Corporation to upgrade the airport’s information universal design. In December 2015, NTTstarted demonstration tests [1] of the upgraded information universal design of the terminal with the following three services: (1) information provision in
multiple languages by utilizing point-and-search UI;(2) congestion reduction by inducing people-flow
through utilization of dynamic signs; and (3) clarifiVol. 14 No. 12 Dec. 2016

cation of audio guidance using speech processing
technology (Fig. 1).
2.2 Technical points(1)	Information provision in multiple languages byutilizing point-and-search UIThis information-provision service utilizes anglefree object search technology developed by NTT.
This service is based on a UI that acquires appropriate
and useful information by enabling the user to intuitively point their smartphone’s camera at advertising
signs and guidance signs in the arrival lobby, at retailbuildings in shopping areas, and at food samples andproducts in order to access information in situations
where suitable keywords or inquiry methods cannot
be thought of quickly. In these demonstration tests,
actual feedback from the test targets—namely, foreign nationals—was evaluated, and challenges to beaddressed in the future—that is, clarification ofrequired information services—were identified.
(2)	Congestion reduction by inducing people-flow
through utilization of dynamic signs
This service helps people avoid congestion within
facilities and induces the optimum flow of people byforeseeing the ever-changing state of congestion anddynamically varying guidance signs accordingly.
Demonstration tests were carried out in 2015 inplaces where people tend to accumulate, for example,
arrival lobbies and departure gates. In these tests,
dynamic guidance signs utilizing projection mapping
were provided, and their effectiveness was evaluated,knowledge about the most-appropriate presentation(e.g., signage, color, and timing) was acquired, andthe effects of inducing people-flow were measured interms of quantitative values.
(3)	Clarification of audio guidance using speechprocessing technology
For visually impaired people who cannot easily
gather information visually, sound plays an important2

Feature Articles

role in helping them understand their surrounding
environment. Accordingly, intelligent sound signs*
were developed with the aim of supporting smoothtransit of visually impaired people and utilizing aspeech intelligibility enhancement technique developed by NTT. These signs provide guidance using avoice that can be heard easily even in noisy environments. They take their environment into account sothat the sounds of the sign itself do not become
unwanted noise in that environment. The effectiveness of these signs was demonstrated by measuring
the hearing and movement of actual visually impaired
people.
2.3 Future development
NTT aims to expand the above-described services
as a case model for Haneda Airport and therefore
collaborated with Panasonic Corporation, Tokyo
International Air Terminal Corporation, and Japan
Airport Terminal Co., Ltd. in establishing the Information UD Exploratory Committee for Airports(chairperson: Professor Tetsuo Akiyama of Chuo
University) in November 2015. From now onwards,
we plan to improve the individual technologies thatwere shown to be effective through the collaborative
tests as well as to upgrade the visuospatial and audio
designs across the entire airport and appeal to otherbusinesses to join us as partners in collaborative testing.
3. Demonstration tests of seamless navigationnear Tokyo Station
We describe here demonstration tests done at Tokyo
Station to evaluate technologies designed to achieve
seamless navigation.
3.1 Overview of demonstration tests
By the year 2020, thanks to advances such as thefour-satellite-set of the Quasi-Zenith Satellite System(with the fourth satellite scheduled to be launched in2018) and indoor positioning technology, the socalled high-accuracy positioning society will be realized. The Ministry of Land, Infrastructure, Transport
and Tourism launched a project called the HighAccuracy Positioning Society Project. This project isinvolved in carrying out studies and demonstration
tests on spatial-information infrastructure in order toproduce various services by utilizing indoor and outdoor positioning technologies and mapping information, as well as the methods and systems for efficiently and effectively maintaining that infrastruc3

ture. The NTT Group aims to meet the demands ofsuch a society and has therefore been part of this project since its inception and has been participating in
the demonstration tests since 2014. NTT Service
Evolution Laboratories is playing a key role in thosetests [2].
At the Seamless Indoor/Outdoor Navigation Feasibility Tests near Tokyo Station, which were supportedby the High-Accuracy Positioning Society Project, apedestrian-movement support concept was proposedand demonstrated. In this concept, pedestrians
acquire navigation information by using natural gestures rather than menus and buttons by simply pointing their smartphone at a target of interest while
walking and holding their smartphone in the usual
way [3]. Consequently, precise map information andthe status of the positioning environment are applied
to seamlessly connect indoor and outdoor environments.
3.2 Technical points
In these demonstration tests, two distinctive functions were created by applying NTT’s parametricmap platform technology and angle-free object
search technology. These functions were a 2.5-dimensional (2.5D) map (flat plane + multiple floor levels),
which presents indoor and outdoor maps and routes
in a concise and unobstructed manner, and point-andsearch guidance, which pinpoints the standing position of the user taking a photo and presents information related to the location of the sign or landmarkthat they photographed (Fig. 2). Moreover, when weapplied actual map information to these functions andmeasured an on-site positioning environment, we
gained a lot of knowledge and know-how throughcollaboration with external business vendors familiar
with the indoor spaces and facilities of Tokyo Stationand its environs and also familiar with the creation of
map information.
3.3 Future development
Through these demonstration tests, the feasibilityof these two functions (i.e., 2.5D-map display andpoint-and-search guidance) in an actual environment
was verified. However, many issues—such as theneed for a scheme for improving the instinctive
understanding by 2.5D-map representation and measures for volatile positioning measurement and the*Sound signs: To aid the transit of visually impaired people, guidance is given by providing information concerning toilet facilities, escalators, and other details in the form of sound (voice andaudio).

NTT Technical Review

Feature Articles

Point-and-search guidance

2.5D map of present spot

Complementaryindoor-location information

Point-and-search“Whereguidance
am I?”Bus
Parking
Outdoors

XXX Bus Station“What’sthat?”

Terminal 2
For xxx via xxx
Terminal 3
For xxx via xxx…

2D-map display

Terminal 1
For xxx via xxx

2.5D-map display
TemporarilyclosedUnderground

Dynamic information linkage

Indoors

Fig. 2. Image of demonstration test of seamless indoor and outdoor navigation technology.

effects of noise—became apparent as a consequenceof the actual environment. We will address theseissues in technical developments as we continue ourefforts in projects initiated in 2016.
4. Demonstration tests of O2O2O
transmit-to-customer service
Here, we describe demonstration tests carried out
on our transmit-to-customer service in the Namba
area.
4.1 Overview of demonstration tests
Aiming towards 2020, NTT is continuing demonstration tests under a wide range of application scenarios ranging from transit guidance in airports, train
stations, towns, and stadiums to drinking and eating,shopping, sightseeing, watching sports, going to thetheatre, and appreciating art.
As a first step, in collaboration with Nankai Electric
Railway Co., Ltd. and Takashimaya Company, Limited, NTT and NTT DATA conducted demonstration
tests of O2O2O services in June 2016. Abbreviated as
O2O2O, out-of-home to online to offline refers to asystem for providing information (online) to users via
signs outside of the home (OOH signs) and guidingthem to their destinations (offline) based on that
information. In these tests, when the users read an
OOH sign via their smartphone with the downloaded
app, they were able to receive information about
Vol. 14 No. 12 Dec. 2016

nearby shopping facilities and to be guided to theirdestinations (Fig. 3).
4.2 Technical points
A universal object-recognition platform developed
by NTT is utilized for reading OOH signs (Fig. 4). In
conjunction with an application programming interface, the universal object-recognition platform jointly
controls various recognition tools, and it can be calledup from various client apps. Its key features are summarized as follows.
•	It is possible to widen the coverage of recognition targets (i.e., the number of targets can beincreased).
•	It is possible to increase the recognition ratecompared to when each individual recognition
tool is used individually.
•	Multiple recognition targets can be processed inparallel simultaneously, so recognition results
can be acquired instantaneously, achieving fast
recognition speed.
In the demonstration tests, electronic-watermarking technology and angle-free object search technology were applied to make it possible to recognize reallogos, posters, and other images, to link to online
content, and to acquire helpful information via a
simple, instinctive operation.
OOH signs (Fig. 5) were prepared for the demonstration tests and set up in 12 locations around Namba
Station and nearby shopping facilities in Osaka City,
4

Feature Articles

Contact area
(Namba Station/commercial facilities)

Inflow route(Kansai International Airport
Nankai Line Namba)

Check-in point(connection points within each facility/duty-free counter )Nankai Electric Railway
Namba City/Namba Parks

OOH signs

Nankai Electric Railway

Takashimaya Osaka Store

Angle-free and electronic watermark(indoor location positioning)

Installationof appAcquisition of information
by pointing smartphone

BLE beacon

Transmission
to customers

Visible-light
communication

Check-in

BLE: Bluetooth Low Energy

Fig. 3. Image of demonstration tests performed in the Namba area.

Elements implemented
in this test

Universal object-recognition platform

Angle-freeobjectsearchElectronic
watermarkBarcode,2DbarcodeCharacterrecognition
OCR

RFID
BLE
Wi-Fi

....

Content
database

OCR: optical character recognition
RFID: radio frequency identification

Fig. 4. Universal object-recognition platform.

5

NTT Technical Review

Feature Articles

Installed poster

Attachment of stickers toexisting signboards

Fig. 5. OOH signs used in demonstration tests.

Airports
線
際線
国
国際
ONAL
AL
ERNATIATION
INTINTERN

Step 1
Haneda Airport
model
Step 2
Expansion to other major areas andfacilities(including added functions)Roads

Accommodation facilities

Step 3
Establishing interoperability
including movement
between areas
Stations

Stadiums

Fig. 6. Three steps involved in expanding hospitality services.

Japan. This service can be introduced by attaching atranslucent sticker to existing promotional media
such as posters and wall advertising; accordingly, it ispossible to start operations without damaging existing designs. Among foreign visitors to Japan who had
downloaded the app for events, about 60% acquired
information from OOH signs, and it was confirmedthat about 20% of them actually visited the place theyreceived guidance to.
4.3 Future development
The NTT Group is planning to carry out new technical development and demonstration tests aiming
Vol. 14 No. 12 Dec. 2016

toward 2020. It is also focusing on commercializing
and introducing point-and-search UI using universal
object-recognition technology while calling for new
business partners to collaborate in experimental trials.
5. Goals targeting 2020
Goals have been set to commercialize services forproviding transit guidance and information to foreign
visitors to Japan. Three steps are involved in achieving these goals (Fig. 6). The first step consisted of
conducting demonstration tests at airports, as
6

Feature Articles

described earlier. The second step involved carryingout demonstration tests at train stations and their surrounding facilities. For visitors in the future, however,
it is unrealistic to require all tourists to master eachapp and the UI/UX design for different sites.
In the third and future step, through co-innovation
involving various stakeholders, we will establish
interoperability, including movement between areas,and provide hospitality services with a sense of uniformity in order to achieve total information support
covering sequential travel actions—from arrival at
the airport to returning to the airport.

7

References
[1]

[2]

[3]

Y. Kuhara, S. Yamashita, S. Kinoshita, H. Tezuka, Y. Ichikawa, and S.
Fukada, “Joint Experiment on Enhancement of Information Universal
Design at Airports,” NTT Technical Review, Vol. 14, No. 7, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201607fa2.html
N. Iso and Y. Takaki, “Seamless Indoor/Outdoor Navigation Feasibility Test near Tokyo Station—Part of the High-Accuracy Positioning
Society Project Sponsored by the Ministry of Land, Infrastructure,
Transport and Tourism,” NTT Technical Review, Vol. 14, No. 7, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201607fa3.html
H. Abe, S. Konagai, S. Chiba, M. Muto, M. Nagahama, H. Seshimo,
A. Kawanobe, Y. Koike, and S. Fukada, “Development of Map Platform Technology to Innovate Seamless Navigation,” NTT Technical
Review, Vol. 14, No. 7, 2016.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201607fa4.html

NTT Technical Review

Feature Articles

Yusuke Ichikawa

Senior Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received a B.E. and M.E. in measurement
engineering from Keio University, Kanagawa, in
1995 and 1997. He joined NTT Multimedia Network Laboratories in 1997. Since 1998, he has
been researching and developing recommendation systems. He is a senior member of the Information Processing Society of Japan (IPSJ) andreceived the 2005 IPSJ Yamashita SIG ResearchAward.

Yukihiro Nakamura

Senior Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received a B.E. and M.E. in engineeringfrom University of Tsukuba, Ibaraki, in 1992 and1994, and a Ph.D. in functional control systems
from Shibaura Institute of Technology, Tokyo, in
2014. He joined NTT in 1994. From 2001 to
2005, he was with NTT Communications, andfrom 2011 to 2014, he was with NTT Advanced
Technology Corporation. His research interests
include network robot platforms and humanrobot interaction. He is a member of the Japan
Society of Mechanical Engineers (JSME), theRobotics Society of Japan (RSJ), and the Societyof Instrument and Control Engineers (SICE).

Taiji Nakamura

Senior Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received a B.E. in industrial engineeringfrom Tokyo University of Science. He joined
NTT DATA Communications Systems Corporation (now NTT DATA Corporation) in 1991. He
has more than 20 years’ experience in information systems planning and development in the
public sector.

Hitoshi Seshimo

Senior Research Engineer, Supervisor, Proactive Navigation Project, NTT Service Evolution
Laboratories.
He received a B.E. and M.E. in mechanicalengineering from Waseda University, Tokyo, in
1995 and 1997. He joined NTT in 1997. His
research interests include computer aided
instruction, web-based learning, content distribution and navigation systems, and geographicalinformation services.

Satoshi Fukada

Senior Manager, Transport and Mobility Business Producer, Strategic Business Creation
Team, NTT Research and Development Planning
Department.

Hideki Mitsui

Manager, Manufacturing Division 1, Manufacturing IT Innovation Sector, NTT DATA Corporation.

Hirohisa Tezuka

Senior Research Engineer, Supervisor, 2020
Epoch-making Project, NTT Service Evolution
Laboratories.
He received a B.E. and M.E. in engineeringfrom Kyoto University in 1991 and 1993. He
joined NTT in 1993. His research interests
include system management, networked robotics, and lifelog systems. He is a member of SICE.

Vol. 14 No. 12 Dec. 20168

Feature Articles: 2020 Showcase

2020 Town—Web Design Converter
for Providing Guidance Assistance to
Individuals in Cities
Kiyoshi Nakahama, Yuji Morinishi, Masahiro Watanabe,
Sayaka Teranaka, and Rika Mochizuki
Abstract

To prepare for the year 2020, NTT Service Evolution Laboratories is striving to develop a system toprovide city guidance to individuals such as senior citizens and foreign visitors to Japan. The systemtakes the characteristics of individuals into account. In this article, we introduce our Web Design Converter system, which presents information tailored to individual user characteristics on signage screensand personal terminals.
Keywords: signage, web, conversion

1. Introduction
Digital signage systems that transmit information
using display screens and other display processes arein wide use today. The efficient way in which they
display information by effectively using moving
images and space makes them a useful means of providing information about cities.
However, two important issues must be taken into
account in developing such systems. First, with theinternational sporting events scheduled to be held inTokyo in 2020, the number of foreign visitors toJapan is expected to increase greatly during the next
few years. Second, Japan faces the problem of a rapidly aging population. To address these issues, it has
become essential to develop systems that meet theneeds of a greater variety of users. This, however,makes it necessary to create content on an individual
basis, which brings added costs.
Accordingly, we at NTT Service Evolution Laboratories are working to develop web design/conversion
technology with which content can be converted tomake it easier to understand. This will enable us toprovide content that will meet service aims and thatwill match the characteristics of individuals such as
1

their age group and the language they speak. We callthe product that utilizes this technology the Web
Design Converter, and we have demonstrated a prototype of it at various exhibitions [1, 2].
By using this technology to create digital signage
content, we expect to be able to not only reduce content creation costs but also to provide content such as
information about local events in a timely manner.
We believe that the Web Design Converter digital
signage system is a useful tool for providing city
guidance to individuals according to their personal
characteristics, and that by making it available tousers worldwide we can achieve a world in which auniversal town development approach is emphasized.
2. Overview of Web Design Converter
Web Design Converter comprises an automatic
design conversion function and a function for automatically generating supplementary information. The
automatic design conversion function is used for providing translated information for foreign visitors. In
addition, it presents information in an easy-to-understand way in accordance with user and service aims
by converting the information structure, fonts, and
NTT Technical Review

Feature Articles

Existing web content

Interview results andbehavioral observations
from user surveys
Features and knowledge
Design requirements
Automatic
conversion
Storefront
signage content

Web Design
Converter
Automatic conversion

In-store menu terminal content(a)• Operations to be
performed on pageshown clearly (to aid
senior citizens)• Simple layout• Attractive colors• Large character size (to aid
senior citizens)• Information conveyed
via differences in color(red or boldface)complements othervisual cues (UD)NTT guidelines• At least 4.5:1 contrast
between backgroundand text colors (UD)• Limited font types(Customer experience)

UD: Universal design guidelines for web content
Senior citizens: Senior-friendly web design guidelines
Customer experience: Vital points for customer
experience design (NTT version)(b)

Fig. 1. Example features of automatic design conversion function.

colors. It does so by applying the information andcommunication technology service design principles
NTT Service Evolution Laboratories has developed
and fostered based on user characteristic surveys.
Vol. 14 No. 12 Dec. 2016

This function makes it easy to provide UI (user interface) design applications that take usability into
account (Fig. 1).
The function for automatically generating2

Feature Articles

Existing web content(2) Extraction process/insertion process
Store logo (photo)Store overview (Japanese)
Overview (English)
Menu
item
(photo)Menu item name (in Japanese)

Menu
item
(photo)Menu item name (Japanese)

Menu
item
(photo)Menu item name (Japanese)

In English
In Chinese characters (traditional)
In Chinese characters (simplified)
In Korean

In English
In Chinese characters (traditional)
In Chinese characters (simplified)
In Korean

In English
In Chinese characters (traditional)
In Chinese characters (simplified)
In Korean

Price

Price

Price

…
(1) Storefront signage template(3) Generated storefront signage

Fig. 2. Process for generating storefront signage content.

supplementary information uses information on theInternet to automatically generate supplementary
information about commercial products and the like[3]. For example, if the signage for foreign visitors ina restaurant simply translates a dish as “A mackerel
merou” [namerou], they will likely not be able toclearly understand what sort of dish it is. This function can help them by adding further informationsuch as, “A traditional food of Chiba Prefecture.” It
may also further their understanding by making comparisons to things they are familiar with, such as, “It’slike ‘tartar for appetizers.’” The translations are provided by machine learning, and as research in thisfield advances, the resulting translations are expected3

to improve in accuracy.
3. Example implementation ofrestaurant service
Applying the Web Design Converter technology to
websites that introduce the type of food a restaurant
serves enables users to automatically generate content such as that in the storefront signage and the instore menu terminal.
3.1 Storefront signage
Users first select the display for the storefront signage template (Fig. 2(1)). This template includes an
NTT Technical Review

Feature Articles(1) Click on the menu item name(2) Supplementary information about menu item displayed(3) Menu item added to order list

Fig. 3. Interactive menu terminal generating supplemental information.

overview of the store (restaurant), its logo, and a viewof the menu items (with photos, names, and prices ofdishes). Users then extract the information they wantfrom the existing web content of the store and insert
it into the template (Fig. 2(2)). If they need to havecharacters translated during the insertion process,they can use a machine translation function to convert
the information into the desired language (Fig. 2(3)).
Currently, it is necessary to specify the location of
the content to be extracted through the service provider operation when constructing the initial template. However, we are doing further studies with theaim of fully automating the extraction process in the
future.
3.2 In-store menu terminal
In the same way as the storefront signage, the instore menu terminal generates content by extracting aview of the menu items on the menu terminal template. In addition, there is a function (Fig. 3(2)) for
displaying supplementary information about menu
items when they are clicked (Fig. 3(1)). It is also possible to use an interaction function (Fig. 3(3)) toselect or deselect additional items to be ordered.

Vol. 14 No. 12 Dec. 2016

4. Future development
In the future, we plan to expand the template so that
it will be able to handle a wide variety of signage
display devices, including smartphones and tablets.
We also plan to carry out field trials on how to use thetechnology, for example, how to use it to provide
details on store information or events being held
within the city. We aim to use the feedback results toachieve early practical applications of the system. We
anticipate that appropriate information will be provided by various contributors such as citizens andlocal governments, so we will strive to develop a universal town that can be understood by all.
We encourage anyone who is interested in this technology to contact us.
References
[1]
[2]
[3]

Website of NTT R&D Forum,
https://labevent.ecl.ntt.co.jp/forum2016/e/index.html
Website of Digital Signage Japan 2016,
http://www.f2ff.jp/dsj/2016/en/
R. Mochizuki and T. Watanabe, “Emotion Sharing Model Based on
Life-log Comparison—Matching Comparable Experiences—,”
Trans. Human Interface Soc., Vol. 17, No. 4, pp. 327–340, 2015.

4

Feature Articles

Kiyoshi Nakahama

Senior Research Engineer, Universal UX
Design Project, NTT Service Evolution Laboratories.
He received a B.E. in engineering from Muroran Institute of Technology, Hokkaido, in 1990.
He joined NTT in 1990 and has been engaged inresearch and development of an IC (integrated
circuit) card public telephone, shared use personal computers, Next Generation Network(NGN) appliances, and an NGN development
support environment. He is a member of theInstitute of Electronics, Information and Communication Engineers (IEICE).

Yuji Morinishi

Senior Research Engineer, Universal UX
Design Project, NTT Service Evolution Laboratories.
He received a B.E. and M.E. in engineeringfrom Kyoto Institute of Technology in 1994 and1996. He joined NTT Service Evolution Laboratories (formerly NTT Human Interface Laboratories) in 1996. His research interests include
Internet protocol video distribution systems andtelecommunication systems for telephony services.

Masahiro Watanabe

Senior Research Engineer, Universal UX
Design Project, NTT Service Evolution Laboratories.
He received a B.E. and M.E. in mechanicalengineering and a Dr.E. in electronics, information, and communication engineering fromWaseda University, Tokyo, in 1991, 1993, and2003. He joined NTT in 1993. His research interests include human-computer interaction,
human-centered design, and universal design. He
is a member of IEICE, the Human Interface
Society (HIS), and the Acoustical Society ofJapan.

5

Sayaka Teranaka

Research Engineer, Universal UX Design Project, NTT Service Evolution Laboratories.
She received a B.E. and M.E. in engineeringfrom Keio University, Tokyo, in 2004 and 2006.
She joined NTT Cyber Solutions Laboratories(now NTT Service Evolution Laboratories) in2006 and has been developing a knowledge sharing system using a video-scene-linked bulletinboard (SceneKnowledge).

Rika Mochizuki

Researcher, Universal UX Design Project,
NTT Service Evolution Laboratories.
She received her B.E., M.E., and Ph.D. inengineering from Chuo University, Tokyo, in
2007, 2009, and 2011. She joined NTT in 2009.
She has been a visiting researcher at Chuo University since 2009. Her research interests include
color vision compensation, life-log processing,and their applications for communication support. She received the Minister of Education,
Culture, Sports, Science and Technology Awardof the 23th Grand Prize for Frontier Technology:
Pave a New Way for Creativity, and a best paper
award from IEICE in 2009. She is a member of
the Institute of Electrical and Electronics Engineers, Commission Internationale de l’Éclairage(International Commission on Illumination),
IEICE, HIS, the Computer Software Associationof Japan, and the Information Processing Societyof Japan.

NTT Technical Review

Feature Articles: 2020 Showcase

2020 Town—Developing
MACHINAKA Service, a Device
Integration Service that Utilizes
Artificial Intelligence Technology
Takayoshi Mochizuki, Manabu Norota,and Kousuke Suga
Abstract

At NTT Service Evolution Laboratories, we are working to develop device integration services that
can accurately understand the situation a person is in and provide guidance to the person in accordancewith the situation. We are working with partner companies and NTT operating companies in this effort
and are applying the results of joint experiments to develop services for settings such as long-term carefacilities, financial institutions, museums, and outdoor parks that exist within cities. We use the term
MACHINAKA (in the city) services to refer to such services. In this article, we introduce our current
efforts and future prospects.
Keywords: communication robots, multi-modal interaction, device integration service

1. Introduction
Currently, most MACHINAKA (in the city) services are provided with human labor. However, withthe progress of artificial intelligence (AI) technology,
the need for human intervention is gradually decreasing. Amazon.com, Inc., a well-known example of alogistics company in the US, has already automatedpart of its operations using AI and robots, from management of its logistic services to product delivery[1]. The increased use of robots in place of humans
has also been seen in the customer service industry
recently with the growth of AI [2, 3, 4]. The communication robot “Pepper” of SoftBank Corporation is atypical example of a robot replacing humans in a
MACHINAKA service industry. It is an example ofleveraging communication robots that can interact
intelligently with humans through words and actions
using AI. The use of communication robots is expected to lead to a reduction in the cost of human labor as1

well as greater effectiveness of advertising presented
using such robots.
As mentioned, we are starting to see communication robots interacting with people and replacing
humans in the customer service field. However, it canbe said that most of these interactions are still very
limited and are centered on routine one-to-one dialogues with customers, for example, to explain goods
and services at a reception counter or provide guidance at facilities. This limited use is presumably
because the main feature of communication robots isinteracting with people using voice and gestures.
At the NTT laboratories, however, we believe that
it is possible to expand the scope of applications forcommunication robots beyond one-to-one customer
service situations by combining communication
robots with a number of other devices and systems toconstitute a device integration service. We describe
here the advances that can be achieved when a device
integration service centered around communication
NTT Technical Review

Feature Articles

R-env enables robots and other devices that surround us to work in concert.
R-env can provide personal assistance based on an accurate understanding of each
person’s situation.
R-env can expand the human potential by integrating various devices and robots.
Dialogue app

Nursing-care app

Music app

AI engine

Image processing
engineSpeech-to-textengineR-env®

Data

IoT-PF

Cloud

RobotsSmart mobility

Tablets/
Smartphones

Sensors

IoT-PF: Internet of Things platform

Fig. 1. Overview of R-env®, a platform for expanding the human potential through integration of devices and robots.

robots is realized.
As previously mentioned, the customer service
industry is increasingly using communication robots
to provide guidance or explain facilities or services
using voice. This is occurring more and more frequently in the hospitality industry such as at hotels
and reception centers, as well as in the banking andretail industry to provide information to customers.
However, combining communication robots withimages or videos shown on a display will greatly
increase the clarity of information that can be conveyed. Also, connecting communication robots withhealth monitoring devices such as blood pressure
monitors opens up the possibility of providing healthadvice to users [5].
It is also possible for moderators of recreational
activities to use a communication robot as a support
device by connecting it with a karaoke system. Furthermore, the use of communication robots in cooperation with various devices placed inside a facility or
outdoors within a target area makes it possible to not
only provide information and guidance to customers,
but also to improve the convenience of getting aroundin the city and to increase the number of visitors andfacilitate visitor movements as well.
Vol. 14 No. 12 Dec. 2016

In this way, we believe that we can assist peoples’
understanding and encourage their behavior and thus
stimulate numerous activities within the city by linking various devices to communication robots. In thefollowing sections, we introduce R-env®, the key
technology to make all this happen.
2. R-env, technology supporting device andservice integration
For more than 40 years, NTT has carried out
research and development (R&D) on speech andaudio processing technologies and natural language
processing technology. NTT is also a world leader inimage processing technology, knowledge processing
technology, and various other media processing technologies. Efforts are underway to achieve a naturaldialogue between communication robots and humans
by making use of these technologies. To expand therole of communication robots and increase their existential value, we started R&D on R-env in 2015.
R-env enables robots to be easily connected withother external devices and systems (Fig. 1).
R-env was developed as a part of ongoing researchon interaction technology aimed at expanding the2

Feature Articles

Support service fornursing care facilities and retailstoresHealth care support
service for the elderly

Provision of corevoTM

NTT DOCOMONTT Communications

Common platform fordemonstration experiment
Robot integration and control (R-env)
SpeechdialogueChat
dialogue

FAQ
search…

Dialoguecontrol

Questionanswering

Machinetranslation

…

Voice
recognition

Speechsynthesis

Image
recognition…

Support service for tourist
information facilities

Provision ofcloud robotics platform

Support service for promoting
regional development

NTT WESTNTT DATANTT EAST

Fig. 2. Examples of device integration services by NTT Group companies.

possibilities of humans. This technology enables easy
connection and integration of various devices such ascommunication robots. By linking these devices, thistechnology will make it possible for communication
robots to accurately understand the situation in which
a person is placed, provide guidance to the person inaccordance with the situation, and prompt the person
to adopt a specific behavior or level of awareness.
With R-env, anyone can develop a device integration service using graphical user interface (GUI)screens on a browser. This is done by creating a statetransition diagram, or in other words, by combining
actions specified for each device connected to R-env,
and the condition for the transition to the next action.
R-env can also be used to easily create a complex
device integration service by reusing and operating
multiple state transition diagrams in parallel. Also,
services developed on the GUI can be instructed tocontinue on with their operation even after closing thebrowser by keeping it on hold in its execution state.
Furthermore, when a new device is added, it can beeasily recognized, registered, and prepared for use in
the R-env service by simply transmitting the predefined JSON (JavaScript Object Notation) format
using WebSocket before use.

3

3. Device integration service using R-env in
MACHINAKA (trial case)We have been conducting trials with NTT Group
companies and their external partners to extract theneeds and identify any technical problems of device
integration services that utilize R-env [5, 6]. In particular, at the end of July 2016, we initiated a joint
demonstration experiment in collaboration with five
major NTT operating companies. The purpose of theexperiment is to evaluate the effectiveness andacceptability of device integration services in a variety of fields and industries in the city (Fig. 2) [7]. In
the following section, we introduce a trial case staged
in the Shinjuku area.
4. Trial in Shinjuku
The number of tourists visiting Japan from abroad
has increased rapidly. This is evident from theincrease in the government’s target number of foreign
tourists to Japan, which was raised from 20 million
per year to 40 million per year by 2020 at a tourism
related meeting chaired by Prime Minister Shinzo
Abe on March 30, 2016 [8].
The driving force behind this is the dramatic
NTT Technical Review

Feature Articles

11th floor event venue

Field trial image
Robots were installed in foot traffic areas
leading to the event venue located on the 11F(At 2F entrance and near the 9F/11F escalator
landing).
Event announcements were made via a
dialogue between two robots (2F).
Robots carried out dialogues with customers
and provided floor guidance of the department
store (9F/11F ).

14F
13F
12F
11F
10F
9F
8F
7F
6F
5F
4F
3F
2F
1F
B1F
B2F

+sensor+sensorFig. 3. Service image in Shinjuku Takashimaya.

increase in the number of tourists from China, which
doubled in 2015 compared to the previous year. Tourists from China now account for the largest number offoreign tourists to Japan and greatly exceed the number from Taiwan, the second-largest group [9]. Theirprimary purpose for visiting Japan is shopping, as
symbolized by the expression baku-gai, or shopping
spree.
Popular areas for Chinese tourists, in addition towell-known tourist destinations, are Ginza and Shinjuku, where there are many shops [10]. Places such as
Shinjuku Golden Gai and the Kabukicho neighborhood are also favored by foreign tourists in Shinjuku[11]. Therefore, from the viewpoint of regional development, services in Shinjuku are required not only
for local residents but also for people coming fromoutside Shinjuku, including tourists.
We carried out two joint demonstration experiments utilizing a device integration service targeting
the Shinjuku area. The first one was a joint experiment conducted by three groups: Takashimaya Company, Ltd., which is engaged in the retail business,
NTT Communications, and the NTT laboratories. In
the experiment, we used a device integration service
to inform and guide customers to event venues using
multiple robots placed in a commercial facility(department store) from August 3 to August 14, 2016.
In particular, robots were placed in the following
locations based on the flow of pedestrian traffic.
1)	The location with the largest amount of pedesVol. 14 No. 12 Dec. 2016

trian traffic, which was the area near the second
floor (2F) entrance leading to JR Shinjuku Station;2)	The 9F escalator landing, where the main target
customers (families with children) of the eventare likely to gather;3)	In the vicinity of the 11F escalator landing inwhich the event venue was located.
At each location, a device integration service composed of an external sensor linked to a communication robot provided smooth induction to the venue by
arousing customer interest in the event through dialogue with customers and by providing floor guidance of the department store (Fig. 3).
The second case is an effort targeting the broad
outdoors and is not limited to the indoor space of a
facility. It is a joint trial being conducted by the Shinjuku branch of the Tokyo Chamber of Commerce andIndustry, the Shinjuku Tourism Promotion Association, NTT EAST, and the NTT laboratories. The trial
provides a device integration service consisting ofrobots and digital signage at multiple locations withina radius of approximately 1 km from Shinjuku Station. In particular, communication robots installed in
shops and in tourist facilities in the area provide
access information about the respective shops andfacilities by utilizing the public wireless local area
network service Shinjuku Free Wi-Fi provided by
Shinjuku ward.
Communication robots are also playing an important4

Feature Articles

Field trial (example)
1) Customers receive a stamp by showing a two-dimensional bar code toa robot inside the store. The robot then performs an action related to theinstallation location.
2) Customers are encouraged to explore the city by participating in thestamp rally game. (Players visit and interact with multiple robots locatedthroughout the city.)3) Customers who interact with all of the robots are given bonus content.
(A link to download the bonus content is shown on a screen placed
beside the robot.)

Fig. 4. Service image in Shinjuku area.

role to increase the number of visitors to the area by
providing attractive services such as a robot stamp
rally that utilizes multiple robots in different places
that cooperate with one another. We are in the processof validating how these services help to increase thenumber of visitors and facilitate visitor movements(Fig. 4).
5. Future development
We are promoting trials to apply device integration
services to a wide variety of fields and industries in
addition to the cases introduced here; some examplesof where these services may be beneficial include
long-term care facilities and tourist facilities. Furthermore, in parallel with the joint demonstration experiment, we are also working to expand the applicationof R-env by broadening cooperation and promoting
joint events with hardware manufacturers and service
developers [12].
We are also working to spread and improve thedevice integration service by expanding four initiatives:1) 	Conducting hands-on events of the latest devices, including robots;2) 	Regularly holding a hackathon event targeted5

for individual developers and service developers with no programming experience;3) 	Holding field trials in cooperation with service
providers;4) Conducting business trials.
We aim to create new business opportunities using
robots by enhancing and promoting the value of our
MACHINAKA service using device integration services, and by improving the performance of the associated element technology based on the verification
results obtained from the joint demonstration experiment.
References
[1]

[2]

[3]
[4]

K. Bhasin and P. Clark, “How Amazon Triggered a Robot Arms
Race,” Bloomberg, June 2016.
https://www.bloomberg.com/news/articles/2016-06-29/how-amazontriggered-a-robot-arms-race
Website of MIZUHO FinTech, News, Pepper Provides New In-branchServices,
https://www.mizuhobank.com/mizuho_fintech/news/pepper/index.
html
The DENTSU HO report issued by Dentsu Inc. on July 29, 2015 (inJapanese).
http://dentsu-ho.com/articles/2843
M. Tai, “New Japanese Hotel Offers Service with a (Robotic) Smile,”
NIKKEI Asian Review, Sept. 2015.
http://asia.nikkei.com/magazine/20150903-LIVING-TOGETHER/Onthe-Cover/New-Japanese-hotel-offers-service-with-a-robotic-smile

NTT Technical Review

Feature Articles

[5]
[6]
[7]
[8]

Press release issued by NTT on July 28, 2015 (in Japanese).
http://www.ntt.co.jp/news2015/1507/150728a.html
Press release issued by NTT on November 25, 2015 (in Japanese).
http://www.ntt.co.jp/news2015/1511/151125a.html
Press Release issued by NTT on July 25, 2016 (in Japanese).
http://www.ntt.co.jp/news2016/1607/160725a.html
Prime Minister of Japan and His Cabinet, “Summary of Tourism
Vision to Support the Future of Japan (proposal)” (in Japanese).
http://www.kantei.go.jp/jp/singi/kanko_vision/dai2/siryou1.pdf

Takayoshi Mochizuki

Senior Research Engineer, Networked Robot& Gadget Project, NTT Service Evolution Laboratories.
He received an M.A. in human sciences fromWaseda University, Tokyo, in 2001. He joined
NTT in 2001 and moved to NTT Service Evolution Laboratories in 2014. His research interests
include human-robot interaction and humanmachine interfaces. He is currently engaged in
R&D on non-verbal interactions that encourage
new behavior and awareness of people by utilizing robots and other devices and AI.

[9]Article issued by D2C Smile on April 6, 2016 (in Japanese).
http://www.d2c-smile.com/201604066860[10] Article issued by ACTZERO on December 14, 2015 (in Japanese).
http://www.actzero.jp/social/report-15277.html[11] Article of NIKKEI TRENDY NET issued on June 17, 2016 (in Japanese).
http://trendy.nikkeibp.co.jp/atcl/pickup/15/1008498/060800312/[12] Press release issued by NTT on February 16, 2016 (in Japanese).
http://www.ntt.co.jp/news2016/1602/160216c.html

Kousuke Suga

Associate Manager, New Business Development Headquarters, NTT EAST Corporation.
He received a B.A. from Rikkyo University,
Tokyo, in 1996. He joined NTT EAST the same
year. He is currently engaged in developing newservices for elderly people to promote their useof the Internet.

Manabu Norota

Senior Manager, Supply Chain & Service
Industry Group, NTT Communications Corporation.
He received a B.A. in economics from Fukushima University in 1996. He joined NTT Communications in 1996. He is currently engaged insales promotion.

Vol. 14 No. 12 Dec. 2016

6

Feature Articles: 2020 Showcase

2020 Public Viewing—Kirari!
Immersive Telepresence Technology
Akihito Akutsu, Akira Ono, Hideaki Takada,
Yoshihide Tonomura, and Maiko Imoto
Abstract

NTT Service Evolution Laboratories is conducting research and development on an immersive telepresence technology called Kirari! in order to achieve ultra-realistic services that provide an experienceakin to actually being at a sporting or live venue to off-site viewers in remote locations. This article gives
an overview of technologies supporting Kirari!, including advanced media streaming and synchronization (Advanced MMT (MPEG Media Transport)), real-time image segmentation for any background,ultra-realistic presence design (virtual loudspeakers), and super high-definition video stitching. Initiatives using these technologies are also introduced.
Keywords: ultra-realistic, immersive telepresence, media processing/synchronization technology

1. Introduction
Diverse ways of viewing sporting events beyond
conventional television coverage have been introduced recently for international competitions andpopular domestic sports such as baseball and soccer.
These include live distribution over the Internet andpublic viewings at off-site venues. Further, with the
spread of 4K and 8K broadcasts, people around theworld are expected to share in the excitement ofsporting events in the year 2020 through televisionand public viewings.
To tackle this world-wide diversification in viewing
styles, NTT Service Evolution Laboratories
announced our concept of Kirari! immersive telepresence technology in February 2015 [1]. Kirari! will
implement high resolution and ultra-realistic presence technologies to deliver the sporting venue spacein real time throughout Japan and the world. The goalof Kirari! is to enable viewers to experience sporting
events as though they were actually at the venue fromanywhere in the world.

1

2. Technologies supporting
ultra-realistic services
Two main technical issues arose in implementing
ultra-realistic real-time transportation of the video
space with Kirari! (Fig. 1):•	Improving the accuracy when separating theimage of a participant and the sound of the competition from the overall situation at the venue inreal time•	Synchronizing the transportation of the extracted
participant image and competition audio as well
as stitching the overall competition backgroundimage and audio data and achieving highly realistic reproduction at the remote venue
We analyzed the technical elements needed toresolve these technical difficulties and carried out
research and development (R&D) on each of them.
As a result of this R&D, we were able to present realtime pseudo-three-dimensional (3D) video transmission of athletes giving a karate demonstration at the
NTT R&D Forum held in February 2016, successfully achieving an ultra-realistic viewing experience(Fig. 2). An overview of the technical elements developed and comments on the results are given below.

NTT Technical Review

Feature Articles

Remote venue

Sporting venue
The sporting venue space is transported in real time to the remote venue.
Materialacquisition

Media processing/encodingMedia
distribution

Media
presentation

Real-time image
segmentation
for any background

Super high-definition
video stitching

Projection
mapping

Highresolution
video
encoding
H.265/HEVC

Pseudo-3D
display
devicesMedia
streaming
andsynchronization

Ultrarealisticpresence
designAdvanced
MMT
Selective audio
collection

VirtualloudHighspeakers resolution
displays

Losslessaudio
encoding

Audio
devices

MPEG4-ALS

ALS: Audio Lossless Coding
HEVC: High Efficiency Video Coding

Fig. 1. Kirari! technical elements.

Sporting venue(gymnasium)

Remote venue
Realistic presentation with pseudo-3D image of a competitor on the stage,sound space reconstruction, and display of multi-angle videos

Extraction of participant
image and competition
audio from the backgroundin real timeMulti-angle video

Pseudo-3D
display

Extracted
image
Real-time
extraction

Extracted
audio
Acoustic image
placement

Synchronizedtransmission

In cooperation with Japan Karatedo Federation

Fig. 2. Live demonstration with Kirari!

Vol. 14 No. 12 Dec. 20162

Feature Articles

Analogtelevision

MPEG-2 TS(digital broadcasts
etc.)

Video

“Talkies”
MPEG-2
Transport
StreamSynchronizedvideo and audio
recordingVideo/Audio synchronized transmission(conventional MMT usage)

Advanced MMT

MMT

Video

EPG
data
Audio

Video
Audio
Subtitles

Synchronized signalsthat can be multiplexed
on a single device

Sporting
venue

EPG: electronic program guideSpatial/Environmental data

y
z

Audio

Spatial/Environmental data

Remote venue
Synchronization atterminal using
absolute timex

Synchronized transmission withadditional spatial/environmental
data (Advanced MMT)

Fig. 3. Media synchronization for realistic sensation (Advanced MMT).

3. Advanced media streaming andsynchronization (Advanced MMT)We developed technology to synchronize and transport video and audio, combined with spatial information, by extending MPEG (Moving Picture Experts
Group) Media Transport (MMT), which is an optimized protocol for media synchronization. We added
definitions for MMT signaling to describe 3D information such as the size, position, and direction of
MMT assets (Fig. 3). This technology makes it possible to correlate physical spatial parameters such as
the size and position of the display device with asset
data (frame pixel data) so that the space can be reconstructed with high realism at the destination at theintended size. Also, transmission of the DMX (Digital Multiplex) signals commonly used in production
to control stage lighting and audio devices together
with the MMT assets enables realistic presentations
that accurately synchronize remote stage equipmentwith the media.
4. Real-time image segmentation forany backgroundTo implement Kirari!, it was necessary to extract
the desired subject from the captured video and display it in a way that resembled the real thing. We
developed a technology for extracting the desired
subject from any background; this technology can beused in situations such as sporting events, plays, and3

lectures where it is difficult to use a green screen orother special background (Fig. 4). The first step
involves extracting an approximate range of the subject using sensor data (trimap generation). The second step is to extract the subject in real time. The
technology we developed to achieve this uses aframework that identifies the boundary between thesubject and the background and accurately separates
them using nearest neighbor search in the color
space. It also uses clustering, with constraints toensure that foreground and background colors do not
overlap.
5. Ultra-realistic presence design(virtual loudspeakers)
We developed a highly realistic acoustic image
placement technology that uses diffuse ultrasonicreflections to position acoustic images within thevideo image, where speakers cannot be placed. It canposition virtual sound sources realistically using
fewer speakers and producing a larger listening area
than with earlier surround-sound speaker systems(Fig. 5). Degradation of ultrasonic characteristics ishandled by using electrodynamic speakers to complement low frequencies. This technology enables virtual sound sources to be generated at any location onlarge, full-size images displayed on a large screen, sothat realistic effects can be generated using a simple
configuration such as having voices or competition
sounds originate from their source on the screen.
NTT Technical Review

Feature Articles

Edge correction

Image withapproximate extractionof subject

Correction result
Edge correction

Correction result(detail)Fig. 4. Real-time image segmentation for any background.

Principles of virtual speakertechnology
Composes an acoustic image for subjects on
the video surface, where speakers cannot be
placed, using diffuse ultrasonic* reflections
Display
surfaceArchitecture of virtual speaker system
Audio
signalAudio source
position data
Subject (acoustic image)

Subject (acoustic image)Highly directional speakers

High-pass

Low-pass

Listening area

Highly directional speaker
Acoustic image
position control
processing

Loudspeakers

Loudspeaker
Compensates for degradation of ultrasoniccharacteristics while maintaining acoustic
image positioning by complementing low
frequencies with a loudspeaker

Smoothly controls left/right position of
acoustic image as subject moves* Uses sound waves in frequencies not audible to the human ear, emitted by highly directional speakers, to transport
audible sound from a linear array

Fig. 5. Ultra-realistic presence design (virtual loudspeaker).

6. Super high-definition video stitching
As a first step toward achieving a highly realistic
service with ultra-wide video not possible with ordinary 16:9 television video, we established algorithms
and a system architecture to capture video using five
4K cameras arranged horizontally, and to compose
them in real time (Fig. 6). To process the large
Vol. 14 No. 12 Dec. 2016

amount of 4K image data at high speed in real time,the data are partitioned and a mechanism is used thatenables successive frames to be processed without
waiting for positioning from the previous frame,
which is necessary to suppress flicker. These innovations made real-time (4K60p) processing possible.

4Feature Articles

Space

Time
Conventional:

4K

n-1th frame

nth frame
Use previous seam.

4K

4K

nth frame
Separate overlapping and non-overlapping
areas and process separately.

Proposed:

No need to wait forprevious seamn+1th frame
Use approximate seam.

Process multiple frames in parallel.

Fig. 6. Ultra-wide image composition.

7. Future development
We plan to conduct real-time transportation trials
on these technologies, focusing mainly on individual
sports. Beyond sporting events, we also intend to
expand our horizons to other types of event that havebeen difficult for many people to appreciate remotely.
This will include public viewings of traditional performances, regional festivals, intangible cultural
assets, music concerts, and live coverage of lectures.
In further R&D, we plan to expand the scope ofKirari! to support competition with multiple athletes(competitions with large overlapping subjects) so thatin the future, large numbers of competitors, or multiple subjects moving over a wide area, can be pre-5sented. This will be applied to sports such as judo andsoccer that are currently difficult to present in thisway.
We are conducting R&D on elemental technologies
needed to provide ultra-realistic services with thegoal of making viewing experiences close to actually
being at a sporting venue available anywhere in theworld by the year 2020.
Reference[1]

A. Akutsu, K. Hidaka, M. Inoue, N. Ito, T. Yamaguchi, S. Fujimura,and A. Nakadaira, “Delivering Technologies for Services that Deliver
the Excitement of Games Worldwide,” NTT Technical Review, Vol.
15, No. 7, 2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201507fa2.html

NTT Technical Review

Feature Articles

Akihito Akutsu

Executive Research Engineer, Natural Communication Project, NTT Service Evolution
Laboratories.
He received an M.E. in engineering from Chiba
University in 1990 and a Ph.D. in natural scienceand technology from Kanazawa University in
2001. Since joining NTT in 1990, he has been
engaged in R&D of video indexing technology
based on image/video processing, and manmachine interface architecture design. From2003 to 2006, he was with NTT EAST, where he
was involved in managing a joint venture
between NTT EAST and Japanese broadcasters.
In 2008, he was appointed Director of NTT
Cyber Solutions Laboratories (now NTT Service
Evolution Laboratories). He worked on an R&D
project focused on broadband and broadcast services. In October 2013, he was appointed Executive Producer of 4K/8K HEVC at NTT MediaIntelligence Laboratories. He received the Young
Engineer Award and Best Paper Award from theInstitute of Electronics, Information and Communication Engineers (IEICE) in 1993 and 2000,
respectively. He is a member of IEICE.

Yoshihide Tonomura

Senior Research Engineer, NTT Service Evolution Laboratories.
He received an M.S. in electronics engineeringfrom Nagaoka Uiversity of Technology, Niigata,in 2004 and a Ph.D. from Tokyo Metropolitan
University in 2010. Since joining NTT, he has
been engaged in R&D of multimedia data transmission technologies. From 2011 to 2012, he wasa visiting scientist at MIT Media Lab, Massachusetts, USA. He received the SUEMATSU-Yasuharu Award of IEICE in 2016 and the 26th Telecommunications Advancement Foundation
TELECOM System Technology Award in 2011.

Maiko Imoto

Researcher, Natural Communication Project,
NTT Service Evolution Laboratories.
She received an M.S. in computer science from
Ochanomizu University, Tokyo, in 2011. She
joined NTT the same year and began studying
geographical information search and HTML5based web applications. She has been working on
the immersive telepresence technology called
Kirari! since 2014.

Akira Ono

Senior Research Engineer, Supervisor, Natural
Communication Project, NTT Service Evolution
Laboratories.
He received an M.E. in computer engineeringfrom Waseda University, Tokyo, in 1992. He
joined NTT in 1992 and began researching anddeveloping video communication systems. From1999 to 2010, he was with NTT Communications, where he was engaged in network engineering and creating consumer services. He
moved to NTT Cyber Solutions Laboratories(now NTT Service Evolution Laboratories) in2010. He is working on the Kirari! immersive
telepresence technology.

Hideaki Takada

Senior Research Engineer, Supervisor, Natural
Communication Project, NTT Service Evolution
Laboratories.
He received a B.E. in information and communication technology from Tokai University,
Shizuoka, in 1995, an M.E. in information systems from the University of Electro-Communications, Tokyo, in 1997, and a D.S. in global
information and telecommunication studies fromWaseda University, Tokyo, in 2007. He joined
NTT Integrated Information & Energy Systems
Laboratories in 1997. He has been researching
3D visual perception and 3D display technology.
He received the Best Paper Award from the 3-D
image conference in 2001 and 2005, the Young
Researcher’s Award from the Virtual Reality
Society of Japan in 2002, the Achievement
Award from IEICE in 2003, a Commendation by
the Minister of Education, Culture, Sports, Science and Technology of Japan in 2006, andTechnical Committee Prize Paper Awards fromthe Institute of Electrical and Electronics Engineers (IEEE) in 2014 and 2016. He is a memberof IEEE and the Institute of Image Information
and Television Engineers.

Vol. 14 No. 12 Dec. 2016

6

Feature Articles: 2020 Showcase

2020 Entertainment—A New Form of
Hospitality Achieved withEntertainment × ICT
Tomohiro Nishitani, Akira Ono, Tomoyuki Kanekiyo,
Takahiro Yamaguchi, Akio Kameda, Shin’ya Nishida,
Junichi Nakagawa, and Motohiro Makiguchi
Abstract

This article introduces new applications of information and communication technology (ICT) in theentertainment field. Specifically, we describe two events involving the collaborative creation of newforms of kabuki using ICT: Dwango Co., Ltd.’s Cho Kabuki event held at Niconico Chokaigi (29–30
April 2016, Makuhari Messe, Chiba Prefecture, Japan) and Shochiku Company Limited’s KABUKI
LION SHI-SHI-O held at the Japan KABUKI Festival (3–7 May 2016, Las Vegas, USA).
Keywords: Kabuki, ICT, hospitality

1. A new way to enjoy kabuki
Until now, kabuki has mostly been enjoyed live on
stage. However, NTT Service Evolution Laboratories
has used information and communication technology(ICT) to create new ways to enjoy kabuki in each of
the pre-performance, mid-performance, and postperformance periods.
1.1 Pre-performanceFirst, for the pre-performance period, we created aninteractive exhibit called Henshin Kabuki (transformation kabuki). This non-theater kabuki performance
was held at the KABUKI LION Interactive Showcase
in front of a Las Vegas theater. The exhibit features
the unique make-up methods characteristic of kabuki
and uses the henshin (transformations) of kabuki
actors to showcase the culture and technology prowess that Japan prides itself on, creating a wondrous
experience. Participants chose a kumadori (stage
makeup worn by kabuki actors) mask of their choiceand stood in front of the large-screen monitors. The
angle-free object search technology automatically1detected the type of mask chosen regardless of themask’s angle or tilt and superimposed the kumadori
patterns onto the faces of the participants using augmented reality (AR) (Photo 1). The AR superimposition was made possible due to edge computing technology, which allows servers in distant locations toprocess images at high speed, allowing for clear, nonblurry images to be placed even if the subjects are
moving around at a high speed.
In addition, projection mapping was done on athree-dimensional (3D) face object. All dynamic performance aspects of kabuki such as timing and facial
expressions were maintained and combined with theconcept of an amplified experience that extracts the
key performance points and makes them stand out ina larger-than-life way.
Moreover, fifty kumadori masks were hung on awall. The masks, which ordinarily should not move atall, made use of the same HenGenTou (deformationlamp) light projection technology that was used in themain performance to freely take on a diverse variety
of facial expressions such as laughter and anger.
Of the 1018 participants from many different age
NTT Technical Review

Feature Articles

Photo 1. Kabuki actor Somegoro Ichikawa experienced the interactive exhibit.

groups, nationalities, and genders, approximately
90% reacted positively to the new interactive experience. The mask-recognition technology that usedangle-free object search had an accurate identification rate of 99.2%. However, some challenges for the
future also became apparent such as finding a way toprovide guest guidance in an exhibit that contains
multiple aspects and optimizing the angle-free object
search lexicon tuning.
1.2 Mid-performance
The mid-performance period introduced the Cho
Kabuki performance and a highly realistic remoterendition of KABUKI LION SHI-SHI-O (Photos 2
and 3).
1.2.1 KABUKI LION SHI-SHI-O
Four initiatives were carried out to achieve thehighly realistic remote rendition of the KABUKI
LION SHI-SHI-O main performance held in Las
Vegas.
(1)	Room reproduction using 4K multi-screens(Photo 4)
Footage from nine 4K-resolution cameras was
encoded using high-compression HEVC (High Efficiency Video Coding). MMT (MPEG* Media Transport), which enables multiple videos and voices to beflexibly synchronized in real time, was then used tocreate the first international 4K multi-screen relay.
Vol. 14 No. 12 Dec. 2016

The Haneda (Japan) venue’s omnidirectional screenson the front stage (three 180-inch screens), stage passage (one lower screen, three rear screens), and ceiling (two upper screens) all simultaneously showed
4K video footage. The display received an energetic
response from over 80% of the 198 participants thatconsisted of media personnel and invitees. Over 70%
responded in a highly favorable manner, stating thatthey would like to visit again. On the system side, thesynchronization and latency both produced positive
results as expected, and various parameters were confirmed for future use. However, problems arose withthe linkages between the multiple screens.
(2)	Remote greeting using pseudo-3D images generated by Kirari! (Photo 5)
Subject abstraction technology was used to finely
extract live footage of the performers from their backgrounds. Synchronization of audio with the pseudo3D video display at the remote stage enabled Somegoro Ichikawa, who was present at the live event in
Las Vegas, to greet those viewing the remote stage atthe Haneda venue as if he were there in person. This
was a world-first for this technology. Despite a lackof prior preparation and confirmation of lighting andcamera locations for the actual performance, on thesystem side, the subject extraction and shadow generation was done well and with high precision. Over*

MPEG: Moving Picture Experts Group2

Feature Articles

Photo 2. Poster of KABUKI LION SHI-SHI-O.

Photo 3. Poster of Cho Kabuki.

Photo 4. Highly realistic remote rendition with omnidirectional screens.

70% of viewers replied that the experience felt as if
Somegoro Ichikawa was really standing on the stage
in front of them.
(3)	4K omnidirectional live footage broadcast formobile devices (Photo 6)
By compressing video and transmitting high-defi3

nition footage solely in the direction that the user islooking, we were able to create a system that controlled bandwidth while maintaining a high-quality
viewing experience (Fig. 1). On the technical side ofthings, the construction and stable operation of a live
transmission were achieved, which reduced bandwidth
NTT Technical Review

Feature Articles

Photo 5. Somegoro Ichikawa in Las Vegas greeting the audience in Haneda, Japan.

Photo 6. 4K omnidirectional live viewing with mobile device.

by approximately 80% when compared to transmissions of the entire 4K omnidirectional video. In addition, many users replied that they felt they were actually at the theater. We confirmed that we were able tomeet the needs of omnidirectional live viewing. However, certain problems arose with image quality.
Vol. 14 No. 12 Dec. 2016(4) HenGenTou performance (Photo 7)
HenGenTou (deformation lamp) technology wasused to create the appearance of a city in the middleof the ocean at the top half of the stage background(15 m high x 4.5 m wide). This technology was also
used to present rippling of the water in the ocean.
4Feature Articles

Proposed method

A: Watching area (high resolution)
B: Whole area (low resolution)
B
B
A
A
Low-resolution
High-resolution tile
tile
Tile corresponding
to user’s gazing
direction

JapanUS
network

EncoderStreaming
serverTIAT SKY HALLat Haneda Airport

Omnidirectional
cameraHMD

David Copperfield Theaterat Las Vegas

HMD: head-mounted display
Omnidirectional camera: A camera that can take a 360-degree video.

HMD

Users can intuitively watch any
area using smartphone HMD.

Fig. 1. A system of 4K omnidirectional live footage broadcast for mobile devices.

Photo 7. Rippling of the water in the ocean was presented on stage using HenGenTou.

Several problems were discovered with the characteristics of the stage, though. For example, when spot5

lights were shown on actors, the stage setting andmovements at the top half of the background did not
NTT Technical Review

Feature Articles

SeattleGEMnet2 NOC(Musashino City)
Internet2

Las Vegas venue

Hanedavenue

Route 1
Route 2

Fig. 2. Outline of Japan-US network for video transmission.

stand out very much. Also, we recognized the importance of having a script in line with the performance.
We constructed a video transmission network linking the Las Vegas and Haneda venues for the remote
live viewing (Fig. 2). Although the period of time waslimited, we were able to construct two paths, whichwere guaranteed for a 1-Gbit/s bandwidth, by combining a line from Internet2, the US research consortium, and GEMnet2, the network testbed owned andoperated by NTT Service Evolution Laboratories.
During the performance, packet losses were smallenough for the error correction function to handle.
Future challenges include monitoring micro-burst
traffic, reducing costs, and further reducing construction times.
1.2.2 Cho Kabuki
The Cho Kabuki event consisted of five performances of the kabuki play Hanakurabe Senbonzakura. Subject extraction and virtual speaker technologies were used to create highly realistic performancesby extracting the figure of the samurai Sato Tadanobu(played by Shido Nakamura) and making it feel as if
voices were coming directly from Hatsune Miku’s
mouth. The performance received high acclaim as a
new form of kabuki performance, with many men-Vol. 14 No. 12 Dec. 2016

tions from television and Internet media sources.
1.3 Post-performance
For the post-performance period, we created anexperience that used a cardboard craft box combinedwith a smartphone to provide a simple 3D video display. Looking inside the box would reveal a palmsized Hatsune Miku rising up in 3D fashion. This
allowed users to experience the vibe of Cho Kabuki.
2. Future development
This project was the first step for kabuki × ICT. In
preparation for the next generation of kabuki, we will
pursue even more realistic forms of expression andcreate new kabuki space experiences (installations).
We also plan to continue these stage performancesand live remote performances in more trials with general users from all over the world to obtain their feedback. In addition to pursuing the development ofpractical-use services with business potential, weplan to take the knowledge gained from ventures in
the field of entertainment and make use of it in hospitality, offering new and emotional experiences with
our eyes on 2020.

6

Feature Articles

Tomohiro Nishitani

Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received his B.S. and M.S. in physics from
Tokyo Institute of Technology in 1997 and 1999.
Since joining NTT in 1999, he has been involved
in researching peer to peer systems and network
address translation traversal and developing VoIP
(voice over Internet protocol) applications for
smartphones. He has been with NTT Service
Evolution Laboratories since 2015. His current
research interests include media art using cutting-edge ICT. He is a member of the Institute ofElectronics, Information and Communication
Engineers (IEICE), the Information Processing
Society of Japan (IPSJ), the Acoustical Societyof Japan, and the Project Management Institute.

Akira Ono

Senior Research Engineer, Supervisor, Natural
Communication Project, NTT Service Evolution
Laboratories.
He received an M.E. in computer engineeringfrom Waseda University, Tokyo, in 1992. He
joined NTT in 1992 and engaged in research anddevelopment (R&D) of video communication
systems. From 1999 to 2010, he was with NTT
Communications, where he was involved in network engineering and creating consumer services. He moved to NTT Cyber Solution Laboratories (now, NTT Service Evolution Laboratories)
in 2010. Since 2015, he has been studying theimmersive telepresence technology called“Kirari!”.

Tomoyuki Kanekiyo

Senior Research Engineer, Supervisor, Natural
Communication Project, NTT Service Evolution
Laboratories.
He received a B.E. in applied physics engineering from Osaka University in 1992. Since joining
NTT in 1992, he has been researching video
distribution systems and developed a commercial
IPTV system and broadcast system for mobile
phones. He is currently researching ultra-realistic
communication systems.

Akio Kameda

Research Engineer, Visual Media Project, NTT
Media Intelligence Laboratories.
He received a B.E. and M.E. in electrical engineering from Tokyo University of Science in1993 and 1995. In 1995, he joined NTT Human
Interface Laboratories, where he was involved inresearching video communication systems on
narrowband networks. Since 2012, he has beenwith NTT Media Intelligence Laboratories,
where he has been engaged in R&D of an interactive panorama video distribution system. He is amember of IEICE.

Shin’ya Nishida

Senior Distinguished Scientist, Group Leaderof Sensory Representation Research Group, NTT
Communication Science Laboratories.
He received a Ph.D. in psychology from Kyoto
University in 1996. He joined NTT in 1992. Hespecializes in psychophysical research on humanvisual processing, particularly motion perception, cross-attribute/modality integration, timeperception, and material perception.

Junichi Nakagawa

Senior Research Engineer, Supervisor, 2020
Epoch-making Project, NTT Service Evolution
Laboratories.
He received a B.E. and M.E. in mechanicalengineering from Waseda University, Tokyo, in1988 and 1990, and an M.S. in information networking from Carnegie Mellon University, USA,in 2000. He joined NTT Communication Network Research Laboratories in 1990. He is amember of IPSJ and the Institute of Electricaland Electronics Engineers.

Motohiro Makiguchi

Takahiro Yamaguchi

Senior Research Engineer, Supervisor, NTT
Network Innovation Laboratories.
He received a Ph.D. in electronic engineeringfrom the University of Electro-Communications,Tokyo, in 1998. He joined NTT Optical Network
Systems Laboratories in 1998 and has been
researching super high-definition image distribution systems. He is a member of IEICE and theInformation and Television Engineers of Japan.

7

Research Engineer, Natural Communication
Project, NTT Service Evolution Laboratories.
He received his B.S. and M.S. in information
science from Hokkaido University in 2006 and2012. Since joining NTT in 2012, he has been
researching human computation and crowdsourcing systems and multilayer floating-image
projection systems using smartphones. His current research interests include virtual reality andmixed reality.

NTT Technical Review

Feature Articles: 2020 Showcase

2020 MICE—New Hospitality
through Exhibitions × ICTManabu Motegi, Kyoko Hashiguchi, Haruno Kataoka,
Takahiro Kusabuka, Yusuke Tanaka, Ushio Shibusawa,
Tetsurou Yamazaki, and Tomoaki Ogawa
Abstract

NTT Service Evolution Laboratories is now verifying technologies on site in preparation for 2020
Showcase. In line with this verification, the R&D Forum Showcase was exhibited at NTT R&D Forum2016 held in February 2016 as an example application in the MICE (meetings, incentives, conventions,exhibitions) field. This article introduces research technologies that visitors were able to experiencethrough the exhibits and through the official application of this event.
Keywords: showcase, MICE, NTT R&D Forum

1. Introduction
At NTT Service Evolution Laboratories, we are
striving to create cutting-edge ubiquitous broadband
service technologies and promoting research anddevelopment (R&D) of service-provision platform
configuration technologies. In particular, as we prepare for 2020 Showcase and target visitors to international conferences and events, we are performing
verification tests focused on the MICE (meetings,
incentives, conventions, exhibitions) [1] field. In thisarticle, we describe R&D Forum Showcase, which
was exhibited at NTT R&D Forum 2016, as an example of the above-described efforts.
2. R&D Forum Showcase exhibition
At NTT Service Evolution Laboratories, we are
expanding demonstration systems that we havedeveloped and performing verification tests focusedon the MICE field. We aim to not only introduce new
business and services based on these technologies
and apply them by 2020 at certain events, but also tocontinuously expand them after the events. To
achieve this aim, we are continuing R&D and systemization aimed at providing an epoch-making hospital1

ity user interface/user experience (UI/UX) by using
the most advanced technologies [2–4].
Held every year in February, the NTT R&D Forumis an exhibition for introducing cutting-edge technologies developed at NTT laboratories. It is an
important event for exhibiting the results of research
performed with our business partners. The main elements of the R&D Forum Showcase are describedhere.
(1) NTT R&D Forum 2016 Official App
The NTT R&D Forum 2016 Official App (hereafter, official app) was developed as a MICE-support
application with the goal of displaying information ina manner that allows visitors to effortlessly tour the
NTT R&D Forum as well as experiencing on the app
itself the latest technologies developed at the laboratories (Fig. 1). These technologies are explained indetail in section 3.
(2) Digital signage
Digital signage (Fig. 2) helped visitors to efficiently tour the Forum by displaying guidance information
via projection mapping and LED (light emitting
diode) panels as well as projecting a congestion map
indicating the state of congestion at each exhibit at
the Forum (as explained later). In contrast to conventional signboards in which information is fixed and
NTT Technical Review

Feature Articles

Visual-search function

One-minute video playbackof recommended exhibits only

Angle-freeobject searchtechnology

Vicinity-map display

PDF display

Display of
exhibition
contents

App download
Google Play or App Store

Congestion-map function
People-flow
prediction
technology

Targets scanned by smartphone
Wi-Fi radio-map
functionRadiowave
environment visualization
technology

1 Exhibition panel (about 120)
2 NTT History Center ofTechnologies display items (150exhibits)
3 Back of download introduction card
4 Companion pochette
5 Information sign at Mitaka Station
6 Musashino R&D Center Main Building
5
6

1

2

3

4

Fig. 1. NTT R&D Forum 2016 Official App.

LED panel installed on walls

Projection mapping on glass wallsof elevator tower

Fig. 2. Digital signage at NTT R&D Forum.

remains static, digital signage can present information in a manner that is instinctively easy to understand by changing the content flexibly according to
the state of congestion and by utilizing moving animation.
Vol. 14 No. 12 Dec. 2016(3) Showcase Cockpit
The Showcase Cockpit was configured with theaim of enabling visitors to the Forum to imagine how
the event was operating (Fig. 3). This was done bydisplaying in real time the state of the venue (i.e.,2Feature Articles

Cockpit: To visualize operations (e.g., unnecessary compatibility adjustment and possible data collection/analysis on the systemenvironment side), the following content was displayed in real time on the multi-console.
Displayed content: status of the R&D Forum 2016 site (congestion map for operators, Wi-Fi radio map, etc.); status of
network utilization; utilization status of application (visual-search function)
XFARM: A system environment for supporting the official app and digital signage was provided.
Entire Showcase Cockpit

Utilization status of cockpit network

Congestion map for
cockpit operators

Recommended

2F Exhibit hall 1
High precision
video/audio

Cloud
Softwaredevelopment
Security
Highly realistic
communication

Hospitality

Application (visual-search function)utilization status 1

Application (visual-search function)utilization status 2

Scanned-items ranking

Scanned-items map
Cloud
Softwaredevelopment
Security
Highly realistic
communication

High precision video/audio

Hospitality

Fig. 3. Showcase Cockpit.

degree of congestion) via a congestion map and
Wi-Fi radio map at each exhibition (explained in thefollowing sections), and the state of utilization of the
network and various applications on a multi-console.
In fact, the information presented on the Showcase
Cockpit is in essence a real-time monitoring of theutilization state of the official app. For example,
when the visual-search function (explained below)
was used at the venue, its utilization state was displayed on the map in real time. This made it possible
to find out whether a particular booth was drawing
attention. XFARM [5]—a cloud platform of NTT
laboratories for service creation—was applied under
this implementation setting. The use of XFARM
makes it possible to smoothly promote activities
ranging from service visualization to development
and operation trials in the manner we achieved for the
R&D Forum Showcase.
3. NTT R&D Forum 2016 Official App
The official app was developed as a MICE-support
application for Android*1 smartphones and iPhones*2.
3

It was made available via the respective app stores foreach type of phone (until August 2016) so that visitors could use it on whichever smartphone they possessed. At NTT R&D Forums held in past years, no
MICE-supported application was developed, so the2016 Forum marks the first time such an app wastried. Of the approximately 10,000 visitors to theForum (over the four days it was held), about 4600
downloaded the official app, more than 40% of thetotal number of visitors. The number of downloads
was about 20% of the number of visitors with standard MICE-support apps, and the official app was
rated highly by those who downloaded it.
The official app provides the functions of standard
MICE-support apps (namely, displaying the content
of each exhibit, lecture schedules, access information, and providing connection to free Wi-Fi services)
and also provides the following functions to showcase technologies developed at the laboratories:*1 Android is a registered trademark of Google Inc.
*2 iPhone is a trademark of Apple Inc., registered in the U.S. andother countries.

NTT Technical Review

Feature Articles

By setting target personas in our designs and making decisions focused on users, we ensure that screentransitions, visual design, and so on, are consistent.
Persona:
A social demographic (40 to 55 years old) that might make use of technologies
developed at our laboratories
Aim of app:
To trigger collaboration with external parties through the appeal of our technologies
Keywords: Introduce the simplicity and appeal of our technologies at a glance.

Makes technology appealing

SimpleMain content is placed at the top
in a predominant manner.
(Wi-Fi radio map is combinedwith free Wi-Fi)
Increased number of accesses

Few colors (only 3)
Fonts are unified.
No unnecessary
information

SimpleSix content items on one screen
A layout that doesn’t confuse
the user

Fig. 4. Human-centered design.

(1) Visual-search function(2) Congestion-map function(3) Wi-Fi radio-map function
Moreover, to improve the UX of each visitor, weapplied human-centered service design [6] at thedesign stage of the UI of the official app (including
the functions listed above). An example of thishuman-centered service design is shown in Fig. 4. In
concrete terms, we studied visitor-application scenarios of the official app, determined the social
demographic of the target users, and designed the appon the basis of the persona (i.e., virtual character) ofthat social demographic. In developing the app, we
set the persona as a character (namely, a person 40 to
55 years of age in a management position) who has
the power to influence decisions on business collaborations with outside organizations, and on the basis ofthat persona, we put the required specifications forcreating the official app into writing. We focused on
three key points regarding screen transitions andvisual design—namely, making the technology
appealing, maintaining simplicity, and introducing
the simplicity and appeal of our technologies at a
glance—and we integrated these points into thedesign of the official app’s UI.
Vol. 14 No. 12 Dec. 2016

For example, displayed content was narrowed
down to six items and placed on the top screen of theofficial app to maintain simplicity, and a function toappeal to the user via a showcase was located boldlyin the upper part of the screen as the main content.
The content (including the function to appeal to theuser) was displayed in purple to give the whole UI acalm look. Furthermore, the official app was configured so that technology introduction pages could be
individually introduced in a simple manner and technological points could be understood easily while theapp was operated.
3.1 Visual-search function
The visual-search function applies angle-freeobject search technology developed by NTT MediaIntelligence Laboratories. When the visitor photographs an exhibit panel, the function displays detailed
information concerning the photographed object.
This technology can distinguish unwanted things
included in the target object even if the image is captured from various angles, and it makes it possible todetermine the content of an exhibit even from a distance by photographing the exhibit panel with asmartphone camera. In this way, even if the writing
4Feature Articles

The use of BLE beacons and smartphone apps makes it possible to collect, forecast, and visualize
people-flow data.
Event administrator/
cockpitCongestion
map
Congestion map(present status andforecast status in 10
minutes) is provided
for administrators.

Congestion map is generated (present
status/forecast status in 10 minutes).
Analysis (forecast) server
Data-collection server
Beacon-detection log

Present-congestion
map is provided
for the app.

App

App

Bluetooth low energy (BLE) beacon

Venue

Fig. 5. Overview of congestion-map function.

on a panel at a booth cannot be deciphered, it is possible for visitors to confirm the details about thebooth and decide whether to walk around a little andcome back to the exhibit at another time if the exhibit is congested.
The objects photographed using the visual-searchfunction are shown at the bottom right of Fig. 1.
When about 150 exhibited objects from the NTT History Center of Technologies (set up at the exhibition
venue) and about 120 display panels were photographed, detailed information about the exhibited
objects was presented to the user. Then, when visualsearch marks (displayed on introduction cards distributed outside the exhibition venue, posters inside
the venue, companion pochettes (small bags), etc.)
are photographed, special content is presented to theuser.
Moreover, when information boards at Mitaka Station are photographed with the app-loaded smartphone, bus timetables to the exhibition venue aredisplayed on the smartphone, and when the NTT
Musashino Main Building is photographed, originalcontent is provided. This enabled visitors to experience a visual-search function in correspondence withvarious physical objects. Furthermore, when we analyzed the recognition results concerning the images
taken by the visitors, it became clear that the captured
objects were accurately recognized even when thetarget was far away or at a difficult-to-capture angle.
In other words, we confirmed that the app-service can5

be utilized even when the target objects and theirenvironment are constrained.
In combination with an advanced transparent display device (developed by Panasonic Corporation)[7], the visual-search function successfully demonstrated a futuristic kind of information navigation.
Using such a transparent display device makes it possible to display information superimposed onto thetarget object, for example, with a sentence informing
the user that the tall nearby structure is Tokyo Tower.
This manner of providing information can be targetedat foreign visitors to Japan (whose number is forecast
to increase over the next few years), and detailed
information navigation can be provided in accordance with the attributes and movements of each andevery visitor.
3.2 Congestion-map function
The congestion-map function utilizes people-flow
forecasting and induction technology developed by
NTT Service Evolution Laboratories to tell visitors
which areas of the exhibition venue are crowded ornot. An overview of the congestion-map function isshown in Fig. 5.
The congestion-map function consisted of Bluetooth*3 low energy (BLE) beacons installed at morethan 150 locations around the exhibition venue.
When visitors carrying an official-app-installed*3 Bluetooth is a registered trademark of Bluetooth SIG, Inc.

NTT Technical Review

Feature Articles

BLE beaconsfor obtaining
location information

Everyone’s locations arecollected and analyzed on
the cloud.

Classifying trends of radio-wave
usage and making utilization patterns

Network
Optimizing wirelessconfiguration policy
according to the patterns

Acquired user-location
data are linked touser-side information.

User terminals

Wi-Fi
accesspoints• Wi-Fi signal strength, control-packet information, and traffic• Information from sensors implemented in smartphones

Updated in real time
Wi-Fi radio map

Reception
CloaksRestaurant

Fig. 6. Eco-cycle of radiowave environment visualization and Wi-Fi radio-map function.

smartphone entered the venue, the identification of
the BLE beacon with the highest received power wastransmitted to the cloud. On the cloud, the numbers ofsuch beacons were counted, and a congestion map(indicating present congestion and forecasted congestion in 10 minutes) was formulated in the form ofa heat map. A characteristic feature of this peopleflow forecasting technology is that even if measurement data are not accumulated from this physicalenvironment over a long period of time, it is possible
to forecast congestion in 10 minutes from the mostrecent data (i.e., data collected in the preceding 30
minutes). Accordingly, an advantage of this technology is that it can also be used at temporary sites in
addition to permanent ones such as that of the R&D
Forum. Although the official app currently only displays the congestion map, the visualized and forecasted (10 minutes in the future) people flow (calculated from data collected in the past 30 minutes) canbe displayed on digital signage (as shown in Fig. 2).
In this way, presenting the degree of congestion ineach exhibition area to the visitors makes it possible
for them to experience people-flow forecasting andinduction technology for themselves.
3.3 Wi-Fi radio-map function
The Wi-Fi radio-map function utilizes radiowaveenvironment visualization technology developed by
NTT Access Network Service Systems Laboratories
to display the signal strength of free Wi-Fi accesspoints. An overview of the radiowave-environment
Vol. 14 No. 12 Dec. 2016

visualization technology and the Wi-Fi radio-map
function is shown in Fig. 6.
Information related to radio waves is gatheredusing an approach called collective intelligence.
Namely, Wi-Fi signal strength is measured and collected using the terminals of all visitors on which theapplication is installed. Thereby, the current radio
map can be formed without dedicated measuring
instruments. When the signal strength is gathered, thedata are combined with the position information
obtained by the location function from the BLE beacons (which are also used by the congestion-map
function described above).
When the Forum was held, free Wi-Fi areas were
set up around the venue so that official-app-installed
smartphones could access free Wi-Fi. Accordingly,
by indicating areas with easy connection to free
Wi-Fi to visitors, we enabled visitors to experience
the radiowave-environment visualization technology
for themselves.
4. Future development
This article introduced the R&D Forum Showcase—which was exhibited at the NTT R&D
Forum—as an example of verification tests in the
MICE field. In particular, three functions provided by
the NTT R&D Forum 2016 Official App—namely, avisual-search function, congestion-map function, and
Wi-Fi radio-map function—were described as showcase technologies developed at NTT laboratories.
6

Feature Articles

At NTT Service Evolution Laboratories, we willcontinue working to improve these technologies intime for the year 2020 through collaborations withpartners in different industries. Moreover, we willcontinue to promote R&D aimed at applying thesetechnologies in MICE fields such as exhibitions,museums, art galleries, airports, and train stations.

[4]

[5]

References
[1]

[2]

[3]

7

Tokyo Metropolitan Government, “2nd Chapter: Promoting MICE
Hosting in ‘Tokyo Metropolitan Tourism Plan,’” (in Japanese).
http://www.metro.tokyo.jp/INET/KEIKAKU/2013/05/DATA/
70n5m106.pdf
N. Kanamaru, H. Tezuka, A. Fukayama, Y. Nakamura, H. Yamaguchi,
and M. Motegi, “Creating Omotenashi Services for Visitors and Spectators in 2020,” NTT Technical Review, Vol. 13, No. 7, 2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201507fa6.html
M. Horii, K. Arai, M. Nagata, K. Kashino, K. Hiramatsu, A. Fukayama, and H. Yamaguchi, “Media Processing Technology for Achieving
Hospitality while on the Go,” NTT Technical Review, Vol. 13, No. 4,

[6]

[7]

2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201504fa4.html
K. Sadamitsu, J. Shimamura, G. Irie, S. Tarashima, T. Yoshida, R.
Higashinaka, H. Nishikawa, N. Miyazaki, Y. Iijima, and Y. Nakamura,“Media Processing Technology for Achieving Hospitality in Information Search,” NTT Technical Review, Vol. 13, No. 4, 2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201504fa5.html
K. Onishi, Y. Tomita, H. Tate, M, Urata, N. Uchida, H. Masuda, T.
Kono, and T. Komiya, “The XFARM Platform for Faster Service
Creation,” NTT Technical Review, Vol. 11, No. 9, 2013.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201309fa6.html
Y. Asano. T. Ohno, S. Miyahara, M. Watanabe, M. Nakashige, A.
Kimura, and K. Wago, “Towards the Creation of Attractive Services
Based on an Understanding of Users,” NTT Technical Review, Vol.
13, No. 7, 2015.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201507fa5.html“Focus on the News: Starting Verification of Technologies for Creating Hospitality Services Provided via Simple and Portable Terminals
Fitted with Transparent Displays,” NTT Technical Journal, Vol. 28,
No. 5, pp. 57–59, 2016 (in Japanese).

NTT Technical Review

Feature Articles

Manabu Motegi

Senior Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received his M.E. and Ph.D. in mechanomicro engineering from Tokyo Institute of Technology in 1995 and 2008. In 1995, he joinedNTT. His research interests include multi-robot
collaboration, human-robot collaboration, anddata mining with multi-sensors. He is currently
involved in R&D on MICE applications andweb-based digital signage. In 1998, he was honored with an Encouragement Award from the
Robotics Society of Japan.

Kyoko Hashiguchi

Researcher, 2020 Epoch-making Project, NTT
Service Evolution Laboratories.
She received her B.E. and M.E. from Keio
University, Tokyo, in 2011 and 2013. She joined
NTT Service Evolution Laboratories in 2013 andhas been engaged in research on human-computer interaction and user-experience design. She iscurrently studying dynamic guide-sign systems
for pedestrians. She is a member of the JapaneseSociety for the Science of Design.

Haruno Kataoka

Researcher, 2020 Epoch-making Project, NTT
Service Evolution Laboratories.
She received her B.E., M.E., and Ph.D. fromthe University of Electro-Communications,Tokyo, in 2006, 2008, and 2015. She joined NTT
Service Platform Laboratories in 2008 and carried out R&D on enhanced network services forend users. She is currently studying dynamic
guide-sign systems for pedestrians.
She received the Information Network
Research Award from the Institute of Electronics,
Information and Communication Engineers(IEICE) in 2010, the Best Paper Award from theJapan Society of Security Management in 2010,
the Best Paper Award from the International
Conference on Intelligence in Next Generation
Networks in 2012, and the 50th Excellent Paper
Award from the Information Processing Societyof Japan (IPSJ) SIGUBI Workshop in 2016. She
is a member of IEICE and IPSJ.

Takahiro Kusabuka

Researcher, Networked Robot and Gadget
Project, NTT Service Evolution Laboratories.
He received his B.E. and M.S. in engineeringfrom Ritsumeikan University, Shiga, in 2010 and2012. He joined NTT in 2012 and studied technical support systems and interaction systems
using taste. He is currently studying architecturesof cooperation systems between devices andcloud services utilizing device-function virtualization. He is a member of the Association for
Computing Machinery.

Vol. 14 No. 12 Dec. 2016

Yusuke Tanaka

Research Engineer, Proactive Navigation Project, NTT Service Evolution Laboratories.
He received his B.E. and M.E. in informationengineering from Kyoto University in 2006 and2008. He joined NTT WEST in 2008 and moved
to NTT in 2015. His current research interests
include systems for measuring and analyzing theflow of people.

Ushio Shibusawa

Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received his B.E. and M.E. in computerscience from the University of Electro-Communications, Tokyo, in 2006 and 2008. He joined
NTT Cyber Solutions Laboratory in 2006 andstudied knowledge management, office automation systems, Internet protocol television (IPTV)
services, and web technologies and developed
the firmware of the Customer Terminal Unit for
the NTT FLET’S Hikari Premium service. From2012 to 2015, he worked at NTT Plala as a planner and developer of new services for an IPTV
service called Hikari-TV. He is currently studying object-recognition-based information services for foreign visitors to Japan. He is a memberof IEICE and IPSJ.

Tetsurou Yamazaki

Research Engineer, 2020 Epoch-making Project, NTT Service Evolution Laboratories.
He received an M.E. in information engineering from the University of Iwate in 1989. He
joined NTT in 1989 and began R&D on speechquality assessment, telematics systems, ID-based
digital signatures, and active-fragment aggregation certification. He is a member of the Acoustical Society of Japan.

Tomoaki Ogawa

Senior Research Engineer, Wireless Access
Systems Project, NTT Access Network Service
Systems Laboratories.
He received a B.A. in environmental information and an M.A. in media and governance from
Keio University, Kanagawa, in 1996 and 1998.
He joined NTT Network Innovation Laboratories
in 1998, where he has been working on indoorlocation systems. His recent interest focuses ondevelopment of wireless local area network services. He is a member of IEICE.

8

Feature Articles: 2020 Showcase

Promotion of Co-innovation throughCollaboration with Different Business
Sectors
Tsuguhiro Ohashi and Shinkuro Honda
Abstract

NTT aims to create new services by working with different business sectors as value partners, while
shifting its business model from B2C (business-to-consumer) to B2B2X (business-to-business-to-consumer, government, or company). NTT Service Evolution Laboratories is also promoting co-innovation
with partners in different business sectors. In this article, we describe some examples of the latest efforts
to implement co-innovation activities of the NTT Group.
Keywords: co-innovation, collaboration, value creation

1. Introduction
With the objective of revitalizing the informationand communication technology (ICT) market throughnew-value creation, NTT EAST and NTT WEST
have shifted their business models to a Hikari Collaboration Model, under which their business partners are allowed to re-sell NTT’s optical access services. To accelerate this switch in business modelfrom the conventional B2C (business-to-consumer)
to B2B2X (business-to-business-to-consumer, government, or company), the NTT Group is involved inongoing co-innovation activities with partners from avariety of different business sectors.
For example, in collaboration with four major companies, NTT WEST began efforts [1] in April 2015 tofurther accelerate co-innovation aimed at creating
new business through facilitating cooperation
between large and established businesses and startups. In July 2016, in collaboration with McLaren
Racing Limited and Honda Motor Co., Ltd., NTT
Communications initiated efforts [2] to establish an
optimum ICT platform to support Formula 1 racing
teams.
Over the last three or four years, NTT has maintained business alliances with Dwango Co., Ltd. [3]and Mitsubishi Heavy Industries, Ltd. [4] and is con1

tinuing efforts aimed at creating new value by utilizing research results in the ICT field. The co-innovation effort with Dwango is targeted at upgrading
video and social networking services, so we havebeen jointly developing interactive streaming technology for omnidirectional video, which can deliver
immersive video even when bandwidth is limited.
This technology was implemented as a virtual-reality
live streaming service [5].
The co-innovation project with Mitsubishi Heavy
Industries involves the commercialization of a highnoise-reduction head-set type intelligent microphone,which can collect sound clearly even in noisy environments (i.e., at a noise level close to 100 dB) suchas factories and building sites, through our combined
research and development (R&D) efforts [6].
In addition, we are involved in several other coinnovation activities. For example, in collaboration
with Daiichi Kosho Co., Ltd., we are carrying out
joint investigations using NTT’s speech processing
technology [7]. We believe that the significant resultsof implementing the above-described co-innovation
activities by utilizing NTT’s own technologies will
help to solve problems in society and create value
through new markets.

NTT Technical Review

Feature Articles

Visualization, streamlining, and automation of agricultural management by upgrading agricultural
machinery and farming support systems to achieve high-quality agriculture

Agricultural field

Agricultural machinery

Farming support system(KSAS)**KSAS: Kubota Smart Agri System

Creation of new value through collaboration between
the agricultural industry and the ICT industry

KubotaNTT

Co-innovation• Enhancing competitivenessof products• Speeding up developmentof products• Attainment of collaboration
between the “XX” industry and
ICT industry by applying resultsof R&D in dissimilar fields

Fig. 1. Agricultural-centered co-innovation with Kubota Corporation.

2. Solutions to social problems and creation ofvalue in new markets
The NTT Group aims to support sustainable development in society and is therefore working on solving
various social problems. However, many social problems—such as Japan’s decreasing birthrate and aging
population as well as sluggish employment generation—can neither be solved single-handedly nor by
ICT alone. Nevertheless, we can expect to open up
completely new markets by applying ICT. Under
these circumstances, we aim to solve social problems
and create value in new markets by utilizing NTT
technology. We describe in this section some of ourongoing efforts.
2.1 A gricultural-centered co-innovation with
Kubota Corporation
The field of agriculture is facing serious problems.
For example, Japan’s farmers are aging and dwindling in number since fewer people are entering thefield. In June 2016, NTT signed a comprehensive
partnership agreement with Kubota Corporation, amajor agricultural machinery manufacturer [8]. By
focusing on the agricultural-solutions field, which
Kubota is trying to expand, and combining ICT serVol. 14 No. 12 Dec. 2016vices provided by various NTT Group companies, weare coming up with ground-breaking innovations thattackle problems such as rejuvenation of farming communities (focusing on the lack of replacement farmworkers and improvement of the international competiveness of Japan’s farms) (Fig. 1).
In particular, we are utilizing global services provided by NTT Communications (such as cloud andsecurity services), cutting-edge ICT services provided by NTT Group companies (such as wireless technology, weather information, and map information),and fault-detection and IoT (Internet of Things) technologies based on artificial intelligence [9] (mainly
being researched and developed at NTT laboratories),
in order to upgrade agricultural machinery and farming support systems provided by Kubota. Throughthese upgrading efforts, we plan to promote R&D andprovide services leading to value creation in the agricultural field by improving international competitiveness (by volume production of high-quality farm
produce) and by visualizing and automating farming
to improve its operational efficiency.

2

Feature Articles

Overview of service and system configuration
Improvement ofanalysis algorithm

Function (2)Analysis using mostsuitable algorithm
Collected data

XX level:

Cloud serverS

97 pts.

Function (1)Fear

Extracting vital data
Nervousness

Function (3)Insecurity

Display of results of analysis
A person’s emotional
states at various events
can be understood.

Data collection device

Happiness

Reassurance

Smartphone etc.
Extracting data without
discomfort by utilizing
technology developed by
NTT Group

Offering a new experienceFig. 2. A service for visualizing the heart.

2.2 D evelopment of a service aimed at creating
new markets: visualizing the heart by utilizing
vital data
In 2015, NTT WEST—in collaboration with NTT
Smart Connect, Aichi Prefectural University, andNTT—launched a service called visualizing the heart[10]. This cloud-type service can analyze people’semotions by analyzing a person’s pulse-wave data(measured by surface-type pulse-wave sensing technology developed by NTT laboratories) and applying
an algorithm for deducing changes in a person’s emotional state (developed by NTT WEST in collaboration with Aichi Prefectural University) (Fig. 2).
This service is intended to open up a new market
that provides new value by utilizing vital data. At the
Umeda Haunted House 2015 event (sponsored by
Mainichi Broadcasting System, Inc., and held from
July 10 to September 6, 2015), the fear people wereexperiencing was visualized by applying the visualizing the heart service. This service was also utilized
at Smart Hikari Futsal (jointly sponsored by Tokyu
Sports Oasis, Inc., Goldwin Inc., and NTT WEST,
and held from January 24 to March 27, 2016) to visualize the amount of exercise of volunteer subjects.
Furthermore, on the Smart Hikari Comedy Theatre
show (jointly hosted with Yoshimoto Kogyo Co., Ltd.
3and performed from February 3 to March 28, 2016),
peoples’ laughter was visualized (with a laughometer) by utilizing this service. To visualize laughter inthis manner, the service applies an analysis algorithm
developed by NTT laboratories.
3. Future development
In this article, examples of our latest efforts concerning co-innovation activities were described. NTT
Service Evolution Laboratories will continue to promote co-innovation in cutting-edge R&D under theaim of creating value in collaboration with NTT
Group companies and partners in various fields.
References
[1]
[2]
[3]
[4]
[5]
[6]

Press release issued by NTT WEST on April 17, 2015 (in Japanese).
http://www.ntt-west.co.jp/news/1504/150417a.html
Press release issued by NTT Communications on July 8, 2016.
http://www.ntt.com/en/about-us/press-releases/news/article/2016/
20160708.html
Press release issued by NTT on July 29, 2013 (in Japanese).
http://www.ntt.co.jp/news2013/1307/130729a.html
Press release issued by NTT on April 28, 2014.
http://www.ntt.co.jp/news2014/1404e/140428a.html
Press release issued by NTT on November 20, 2014.
http://www.ntt.co.jp/news2014/1411e/141120a.html
Press release issued by NTT on October 27, 2015 (in Japanese).

NTT Technical Review

Feature Articles

[7]
[8]

http://www.ntt.co.jp/news2015/1510/151027b.html
Press release issued by NTT on July 13, 2015 (in Japanese).
http://www.ntt.co.jp/news2015/1507/150713b.html
Press release issued by NTT on June 7, 2016.
http://www.ntt.co.jp/news2016/1606e/160607a.html

Tsuguhiro Ohashi

Senior Research Engineer, Co-innovation Promotion Project, NTT Service Evolution Laboratories.
He received a B.E. and M.E. in materials science and engineering from Nagoya Institute ofTechnology, Aichi, in 1998 and 2000. He joined
NTT WEST in 2000. In August the same year, hemoved to NTT Cyber Solutions Laboratories,
where he researched human interface technology. In 2004, he moved to NTT WEST again,
where he developed network services. He is currently involved in co-innovation promotion.

Vol. 14 No. 12 Dec. 2016

[9]

Press release issued by NTT on May 30, 2016 (in Japanese).
http://www.ntt.co.jp/news2016/1605/160530b.html[10] Press release issued by NTT WEST on July 9, 2015 (in Japanese).
http://www.ntt-west.co.jp/news/1507/150709a.html

Shinkuro Honda

Senior Research Engineer, Supervisor, Coinnovation Promotion Project, NTT Service
Evolution Laboratories.
He received a B.E., M.E., and Ph.D. in engineering from Keio University, Kangawa. He
joined NTT Cyber Space Laboratories in 1999,
where he led a virtual reality and cyberspace
R&D project and developed collaboration systems. He spent time at the NTT Group San Jose
Office, USA, where he was involved in incubating a new business endeavor with venture firms.
He joined NTT Service Evolution Laboratories
in 2015, where he is involved in co-innovation
promotion. He is a member of the Institute ofElectronics, Information and Communication
Engineers, the Information Processing Society ofJapan, and the Virtual Reality Society of Japan.

4Regular Articles

Milagro Multi-Factor Authentication
Masahiro Matsui, Hiroaki Ohtsuka, Tetsutaro Kobayashi,
Hironobu Okuyama, Akira Nagai, and Go Yamamoto
Abstract

Apache Milagro (incubating) is an open source project to establish open source software (OSS) for
cloud computing. A system designer can choose the M-Pin Authentication Protocol (M-PIN) or theextended M-Pin Authentication Protocol (e-M-PIN) in Milagro Multi-Factor Authentication (MilagroMFA), which is an authentication system in Apache Milagro (incubating). Additionally, e-M-PIN is a
non-interactive protocol and is compatible with password-based Hypertext Transfer Protocol (HTTP)
authentication methods such as Basic and Digest Access Authentication since password-based HTTP
authentication is also non-interactive. Thus, an authentication system that uses password-based HTTP
authentication can be easily migrated to e-M-PIN. We presented e-M-PIN at ApacheCon North America
held in May 2016 as a contribution for the OSS community.
Keywords: identity-based authentication, M-PIN, Apache Milagro

1. Introduction
Apache Milagro (incubating) [1] is an open source
project focused on new authentication technology
and was developed by MIRACL, NTT Innovation
Institute, Inc. (NTT i3), and NTT [2]. One of theauthentication systems in Apache Milagro (incubating) is Milagro Multi-Factor Authentication (Milagro-MFA). When using Milagro-MFA, the systemdesigner can choose the M-Pin Authentication Protocol (M-PIN) [3] or the extended M-Pin Authentication Protocol (e-M-PIN). We introduce in this article
e-M-PIN, an improved M-PIN authentication protocol that the authors developed.
The common features of M-PIN and e-M-PIN are
as follows.
(1) ID-based cryptosystem
M-PIN and e-M-PIN are identity (ID)-based
authentication methods that are powered by pairingbased cryptography. A user has his own ID and asecret key, which is related to the ID. The server
authenticates the user with the ID, the user secret key,and server secret key.
(2) Multi-Factor Authentication
M-PIN and e-M-PIN are user authentication methods based on the user (client) - web server model.
1

Each user has two secrets; one is a secret key stored
in the user’s computer, and the other is a PIN (personal identification number) code that the user must
remember. Even if one of the secrets is leaked, thesecurity of both protocols is still secure.
(3) No need for specific hardware
The user can login using a web browser. Thus, theserver does not need to prepare specific secure hardware such as a hardware token for the user. Each user
only needs his own computer or smartphone.
(4) No stored user hashed passwords
The server does not have to store a user’s hashedpassword in order to authenticate the user. Therefore,
if the server is compromised, the user’s hashed password is not leaked. In contrast, password-based
Hypertext Transfer Protocol (HTTP) authentication
methods such as Basic and Digest Access Authentication need to store the user’s hashed password toauthenticate the user. Consequently, leakage of theuser’s hashed password occurs frequently when theserver is compromised.
The difference between M-PIN and e-M-PIN is in
the amount of communication.
(5) Interactive or non-interactive protocol
M-PIN is an interactive protocol, and e-M-PIN is a
non-interactive protocol. In M-PIN, the server and
NTT Technical Review

Regular Articles

TAsI. Key generation (sA)
II. Getting time permit (sT)Client (IDa)A, T

IDa, UyV
III. Authentication process

ServersQ, TiOnce for each registration
Once for each day
Once for each authentication

Fig. 1. Entities and processes in M-PIN.

the user have to send data to each other. However, thiskind of interactive protocol is not compatible withpassword-based HTTP authentication, which is noninteractive. Therefore, e-M-PIN is designed as a noninteractive protocol in order to easily migrate frompassword-based HTTP authentication to e-M-PIN.
To achieve this, we modified M-PIN without reducing its security. We explain the details of M-PIN in
section 3 and the points that were improved in section 4.
2. Notations
We define here the notations used in the following
sections. The entities are Client, Server, and Trusted
Authority (TA). Three functions are discussed: a pairing function (e), which is on an elliptic curve overfinite field IFp, a one-way function (Hq), and a oneway function on a map to a point on the elliptic curve(H1, H2). The parameters of these functions are basedon the security of cryptography. Therefore, systemdesigners must set the parameters of the functions
securely. Recommended parameters can be found in
publications of the National Institute of Standards
and Technology (NIST) [4], Standards for Efficient
Cryptography (SECG) [5], and the Elliptic Curve
Cryptography (ECC) Brainpool [6]. Secure parameters have already been implemented in the repositoryof Apache Milagro (incubating) [7].
• Entities: Client, Server, TA• q is prime.
•	G1 and G2 are the q-torsion points on the elliptic
curve.
Vol. 14 No. 12 Dec. 2016

• Zq is the q-order subgroup of the finite field.
• Pairing e: G1 × G2 → Zq
• Hq: {0,1}*→ Zq• H1: {0,1}*→ G1• H2: {0,1}*→ G2•	P and Q are the respective generators of G1 andG2.
3. M-PIN
In this section, we provide the details of the M-PIN
protocol (Fig. 1). M-PIN comprises three steps: Keygeneration, Getting a time permit, and an Authentication process.
As a preliminary, the server gets a server secret key(sQ) and a today’s date (Ti) in advance. Then, the client processes the Key generation and Getting a time
permit steps. In the Key generation part, the client
gets a client secret key (sA). Key generation is done
only once for each user. In Getting a time permit, theclient gets today’s time permit (sT), which is generated from today’s date (Ti). Thus, the Getting a time
permit step has to be done once a day for each user as
the first step to authenticate the user.
I. Key generation:1)	The client calculates a point (A) on the ellipse
curve by using his ID (IDa) and sends it to TA inorder to generate a client secret key (sA).
2)	The client calculates the token ((s–α)A) using
his own PIN (α) and stores the token in local
storage.
An image of the M-PIN user registration page is2

Regular Articles

IDa(email address)

PIN (α)Fig. 2. Image of user registration page of M-PIN.

shown in Fig. 2. The user enters his PIN (α), and thebrowser calculates the token ((s–α)A).

IDa(emailaddress)II. Getting a time permit:1)	The first time the client logs in to the server eachday, the client sets T← H1 (Ti||IDa) according totoday’s date (Ti).
2)	The client gets a time permit (sT) by sending T
to TA and stores it in local storage.
After completing these steps, the client has his own
ID (IDa) and a client secret key (sA) and a time permit(sT). The client generates a one-time signature (V)with the above information and sends this one-time
signature to the server. The server verifies the clientwith the one-time signature and the server secret key(sQ).
III. Authentication process:1) The client inputs the PIN (α).
2) The client generates random numbers (x) in Zq.
3)	The client computes D←A+T from A← H1 (IDa)and T← H1 (Ti||IDa).
4) The client computes U←xD from D and x.
5) The client sends {IDa, U} to the server.
6) The server generates a random number (y) in Zq.
7) The server sends y to the client.
8)	T he client computes V← –(x+y)((s–α)A+αA+sT).
9) The client sends {V} to the server.
10) The server computes D← H1 (IDa)+H1 (Ti||IDa).
11) The server computes g←e(V,Q)·e(U+yD,sQ).
12) If g = 1, return true; otherwise, return false.
3

PIN (α)Fig. 3. Image of M-PIN login page.

An image of the login page of M-PIN and the
M-PIN authentication protocol are respectively
shown in Figs. 3 and 4.
3.1 Advantages of M-PIN
M-PIN has some advantages over password-based
HTTP authentication. One advantage is that it uses
two-factor authentication. Another is that the server
does not have to store a user’s hashed password. The
NTT Technical Review

Regular Articles

Client IDa
Input PIN (α) in Z*q
Generate random (x) ∈ Z*q
A ← H1 (IDa)T ← H1 (Ti || IDa)
D ← (A+T )U ← xD

ServerIDa, UyV ← − (x+y) ((s−α)A + αA + sT )V

Generate random (y) ∈ Z*q
D ← H1 (IDa) + H1 (Ti || IDa)g ← e(V, Q).e(U+yD, sQ)
If g = 1 then return true; otherwise
return false

Fig. 4. M-PIN authentication protocol.

client calculates αA with the user’s PIN (α) and adds
it to the token ((s–α)A) stored in local storage. With
the result of this calculation, the client can create aclient secret key (sA). Even if an adversary gets one
piece of secret data (PIN (α) or token ((s–α)A)), hecannot calculate the secret key (sA) without having
the other piece of secret data. In addition, the server
only has to keep the server secret key (sQ) secure.
This means that the server does not have to have theuser’s hashed password. Hence, there is no risk ofleakage of the user’s hashed password.
3.2 Problem in M-PIN
In M-PIN, the client sends his data (IDa, U) to theserver. Then the server sends a random number (y) tothe client, and the client makes a one-time signature(V) that includes y. This means the communication
between the client and server requires three passes.
However, password-based HTTP authentication,
which is one of the most popular forms of authentication, is only one pass. The client sends his password
or his hashed password to the server only one time. If
the server wants to use both password-based HTTP
authentication and M-PIN, the server has to change
the communication protocol of the system in order tocarry out three-pass communications. This is a major
obstacle to be overcome in order to migrate M-PIN.
4. e-M-PIN
In this section, we present the details of the noninteractive authentication protocol e-M-PIN and discuss its advantages and security. This proposed protocol uses one-pass communication between the clientand server. This is the same as password-based HTTP
authentication. Therefore, the server can use both
password-based HTTP authentication and e-M-PIN
without any modification of the communication proVol. 14 No. 12 Dec. 2016

tocol in servers.
However, it is necessary to change the challenge
and response method between the client and server,which brings the risk of a replay attack. To maintainthe same level of security as M-PIN, e-M-PIN uses
some additional information in the authentication
process. The proposed protocol can prevent a replay
attack by using the current time (CCT: Current Client
Time, SCT: Current Server Time) and a nonce (anarbitrary number that can be used only once).
While the Key generation and the Getting time permit processes are the same as that for M-PIN, thepreliminary process is slightly different. In addition
to getting a private key and a time permit, the server
has to set an expiration time (t), which is a time range
to accept or not accept a one-time signature. Entities
and processes in e-M-PIN are shown in Fig. 5.
The authentication process is as below (Fig. 6).
III. Authentication process:1) The client inputs the PIN (α).
2)	The client generates random numbers (x) and anonce in Zq and gets the current time (CCT) fromthe client device.
3)	The client computes D←A+T from A← H1 (IDa)and T← H1 (Ti||IDa).
4)	The client computes U←xD and W←xA from
D, A and x.
5)	The client computes y← Hq (IDa || U || W || nonce|| CCT).
6)	T he client computes V← –(x+y)((s–α)A+αA+sT).
7)	The client sends {IDa,U,W,V,nonce,CCT} to theserver.
8) The server gets the current time (SCT).
9)	If a nonce exists in the database (DB) that theserver has, or the time-gap between SCT and
CCT is not in t, the authentication fails.
4Regular Articles

TAsI. Key generation (sA)
II. Getting a time permit (sT)Client (IDa)A, T

IDa, U, W, V, nonce, CCT
III. Authentication process

ServersQ, Ti, t

Once for each registration
Once for each day
Once for each authentication

Fig. 5. Entities and processes in e-M-PIN.

Client IDa
Input PIN (α) in Z*q
Generate random (x), nonce ∈ Z*q
Get client current time (CCT)A ← H1 (IDa)T ← H1 (Ti || IDa)
D ← (A+T )U ← xDW ← xAy ← Hq (IDa || U || W || nonce || CCT )V ← − (x+y) ((s−α)A + αA + sT )ServerIDa, U, W, V,
nonce, CCT

Get server current time (SCT)
If nonce exists in DB or|SCT−CCT| > t then return false
Add nonce to DBy ← Hq (IDa || U || W || nonce ||CCT )D ← H1 (IDa) + H1 (Ti || IDa)g ← e(V, Q).e(U+yD, sQ)
If g = 1 then return true; otherwise
return false

Fig. 6. e-M-PIN authentication protocol.

10)	The server adds the nonce to its own DB.
11)	The server computes y← Hq (IDa || U || W ||nonce || CCT).
12)	T he server computes D← H 1 (ID a )+ H 1(Ti||IDa).
13) The server computes g←e(V,Q)·e(U+yD,sQ).
14) If g = 1, return true; otherwise, return false.
In the challenge and response method, the server
first sends a challenge, which is a random number.
After receiving the challenge from the server, the client issues a signature with the challenge. Then, theserver verifies the signature. If the signature is issued
with the server’s challenge, the verification is accepted.
5

In the proposed method (e-M-PIN), the client sends
a signature first, without receiving the server’s challenge. If the client issues the signature without thechallenge, an adversary can initiate a replay attack
because the signature becomes a deterministic value.
To solve this problem, we use a nonce and the current
time.
The nonce works as the server’s challenge. The client uses a nonce when he issues a one-time signature(V), and the server checks that the same nonce wasnever used before. Then, the adversary cannot implement a replay attack.
However, in this protocol, the server has to keep allnonce log data in his DB, which can place a big load
NTT Technical Review

Regular Articles

IDa(email address)
PIN (α)Fig. 7. Login page image of e-M-PIN.

on the server. Therefore, the server sets an expiration
time (t) for a one-time signature. The server can thenreject an old one-time signature which is |CCTSCT|<t. With the setting of an expiration time, theserver does not have to keep large nonce data for longer than the expiration time. These steps of using a
nonce and the current time mean that e-M-PIN canachieve the same level of safety as M-PIN.
The client login page in the authentication process
is shown in Fig. 7. An ID and PIN are input on thelogin page. This design is similar to that of the password login page.
4.1 Advantages of e-M-PIN
The advantages of e-M-PIN over M-PIN are as follows; some points have been improved, and there is
no reduction in security.
e-M-PIN has almost the same level of security as
the previous one. As same as M-PIN, the user splits
his secret key into PINα and Token (s–α)A. So, theadversary cannot attack with one factor. Also, theserver does not have to store a user’s hashed password
in the server’s storage.
In the e-M-PIN, the communication between clientand server is only one time. The client just sends his
one-time signature (V) which includes nonce and ismade within expiration time (t). So, e-M-PIN canmigrate easily from an existing one-pass authentication protocol such as password-based HTTP authentication.

Vol. 14 No. 12 Dec. 2016

5. Conclusion
We have developed e-M-PIN, an authentication
protocol that improves upon M-PIN. e-M-PIN is a
non-interactive protocol, which means it is compatible with the traditional password-based HTTP
authentication protocol, while the security is equivalent to that of M-PIN. Thus, e-M-PIN is easier tomigrate to than M-PIN. e-M-PIN has already been
published as open source software in the Apache
Milagro (incubating) repository. We presented e-MPIN at ApacheCon North America held in May 2016
in Vancouver [8]. The demonstration of e-M-PIN
showing the ease of migration to it received a positive
reaction from visitors. We will continue working to
improve Milagro-MFA in order to expand its use in
the future.
References
[1]
[2]
[3]
[4]
[5]
[6]

Apache Milagro, Milagro - The Apache Software Foundation!,
http://milagro.apache.org/
Press release issued by NTT on May 11, 2016.
http://www.ntt.co.jp/news2016/1605e/160511b.html
M. Scott, “M-PIN: A Multi-Factor Zero Knowledge Authentication
Protocol,” https://cdn2.hubspot.net/hub/230906/file-63954152-pdf/
downloads/certivox_labs_mpin.pdf
NIST, “Recommended Elliptic Curves for Federal Government Use,”1999.
http://csrc.nist.gov/groups/ST/toolkit/documents/dss/NISTReCur.pdf
SECG, “SEC 2: Recommended Elliptic Curve Domain Parameters,”
2010.
http://www.secg.org/sec2-v2.pdf
ECC Brainpool (RFC 5639), “ECC Brainpool Standard Curves andCurve Generation,” 2005.
http://www.ecc-brainpool.org/download/Domain-parameters.pdf

6Regular Articles[7]

The Apache Software Foundation, Apache Milagro (incubating)
Repositories, https://github.com/apache?query=MILAGRO

Masahiro Matsui

ApacheCon North America, http://events.linuxfoundation.org/events/
apachecon-north-america

Hironobu Okuyama

Engineer, Data Security Project, NTT Secure
Platform Laboratories.
He received a B.A. in human sciences and an
M.S. in global information and telecommunication studies from Waseda University, Tokyo, in2013 and 2015. He is presently researching information security.

Senior Research Engineer, Data Security Project, NTT Secure Platform Laboratories.
He received a B.S. from Tohoku University,
Miyagi, in 1990 and an M.S. from Chiba University in 1992. He is presently engaged in researchon information security.

Hiroaki Ohtsuka

Akira Nagai

Research Engineer, Data Security Project,
NTT Secure Platform Laboratories.
He received a B.Eng. and M.Eng. from Hokkaido University in 1994 and 1996. He is presently engaged in research on information security.

Researcher, Data Security Project, NTT Secure
Platform Laboratories.
He received a B.Eng. and M.Eng. from Tokyo
University of Science in 2006 and 2008. He ispresently researching information security.

Tetsutaro Kobayashi

Go Yamamoto

Senior Research Engineer, Data Security Project, NTT Secure Platform Laboratories.
He received a B.Eng. and M.Eng. from Tokyo
Institute of Technology in 1993 and 1995 and a
Ph.D. from the University of Tokyo in 2005. He
is currently conducting research on information
security. He was awarded the SCIS (Symposiumon Cryptography and Information Security) 2000
paper prize.

7[8]

Technology Lead and Associate Director, NTT
Innovation Institute, Inc.
He received a B.S., M.S., and Ph.D. in mathematical sciences from the University of Tokyo in
1996, 1998, and 2001. He is currently researching and developing products involving cryptography and distributed systems. He was awarded
the SCIS 2007 paper prize.

NTT Technical Review

Regular Articles

Path Loss Models for WirelessAccess Network Systems Using
High Frequency Bands
Motoharu Sasaki, Minoru Inomata, Wataru Yamada,and Takeshi Onizawa
Abstract

To design coverage areas for wireless communication systems and evaluate the performance of thesystems, it is necessary to develop a path loss model that can be used for the systems’ frequency bands.
The fifth-generation mobile communication system (5G) is expected to use high frequency bands above
6 GHz. Therefore, developing a path loss model for wireless access communication systems using highfrequency bands has become a pressing issue. NTT Access Network Service Systems Laboratories hasdeveloped path loss models for the 20–40 GHz bands that are suitable for use scenarios of mobile phones
and wireless local area networks. This article introduces the developed models.
Keywords: propagation, 5G, high frequency band

1. Introduction
In developing wireless communication systems, itis necessary to design their coverage areas and evaluate their performance such as the system throughput,
the transmission capacity, and the system capacity.
The system performance and the size of the area
depend on the degree of propagation attenuation(path loss), as seen in Fig. 1. The propagation characteristics, including path loss, vary greatly dependingon the frequency band. Therefore, it is necessary todevelop a path loss model that can be used for thefrequency band of the system.
Although it is assumed that the fifth-generation
mobile communication system (5G) will use highfrequency bands above 6 GHz, it is known that the
propagation characteristics in high frequency bands
differ from those for frequency bands below 6 GHz,
which have long been used for conventional mobile
communication systems. For example, the path loss
increases as the frequency increases, and radio waves
in high frequency bands do not reach shielded areas
such as the areas behind buildings.
1

The frequency allocation of the high frequency
bands for 5G will be discussed at the World Radiocommunication Conference in 2019. The propagation
characteristics of the high frequency bands will be thebasis of the discussion and should thus be studied inorder to discuss appropriate allocation. Therefore,
modeling the propagation characteristics of the highfrequency bands has become a pressing issue.
At NTT Access Network Service Systems Laboratories, we are carrying out propagation modeling of awide range of frequency bands in order to develop avariety of wireless communication systems. In thisarticle, we introduce path loss models for the 20–40
GHz bands that can be useful for wireless communication systems in high frequency bands.
2. Path loss models
We clarified a propagation mechanism by measuring and simulating propagation, focusing on urban
and residential environments. These environments
are important use case scenarios of future wirelesscommunication systems. They include mobile
NTT Technical Review

Regular Articles

Base station

User terminal

Coverage area

Path loss caused by buildings and terrain affects
coverage areas and system performance.

Fig. 1. Path loss model for designing wireless communication systems.

Path loss model in urban environments [1]Path loss model in residential environments [2]The path loss model accurately predicts path loss byclassifying streets into LoS and NLoS areas.

The path loss model accurately predicts path loss
by considering dominant propagation paths.
Over rooftops (Lv)
Rx

Rx
NLoS
area (Latt)Tx
LoS area (LLoS)

Corner area (LC)Tx

Between buildings (Lb)

L = LLoS + LC + Latt

L = −10 log (1/Lr + 1/Lb + 1/Lv)(a) Urban environments(b) Residential environments

LoS: line-of-sight
NLoS: Non-line-of-sight
Rx: receiver
Tx: transmitter

Along roads (Lr)*Including several formulas for prediction

Fig. 2. Path loss models developed for urban and residential environments.

communication systems and wireless local area network systems. We obtained dominant paths frominvisible propagation phenomena. The results
enabled us to develop path loss models for high frequency bands that had been unclear except for thebands below 6 GHz. These models are depicted inFig. 2.
Vol. 14 No. 12 Dec. 2016For urban environments, we developed a model byderiving dominant path loss characteristics for threeareas in which waves sent from a transmitter (Tx) toa receiver (Rx) are either visible or invisible: a lineof-sight (LoS) area, a non-line-of-sight (NLoS) area,
and a corner (i.e., an intersection) area that is a transition area between LoS and NLoS areas [1]. The
2

Regular Articles

8060
Meas. (26 GHz)
Meas. (37 GHz)
Pred. (26 GHz)
Pred. (37 GHz)

100
110
120
130
140
150
160

A

B

C

100

200 300 400 500
Traveling distance (m)

100 m

Tx

600

E

100
120
140

Meas. (26 GHz)
Pred. (26 GHz)

160
0

D

80
Path loss (dB)Path loss (dB)

90

700

0

Accurate prediction:less than 5 dB of RMSE

A

Running
direction
Around Tokyo Station(a) Urban environment

200

400
600
800
Traveling distance (m)

1000

100 m

Running
direction

Tx
B

E
C

Tsukuba City

D(b) Residential environment

RMSE: root mean square error

Fig. 3. Results of predicting and measuring path loss in urban and residential areas.

respective path losses in these areas are indicated as
LLoS, Latt, and LC, as seen in Fig. 2(a). The radio waves
transmitted from the Tx propagate to the Rx throughthese areas, with the three path losses added. We
modeled the total path loss by using the frequency,
the distance along a road between the Tx and the Rx,
and the distance along a road between the Tx and thecorner as parameters.
For residential environments, we developed amodel by deriving three dominant paths: a path alonga road, a path between buildings, and a path above arooftop. The respective losses for these paths are indicated as Lr, Lb, and Lv, as seen in Fig. 2(b) [2]. We
modeled these path losses using parameters including
the distance along a road between the Tx and the Rx,
building density, and building height. In our study,
the equation for the residential environment model
differs from that for the urban environment model
because the received power at the Rx in a residential
environment depends on which of the three paths has
the least path loss.
In this way, we were able to develop models fromthe dominant paths that can easily predict actual pathloss with high accuracy using simple parameters.
These models have the potential to play a major role3

on the world stage in studies on the propagation characteristics of high frequency bands.
3. Prediction results obtained with developed
path loss models
The results of predicting path loss using the models
are plotted in Fig. 3. We verified the models’ validity
by comparing the prediction results and the measurement results obtained using 26 GHz and 37 GHz
frequencies in an urban and residential environments.
In the urban environment, as shown in Fig. 3(a), theRx moves along the road about 180 m and then turns
the corner. The path loss increases sharply at the corner because the Rx moves from a LoS area to a NLoS
area.
In the residential environment on the other hand, as
shown in Fig. 3(b), the path loss shows significant
variation when the Rx moves from area A to area E.
This is caused by situational variations such as thenumber of corners between the Tx and the Rx and thedegree of building shielding.
In both environments, the developed models accurately predicted the actual path losses. The root meansquare error (RMSE) values are less than 5 dB. TheseNTT Technical Review

Regular Articles

results indicate that the models provide sufficient
prediction accuracy for designing future wirelesscommunication systems.
4. Future overview
This article presented path loss models that canaccurately predict actual path losses using simple
parameters. The derived models are suitable fordesigning coverage areas and evaluating the performance of various future wireless communication
systems using high frequency bands. The models are
expected to be utilized for the 5G system, which has
been the focus of much research and development in

Motoharu Sasaki

the wireless communication field worldwide.
To contribute to the construction of various wirelesscommunication systems, we intend to actively engage
in the modeling of propagation phenomena for awider range of frequencies and environments.
References
[1]

[2]

M. Inomata, W. Yamada, M. Sasaki, M. Mizoguchi, K. Kitao, and T.
Imai, “Path Loss Model for the 2 to 37 GHz Band in Street Microcell
Environments,” IEICE Commun. Express, Vol. 4, No. 5, pp. 149–154,
2015.
M. Sasaki, W. Yamada, N. Kita, and T. Sugiyama, “Path Loss Modelwith Low Antenna Height for Microwave Bands in Residential
Areas,” IEICE Trans. Commun., Vol. E96-B, No. 7, pp. 1930–1944,
2013.

Wataru Yamada

Research Engineer, NTT Access Network Service Systems Laboratories.
He received a B.E. in engineering and an M.E.
and Ph.D. in information science and electricalengineering from Kyushu University, Fukuoka,
in 2007, 2009, and 2015. He joined NTT AccessNetwork Service Systems Laboratories in 2009,
where he has been conducting research on propagation modeling of interference between mobile
terminals for spectrum-sharing wireless accesssystems. He received the Young Researcher’sAward and the Best Paper Award from the Institute of Electronics, Information and Communication Engineers (IEICE) in 2013 and 2014,
respectively. He is a member of the Institute ofElectrical and Electronics Engineers (IEEE).

Senior Research Engineer, Planning Section,
NTT Access Network Service Systems Laboratories.
He received a B.E., M.E., and Ph.D. from Hokkaido University in 2000, 2002, and 2010. Sincejoining NTT in 2002, he has been researchingpropagation characteristics for wide band accesssystems. From 2013 to 2014, he was a visiting
research associate at the Centre for Telecommunications Research at King’s College, London,
UK. He received the Young Researcher’s Award,
the Communications Society Best Paper Award,
and the Best Paper Award from IEICE in 2006,
2011, and 2014, respectively. He is a member ofIEEE.

Minoru Inomata

Senior Research Engineer, Supervisor, Group
Leader of Wireless Access Systems Project, NTT
Access Network Service Systems Laboratories.
He received a B.E., M.E., and Ph.D. from
Saitama University in 1993, 1995, and 2003.
Since joining NTT in 1995, he has been engaged
in the research and development of personalcommunication systems and high data rate wireless LAN systems. He is now in charge of strategies and propagation of wireless communication
systems. He received the Best Paper Award, theYoung Researcher’s Award, and the Achievement
Award from IEICE in 2000, 2002, and 2006,
respectively. He also received the Maejima
Hisoka Award from the Tsushinbunka Association in 2008. He is a senior member of IEICE anda member of IEEE.

Research Engineer, NTT Access Network Service Systems Laboratories.
He received his B.E. and M.E. in electricalengineering from Tokyo University of Science in2009 and 2011. He joined NTT Wireless Systems
Innovation Laboratories in 2011, where he has
been researching radio propagation for the 5G
system and system design for unlicensed wireless communication systems. He received theYoung Researcher’s Award from IEICE in 2016.
He is a member of IEEE.

Vol. 14 No. 12 Dec. 2016

Takeshi Onizawa

4

Global Standardization Activities

Report on Apache Big Data North
America 2016 and Spark Summit
2016
Takeshi Yamamuro and Tsuyoshi Ozawa
Abstract

Two key conferences were held this year on open source projects concerning big data processing systems: Apache Big Data North America 2016 and Spark Summit 2016. This article reports on some of the
topics discussed at the conferences.
Keywords: distributed processing, Apache Hadoop, Apache Spark

1. Apache Big Data North America
Apache Big Data North America is one of the largest conferences related to open source projects on big
data processing and is supported by the Apache Software Foundation. The conference features interestingpresentations given by users and developers on various big data processing systems using Apache open
source software (OSS) products such as Hadoop [1],Spark [2], Kafka [3], and Cassandra [4]. This is a key
conference for OSS developers and typically has
higher numbers of developers than other events.
Lively discussions were held at the conference that
continued even during the coffee breaks.
1.1 Conference summary
Apache Big Data North America 2016 [5] was heldin Vancouver, Canada from May 9 to 12. Many people attended the conference from hardware vendors tocontent providers, including representatives fromIntel Corporation, Netflix, Inc., eBay Inc., Yahoo
Japan Corporation, and Recruit Holdings Co., Ltd.
These companies are active users of OSS products.
1.2 Business use cases
The notable keywords appearing in the titles of thepresentations at the conference were Spark, Hadoop,
Kafka, and Cassandra. Of the total presentations, 55of them, or more than 40%, were related to these1

keywords.
In a keynote speech entitled, “How Netflix Leverages Big Data,” Brian Sullivan, Director of Streaming
Analytics at Netflix, talked about how open source
products are being used in Netflix’s services. Netflix
is a worldwide video streaming service used by 81
million subscribers. A huge amount of video content—more than 125 million hours—is uploaded anddistributed per day. Surprisingly, Netflix’s communication traffic occupies more than one-third of all theInternet traffic in North America. Netflix creates andimproves its services by using big data. By analyzingover 3 petabytes of data each day, Netflix can create,deliver, and provide excellent content. They utilize
OSS developed in community supported by the
Apache Software Foundation such as Hadoop, Spark,
Kafka, and Cassandra.
2. Spark Summit
The Spark Summit conference was first held in
2013. Since 2015, it has been held three times a year
on the east and west coasts of North America and in
the EU. Developers present the latest features of
Spark, an open source cluster computing framework,and users share Spark use cases and know-how onusing it.
Spark was developed by the AMPLab team at the
University of California, Berkeley. Developers at
NTT Technical Review

Global Standardization Activities

Databricks are now taking the lead in this community.
In addition to the underlying mechanisms necessary
for distributed processing, Spark includes easy-to-use
libraries for SQL queries, machine learning, streaming, and graph data processing. A report on a 2015questionnaire survey [6] on Spark indicated that the
Spark community continues to grow year by year andthat Spark is one of the most popular OSS products
worldwide. The development community is highly
active, and version 2.0 was officially released at theend of July 2016.
2.1 Conference summary
The 7th Spark Summit 2016 [7] was held from June
6 to 8 in San Francisco. It was supported by many
companies including IBM Corporation, Microsoft
Corporation, Intel, and EMC Corporation. The conference was attended by more than 2500 people fromover 720 companies.
2.2 Development community trends
At the conference, the main developers talked about
the upcoming features in version 2.0, which is characterized by continuousness and structuralization.
The new features provide effective processing ofstream data such as sensor log data. Spark version 1.6
has a feature called DataFrame/Dataset for processing static data with schema. Version 2.0 has a newfeature for stream data called Structured Streaming.
This Structured Streaming feature is intended forusers who need to process log data in near-real time.
Another session at the conference focused on memory and query management. All indications are that
Spark has great potential to grow because of theincreasing number of developers and users.
2.3 Business use cases
Spark includes numerous libraries necessary for
analytical processing in various use cases. In particular, many use cases concerned Spark Streaming. Thisfunction was implemented in version 0.7. SparkStreaming is similar to Structured Streaming, which
is currently under development, as it deals withdynamic data. Spark Streaming has certain limitations, however, so it is expected to be replaced overthe long term by the sophisticated functions of Structured Streaming.
Presentations were also given at the conference on
the analysis of user logs generated daily for Microsoft’s search engine Bing and on the behavior analysis of the popular Airbnb, a rental service for lodgingsand guest houses. The common point of these twoVol. 14 No. 12 Dec. 2016

cases is that they use Kafka, a distributed message
processing infrastructure. Kafka is used to store generated data, and Spark fetches the data from Kafka.
Because Kafka and Spark both use the programming
language Scala, they mesh well. Consequently, Kafka
is becoming the de-facto standard to input into SparkStreaming.
In the case of the Weather Company, a weather
forecasting company purchased by IBM in January
2016, petabytes of data are generated every day, andSpark is used for data processing. Thus, in recent
years, the number of large-scale examples has gradually been increasing, so it is expected that the scale ofapplications will continue to grow even more.
3. Activities of NTT Software Innovation Center
NTT Software Innovation Center performs researchand development on big data processing infrastructures, which is the common theme between the two
conferences reported in this article. Therefore, wewould like to introduce one of our efforts related tothese conferences.
We gave a presentation at Apache Big Data North
America 2016 on Hadoop’s function to manage computer resource allocation, which we developed withother OSS developers [8]. At the conference, we discussed various issues with OSS developers. For
example, we discussed how to use various devices on
the Hadoop framework with researchers from the
University of Toronto. Devices such as a GPGPU*1
and FPGA*2 are the most important accelerators forprocessing big data. Hadoop’s current resource manager does not control these accelerators because it is
not clear how they should be allocated to various
tasks.
However, two main bottlenecks can occur with theaccelerators. One is a CPU*3 intensive bottleneck thatcan occur with, for example, scientific computing andmachine learning. The other is a network I/O*4 intensive bottleneck, which can occur when extracting a
specific pattern from a large amount of data. A separate solution must be found for these bottlenecks.
Unfortunately, the problem is that both bottlenecks
can occur in one use case. This means we must solve
the problems under these constraints, which involves
a trade-off.
*1
*2
*3
*4

GPGPU: general-purpose graphics processing unit
FPGA: field-programmable gate array
CPU: central processing unit
I/O: input/output

2

Global Standardization Activities

It is difficult to inject accelerator management into
the Hadoop framework in terms of interchangeability.
However, as the use cases of Hadoop are extended, it
will be necessary to solve the problem of the utilization of accelerators in Hadoop.
References[1]

Apache Hadoop, http://hadoop.apache.org/

Takeshi Yamamuro

Senior Researcher, Distributed Computing
Technology Project, NTT Software Innovation
Center.
He received an M.E. from Faculty of Scienceand Technology of Sophia University, Tokyo, in
2008. He joined NTT in 2008 and has been working on database management systems. His
research interests include compression andhardware-aware algorithms (e.g., SIMD,NUMA, and GPU). He received the IPSJ
Yamashita SIG Research Award from the Information Processing Society of Japan (IPSJ) in
2015. He is a member of IPSJ and the Database
Society of Japan (DBSJ).

3

[2]
[3]
[4]
[5]
[6]
[7]
[8]

Apache Spark, http://spark.apache.org/
Apache Kafka, http://kafka.apache.org/
Apache Cassandra, http://cassandra.apache.org/
Apache Big Data North America, http://events.linuxfoundation.org/
events/apache-big-data-north-america
Spark Survey 2015 Results, https://databricks.com/blog/2015/09/24/
spark-survey-2015-results-are-now-available.html
Spark Summit 2016, https://spark-summit.org/2016/
Apache Big Data 2016, https://apachebigdata2016.sched.org/
event/6M0N/yarn-a-resource-manager-for-analytic-platform-tsuyoshi-ozawa-ntt

Tsuyoshi Ozawa

Research Engineer, Distributed Computing
Technology Project, NTT Software Innovation
Center.
He received a B.E. in information and systemengineering from Chuo University, Tokyo, in2008 and an M.E. in computer science from the
University of Tsukuba, Ibaraki, in 2010. He
joined NTT in 2010 and is working on distributed
processing frameworks such as Hadoop. His
research interests include distributed computing
and distributed databases. He received the Computer Science Research Award for Young Scientists from IPSJ in 2013 and the 9th Japan OSS
Incentive Award from the Japan OSS Promotion
Forum in 2014. He has served as a committer
since 2014 and as a member of the project management committee since 2015 in the Apache
Hadoop community. He is a member of the Association for Computing Machinery (ACM), IPSJ,
and DBSJ.

NTT Technical Review

Short Reports

Deployment of NTT’s Open Source
Software GoBGP in
INTERNET MULTIFEED’s JPNAP
Service Achieves Automation of
Operation and Vast Improvement in
Efficiency

1. Introduction
NTT and INTERNET MULTIFEED CO. have partnered to apply GoBGP [1], an open source Internet
routing control software provided by NTT, to JPNAP,an Internet exchange (IX) point provided by INTERNET MULTIFEED. They have achieved commercial
deployment. GoBGP enables INTERNET MULTIFEED to automate the operation of JPNAP’s RouteFEED service* and reduce the lead time under the
new RouteFEED service contract to 1/10th the timerequired previously. The time needed to process customers’ existing configuration change orders is
reduced to 1/30th of that previously. Automating theoperation means that problems due to human errors inprevious manual configuration changes can be prevented. Operational loads are also reduced to 1/10thof their previous amounts. JPNAP is the world’s first
IX point to introduce the commercial deployment of
GoBGP for IX providers.
2. Background
IX services are becoming more and more importantfor stable operation of the Internet. An IX is an Internet exchange connection point between Internet service providers (ISPs) and content providers. Morethan 600 IX points are in operation around the world.
JPNAP is one of the largest IX points in Asia in termsof exchange traffic volume. As the amount of Internet
traffic and the number of routes continue to increase1annually, operational efficiency and costs are becoming serious issues.
NTT has engaged in open source software (OSS)
projects such as the development of the Ryu SDN(software-defined networking) Framework, which
was released as open source in 2012, and OpenStack.
Through this participation, NTT has gained technological expertise in network software and OSS development know-how. INTERNET MULTIFEED operates one of the largest IX points in Asia, and has
gained advanced knowledge in Internet routing control technologies and operational experience. By
working together closely to advance OSS development, NTT and INTERNET MULTIFEED have succeeded in achieving the deployment described above.
3. Features of the technology
GoBGP efficiently leverages the features of modern-day hardware architecture such as a multicore
CPU (central processing unit), enabling it to achieve
the scalability necessary for processing the current
number of Internet routes and connection points
required by IX services (Fig. 1).
GoBGP uses a design premised on software control
instead of conventional manual operation. GoBGP
provides application programming interfaces (APIs)*

RouteFEED service: Service provided for JPNAP users that automatically carries out routing exchanges with numerous connection points.

NTT Technical Review

Short Reports

Operational system

Operational system

Automated operation through APIs

Manual operation
ISP
A

Content provider
C

ISP
A

Content provider
CAPI
Old route server

Route server(GoBGP)

JPNAP service

JPNAP service

ISP
B

Content provider
D

ISP
B

Old system

New system

Fig. 1.

Overview of RouteFEED Architecture.

and software for configuration changes and otherpurposes, resulting in a design that can rapidly handle
frequent API requests and that makes it easy to automate operational processes. Connections to processes
such as data analysis and event notification to external systems can also be easily realized by utilizingthese APIs.
4. Future plans
Going forward, NTT is seeking to not only expand
the GoBGP open source community and promote thespread of the technology, but also to apply GoBGP touse cases beyond IX services such as datacenter networks and commodity network hardware. It is also
seeking to further accelerate the expansion of GoBGP’s functions and performance. Furthermore, NTT
is seeking to expand the business market with OSS

Vol. 14 No. 12 Dec. 2016

Content providerDand energize the business and technological development of software networking infrastructure. INTERNET MULTIFED is engaged in providing pioneering
IX services that contribute to improving the reliability of IX points as social infrastructure. It seeks toincrease its operational knowledge of GoBGP’s
RouteFEED service and provide feedback to other IX
providers in order to continue to expand the Internet
quickly and soundly.
Reference[1]

GoBGP, http://osrg.github.io/gobgp/

For Inquiries
PR Department, Planning Division
NTT Service Innovation Laboratory Group
Email: randd@lab.ntt.co.jp

2

New NTT Colleagues––We welcome our newcomers to the NTT Group
This is a corner of the NTT Technical Review where we introduce our new affiliate companies.

Nefos
Salesforce consulting partner; established in 2007; headquartered in Zurich, Switzerland
Headquartered in Zurich, Nefos AG specializes in delivering tailor-made Salesforce solutions to Germanspeaking markets via its offices in Germany, Austria, and Switzerland, as well as through its near-shore clouddevelopment center in Croatia. Since it was established in 2007, Nefos has implemented over 500 Salesforce
projects across the manufacturing, automotive, telecommunications, and service sectors.
In September 2016, NTT DATA EMEA Ltd, a London-based NTT DATA subsidiary, and Nefos announced
that they had entered into a definitive agreement for NTT DATA to acquire Nefos, one of the leading Salesforce
consulting partners in Germany, Austria, and Switzerland.
The agreement strengthens NTT DATA’s world-class Salesforce consulting and implementation practice andreflects the company’s continued focus on expanding its leadership position in customer management/customer
experience solutions, key elements of both digital business and digital transformation solutions. For further
information about Nefos, please visit: http://www.nefos.com/?lang=en
Contact:
Public Relations Department
NTT DATA Corporation
Tel: +81-3-5546-8051
http://www.nttdata.com/global/en/news-center/pressrelease/2016/092800.html

1

NTT Technical Review

External Awards
The Honorable Mention Paper
Winner: Hidetomo Sakaino, NTT Network Technology Laboratories
Date: June 1, 2016
Organization: The Institute of Electrical and Electronics Engineers(IEEE) Intersociety Conference on Thermal and ThermomechanicalPhenomena in Electronic Systems (ITherm) 2016 committee
For “A Dynamic Stochastic Optimization Control Model for Data
Centers Based on Numerical Modeling.”
Published as: H. Sakaino, “A Dynamic Stochastic Optimization
Control Model for Data Centers Based on Numerical Modeling,”
Proc. of ITherm 2016, pp. 675–684, Las Vegas, NV, USA, May/June
2016.
Young Scientist Award
Winner: Hiroshi Hamada, Toshihiko Kosugi, Ho-Jin Song, Hideaki
Matsuzaki, Amin El Moutaouakil, Hiroki Sugiyama, Makoto Yaita,
Takuro Tajima, Hideyuki Nosaka, and Osamu Kagami, NTT Device
Technology Laboratories; Yoichi Kawano, Tsuyoshi Takahashi, Yasu­
hiro Nakasha, and Naoki Hara, Fujitsu Limited; Katsumi Fujii, Issei
Watanabe, and Akifumi Kasamatsu, National Institute of Information
and Communications Technology (NICT)
Date: August 24, 2016
Organization: International Union of Radio Science (URSI)For “20-Gbit/s ASK Wireless System in 300-GHz-band and Frontends with InP MMICs.”
Published as: H. Hamada, T. Kosugi, H. Song, H. Matsuzaki, A.
Moutaouakil, H. Sugiyama, M. Yaita, T. Tajima, H. Nosaka, O.
Kagami, Y. Kawano, T. Takahashi, Y. Nakasha, N. Hara, K. Fujii, I.
Watanabe, and A. Kasamatsu, “20-Gbit/s ASK Wireless System in
300-GHz-band and Front-ends with InP MMICs,” Proc. of URSI
Asia-Pacific Radio Science Conference, Seoul, Korea, Aug. 2016.
JSAP Fellow
Winner: Tetsuomi Sogawa, NTT Basic Research Laboratories
Date: September 13, 2016
Organization: The Japan Society of Applied Physics (JSAP)
For his research on control of photonic and spin-related properties
in semiconductor quantum structures by surface acoustic waves.
JSAP Young Scientist Award
Winner: Pierre-Alix Carles, NTT Basic Research Laboratories
Date: September 13, 2016
Organization: JSAP
For “Deviation from the Law of Energy Equipartition in a Small
Dynamic-random-access Memory.”
Published as: P. Carles, K. Nishiguchi, and A. Fujiwara, “Deviationfrom the Law of Energy Equipartition in a Small Dynamic-randomaccess Memory,” Jpn. J. Appl. Phys., Vol. 54, 06FG03, 2015.
Paper Award
Winner: Takeshi Mishima and Yasuhiro Fujiwara, NTT Software
Innovation Center
Date: September 14, 2016
Organization: WebDB Forum 2016 committee

1

For “Database Live Migration Middleware in Cloud Environment.”
Published as: T. Mishima and Y. Fujiwara, “Database Live Migration Middleware in Cloud Environment,” IPSJ TOD, Vol. 9, No. 1,
2016.
CollabTech Best Paper Award
Winner: Xun Cao, Kyoto University; Naomi Yamashita, NTT Communication Science Laboratories; Toru Ishida, Kyoto University
Date: September 16, 2016
Organization: Information Processing Society of Japan (IPSJ) Special Interest Groups on Groupware and Network Services
For “How Non-native Speakers Perceive Real-time Listening
Comprehension Problems: Implications for Adaptive Support Technologies.”
Published as: X. Cao, N. Yamashita, and T. Ishida, “How Non-native
Speakers Perceive Real-time Listening Comprehension Problems:
Implications for Adaptive Support Technologies,” Proc. of Collab­
Tech 2016 (the Eighth International Conference on Collaboration
Technologies), Kanazawa, Ishikawa, Japan, Sept. 2016.
Duke’s Choice Award (Tool Award)
Winner: Yuji Kubota and Shinji Takao, NTT Software Innovation
Center; Yasumasa Suenaga, NTT Comware
Date: September 20, 2016
Organization: Java Community/Oracle
For development of HeapStats.
HeapStats is a lightweight Java troubleshooting analysis tool thatuses JVMTI (Java Virtual Machine Tool Interface) and SIMD (single
instruction multiple data) instructions in x86 and ARM for optimization.
Communications Society: Distinguished Contributions
Award
Winner: Kazuhide Nakajima, NTT Access Network Service Systems Laboratories
Date: September 21, 2016
Organization: The Institute of Electronics, Information and Communication Engineers (IEICE) Communications Society
For his contribution as a manager of the IEICE Ad Hoc TechnicalCommittee on Extremely Advanced Optical Transmission Technologies (EXAT).
Communications Society: Distinguished Contributions
Award
Winner: Jun Mashino, NTT Network Innovation Laboratories
Date: September 21, 2016
Organization: IEICE Communications Society
For his contribution as a secretary assistant of the IEICE TechnicalCommittee on Radio Communication Systems (RCS).
Communications Society: Distinguished Contributions
Award
Winner: Tadao Nakagawa, NTT Network Innovation Laboratories
Date: September 21, 2016
Organization: IEICE Communications Society

NTT Technical Review

Papers Published in Technical Journals and Conference Proceedings

For his contribution to the management and invigoration of theexecutive committee of the IEICE Communications Society.

Organization: All Japan Radio & Television Commercial Confederation (ACC)

Communications Society: Distinguished Contributions
Award
Winner: Takeshi Kinoshita, NTT Network Innovation Laboratories
Date: September 21, 2016
Organization: IEICE Communications Society

For “Cho Kabuki Supported by NTT.”
Highly realistic and new forms of kabuki viewing experience areprovided using the immersive telepresence technology “Kirari!”.

For his contribution to the operation of international symposia
hosted by the IEICE Technical Committee on Network Virtualization.
Technical Director Award (Interactive Category) of the2016 56th ACC CM Festival
Winner: Hidenobu Nagata, NTT Service Evolution Laboratories
Date: September 28, 2016

APSIPA Industrial Distinguished Leader
Winner: Takehiro Moriya, NTT Communication Science Laboratories
Date: October 7, 2016
Organization: Asia-Pacific Signal and Information Processing
Association (APSIPA)
For his extraordinary accomplishments in the field related to
APSIPA scope.

Papers Published in Technical Journals andConference Proceedings
Lightness and Brightness Match under Colored Illuminants with Special Illuminant Gradients
H. Kawamura, M. Date, T. Yamakawa, T. Kojima, M. Miyao, andA. Kojima
Proc. of AIC (International Colour Association) 2016 Interim
Meeting, pp. 417–420, Santiago, Chile, October 2016.
This paper reports an investigation of perceived colors under colored illuminants with spatial illuminant gradients. In performedexperiments, two types of illuminant distribution were used; in one
the illuminant strength changes suddenly at the center of an image
and in the other the strength changes linearly throughout the right andleft side of the image. Subjects were asked to match the test patcharea to identify the same luminance as the reference one (called thebrightness match) or to match the area to be cut from the same pieceof paper as the reference one (called the lightness match) under white
illuminant or reddish illuminant. The results showed that the responseof the subjects in the brightness match is affected by the luminanceof the test patch area; however, with the lightness match the response
is slightly more stable than the brightness match. This tendency

Vol. 14 No. 12 Dec. 2016

occurred in both illuminant distribution patterns. When using a reddish illuminant, the response of the subjects was lower than that for
the white illuminant.

Collapse of the Hierarchy of Constant-depth Exact Quantum Circuits
Y. Takahashi and S. Tani
Computational Complexity, Vol. 25, No. 4, pp. 849–881, December 2016.
We show that there exists a constant-depth polynomial-size quantum circuit for the OR function. We apply this circuit to constructing
a constant-depth quantum circuit for the threshold function. Moreover, we show that there exists a classically hard problem that can besolved by a constant-depth quantum circuit with gates for the quantum Fourier transform.

2