NTT Technical Review

December 2018 Vol. 16 No. 12
View from the Top

Mitsuyoshi Kobayashi, President and Chief Executive Officer, NTT WEST

Feature Articles: Ultra-realistic Communication with Kirari! Technology
Kirari! Ultra-realistic Communication Technology: Beyond 2020Real-time Extraction of Objects from Any Background Using Machine Learning“Kirari! for Arena”—Highly Realistic Public Viewing from Multiple Directions



360-degree Tabletop Glassless 3D Screen SystemVideo Processing/Display Technology for Reconstructing the Playing Field in Sports Viewing
Service Using VR/AR

Global Standardization Activities

Forum Survey Activities by TTC (The Telecommunication Technology Committee)
Technology Research Advisory Group

Practical Field Information about Telecommunication Technologies


Initiatives Targeting Snow Damage to Communication Facilities

External Awards/Papers Published in Technical Journals and Conference Proceedings

View from the Top

Invigorating Regional Economies andSolving Social Problems with “Local
Revitalization Clouds”—Building the
Present with a View to the Future
through “Moonshot” Thinking
Mitsuyoshi Kobayashi
President and Chief Executive Officer,
NTT WEST
Overview

Aiming to be a company that contributes to solving social problems, NTT
WEST is transforming itself into a pioneer in the era of Internet of Things
and artificial intelligence by leveraging the technologies, know-how, andservices it has cultivated to date. We asked Mitsuyoshi Kobayashi, NTT
WEST President and Chief Executive Officer, to tell us about his plans andspecific initiatives in support of the new NTT WEST Medium-term Management Plan that is aimed at creating new value to realize a prosperous,
vibrant, and safe and secure society.
Keywords: regional revitalization, ICT, cloud

Daring to create a synergistic effect among 800
local governments and 2 million companies—Mr. Kobayashi, please tell us about the current
business environment surrounding NTT WEST.
July 2019 will mark the 20-year anniversary of thefounding of NTT WEST. At the time of founding,operating conditions were such that we were operating at a loss of about 200 billion yen. However, in
fiscal year (FY) 2016, operating profits reached 95.1
billion yen, a record high in the history of NTT
WEST, and after making incremental improvements
such as further rationalization of operations, they
climbed to 167.4 billion yen in FY2017, making two
NTT Technical Review

Vol. 16 No. 12 Dec. 2018consecutive years of record profits. I attribute these
positive results to nothing other than the tirelessefforts of our employees.
On the other hand, if we look only at revenues, we
can see a 20-year drop, and this is continuing forvarious reasons, for example, the severely competitive environment and a growing shift toward mobile
communications. Specifically, operating revenues
were 2700 billion yen 20 years ago but not much
more than 1400 billion yen in recent years, an overall
drop of about one-half. No matter how much ouroperating profits rise, a company whose revenues
continue to drop can produce some anxiety about the
future in its employees. My mission is to resolve thissituation, which I think is the earnest desire of all1

View from the Top

NTT WEST employees. I would like to get on asound path toward revenue growth as soon as possible.
—What kind of strategies do you have in mind?
To begin with, what kind of image has NTT WEST
as a company been projecting to society? It would
probably be a network company or circuit provider
that provides fixed-telephone (landline) and opticalbroadband services. From here on, however, I would
like to be known as an ICT/IoT (information andcommunication technology/Internet of Things) company that provides its customers with a wide array ofservices that add value to our existing infrastructure.
However, there are already many enterprises working
on ICT and IoT and providing services, so we must
think carefully about what kind of vision we shouldhave toward the future.
To give an example, Japan has been called a developed country facing many challenges such as thedeclining birth rate and aging society as well as
regional disparities. We would like to be a company
that can solve such social problems or provide support by using ICT. NTT WEST, in particular, is incharge of the telecommunication networks of a totalof 30 prefectures spanning the Kinki, Chugoku,
Shikoku, Kyushu, Tokai, and Hokuriku regions,excluding Niigata. The markets that we cover have
many remote islands and are dispersed over a large
area. In short, there are many small markets under ourumbrella, some of which are a microcosm of Japanesesociety. Problems that are left unattended can provoke crises, so I would like to be a company that seessuch situations as opportunities for rolling out newservices that will endear us to the people of thoseregions.
The NTT WEST coverage area includes about 800
local governments of diverse sizes and about 2 million companies, most of which are small or medium
in size. Each of these entities provides services to the
public or to other customers while facing a variety ofproblems. I would like to take on these local governments and companies as our customers and help themsolve their problems or expand their businesses. In
this way, I want to meet the expectations of their end
users. This is none other than the B2B2X (businessto-business-to-X) model that we have been promoting. With this model, we would like to contribute to
the wellbeing of regional communities.
To this end, it is important, first and foremost, that
we keep in mind our own strengths and features.
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

Fortunately for us, we have communication channels
with our customers from past business activities andtrusting relationships that we have been building
through the years. With these as a foundation to workfrom, and using technologies not only for communication but also for artificial intelligence (AI) and IoTplus our physical assets (about 4000 central offices
and related equipment, about 40 datacenters, 26 callcenters, etc.), I would like to promote business process outsourcing services, where companies outsource their work to us, to provide added value to ourcustomers.
For example, many local governments haveexpressed a desire to convert their systems to cloudbased operations, but they are also somewhat anxious
about placing their systems and data outside theirregions. I think we can eliminate this anxiety by providing comprehensive cloud services that enablelocal governments to entrust us with their systems at
our 40 or so datacenters and to even outsource someof their operations.
In addition, there are many small- and mediumsized companies within the NTT WEST coveragearea where regional characteristics make for uniquetypes of businesses. Collaborating with such companies provides another opportunity for contributing tothese regional communities. It should also be possible to collect large amounts of data by establishinglinks between local governments and companies andto extract problems and needs by analyzing thosedata. We can then match up companies and otherenterprises with those problems and needs to workout solutions. I have named this process of regional
revitalization through community-based collaboration “local revitalization clouds.”2

View from the Top

Supporting communities through local
revitalization clouds and promoting regionalfeatures with tourist-oriented services—That’s a unique and easy-to-understand name.
How will you apply local revitalization clouds to the
NTT WEST Medium-term Management Plan?
I believe that these local revitalization clouds will
provide a means of combining people, technologies,
and assets to solve social problems and invigorate
regions and that we can be of assistance in achieving
mutually beneficial collaboration and development.
By forming new partnerships and widening our circle
with each and every achievement, I would eventually
like to create a synergistic effect among 800 local
governments and 2 million companies.
Under the new Medium-term Management Plan,
we plan to substantiate this initiative to build a foundation for achieving revenue growth and stable profits. One specific tactic here is edge computing. In thisregard, I would like to make the primary processing
and analysis of video and other types of data and theirapplication to diverse measures as the centerpiece ofthe local revitalization clouds. This approach could
be used to tackle the energy problem, which is one
key social issue. For example, lead storage batteries
are usually installed in our central offices and datacenters as insurance against power outages caused bydisasters or other events, and they are of the type that
deteriorates through repeated charging and discharging. Since such backup batteries are not used very
often, and periodic inspection and replacement are
conducted, battery deterioration would not be a major
problem. However, such repeated charging and dis-

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

charging makes lead storage batteries unsuitable foreveryday use. However, if lithium-ion batteries could
be used instead, no such deterioration would occur,
which means that they could be used on a day-to-day
basis. If this were to happen, I would like to explore
the use of central offices and datacenters as coreenergy-supply facilities by utilizing AI or ICT tomonitor battery conditions and perform optimal
power transmission and distribution. At the present
stage, cost and safety problems are a concern, so weare studying solutions to these issues together with
the NTT laboratories.
In a way, this is related to the local production andconsumption of energy. A central office typically covers customers dispersed over an area with a radius of
about 8–10 km, and it would be located in the city
center from the viewpoint of efficiency. The electric
power generated by solar panels located in this area
could be used to charge the lithium-ion batteries in
the central office, and customers in the same area
could then be supplied with power. Although a variety of means already exist for transmitting energy
from a power plant to remote locations, they result inmuch power loss that could be reduced by local production and local consumption. I would like toachieve continuous growth as a company that createsa virtuous cycle and endears us to a region throughsuch community-based services and businesses that
invigorate local economies.
—2020 is near at hand. How does NTT WEST plan todeal with this historic year?
The international sports event of 2020 will be held
in and around Tokyo, but this does not mean that all
inbound visitors to the events will remain in theTokyo area. At present, visitors to Japan using KansaiInternational Airport make up about 30% of all visitors. Together with visitors to Japan who use Chubu
Centrair International Airport in Nagoya, Fukuoka
Airport, and other airports in western Japan, we cansay that about 50% of all visitors to Japan get off anairplane somewhere in this area. To provide theseguests with even better services, we are currently
deploying augmented reality systems at airports toprovide information services that can be accessed bysimply holding up a smartphone, and we are providing digital signage to attract customers. I believe thatgoing forward, we can use ICT to make more contributions like these in an indirect way.
We are also preparing services for all major international events to be held before and after 2020 including3

View from the Top

the Rugby World Cup and G20 Summit in 2019.
These services will naturally include navigation services in the manner of airport information services,
but our main mission here is to support communications. In addition, other events that are expected to
draw large numbers of guests are being planned orenvisioned based on the concepts of integrated resorts
and MICE (meetings, incentives, conventions/conferences, and exhibitions). Our main role here will be toprovide support through ICT and AI technologies
including cybersecurity measures.
I feel that taking advantage of these opportunities tosupport inbound visitors may also be effective in revitalizing Japan on the whole. Also, responding to andsupporting people with diverse customs and values
can give us a new perspective on the world. It could
lead to activities for solving social problems such as
the declining birth rate/aging society and regional
disparities.
I think that such initiatives and activities are somehow helping western Japan to come alive comparedwith conditions a few years ago. As I alluded to earlier, it could be said that the 800 local governments
and 2 million large and small companies in westernJapan are collaborating with each other, and that thisdiversity in values and viewpoints is helping to support the economy of western Japan.
Let’s create excitement, inspiration, andbrilliance through “moonshot” thinking—In such an environment, how do you carry out yourbeliefs as a top executive?
NTT WEST employees may be tired of hearing it,
but I often use the word “moonshot.” This word isused by engineers in the United States and goes backto the space development policy and Apollo program
put forth by President John F. Kennedy upon taking
office in 1961. The Cold War between the UnitedStates and Russia (formerly called the Soviet Union)
was an era of fierce competition in space development as a matter of honor. At that time, Russia wasahead, but President Kennedy declared, “This nation
should commit itself to achieving the goal, before thisdecade is out, of landing a man on the moon andreturning him safely to Earth.” These words were notsimply a call to compete through a steady improvement of technology. Rather, they expressed theimportance of setting a goal and looking at whatshould be done to achieve the original objective ofsending human beings into space. The Apollo 11
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

lunar module touched down on the lunar surface in1969 and fulfilled the mission of landing people on
the moon and returning them to Earth.
This word “moonshot” originates from this historic
mission. It expresses the idea of setting a primaryobjective or establishing the goal of understanding
the true essence of something and thinking how toapproach the problem. We tend to make an effort tosolve problems that are close at hand, but it is more
important to consider primary objectives and issues.
By the way, there are three principles behind a goodmoonshot. First, is it original? Second, does it have acertain technical foundation and can its success beenvisioned? And third, does its proposal and planfascinate many people and create a desire for cooperation? I can proudly say that the new NTT WEST
Medium-term Management Plan is a “moonshot.”—Do you have a message for researchers?
I would like to see more research related to regional revitalization, and I would like to work together
with researchers to return the results of that researchto our customers. Some products such as WinActor,
ForeSight Voice Mining, and @InfoCanal havealready been introduced, and I expect more to follow.
WinActor is a robotics process automation tool that
improves business efficiency by using a bot to automate repetitious input tasks, thereby shortening work
time and reducing input errors. Many customers inwestern Japan are using it, and it’s a product that I
heartily recommend to other customers.
ForeSight Voice Mining, meanwhile, is a voiceoriented big data solution. It can be used, for example, at call centers to automatically extract customer
complaints from phone calls. It is not limited to4View from the Top

extracting emotional complaints in which a voice israised in anger, for example. It can also infer andextract emotionally contained, matter-of-fact complaints based on intonation or pauses, which havebeen difficult for existing systems to automaticallyextract. This solution can improve business operations and enable the provision of services that better
fit customer needs.
Finally, @InfoCanal is an information delivery service targeting a wide range of terminals that use an IP(Internet protocol) communications network including the cloud and mobile phone network. It has
already proven to be extremely useful in recent disasters as a means for local governments to deliver disaster information. To give some background, broadcasts made over a community wireless system may bedrowned out by the sound of heavy wind, and it may
not be possible to confirm whether that information
has actually reached the residents. The @InfoCanalproduct makes it possible to deliver disaster information all at once by text and voice to smartphones andtablets, dedicated home receivers for the elderly, anddisaster-prevention radio terminals. It also has bidirectional capabilities that can be used for safety confirmation purposes and for sending individual messages to non-responding terminals for double-checking. This feature enables a local government to visualize the ratio of residents who have received theinformation.
These products all use technologies developed by
the NTT laboratories. NTT Advanced Technology
and NTT TechnoCross transformed these technologies into practical and commercial products, and wehave the role of delivering these products to our customers. There are many examples of such relationships within the NTT Group, too many to mention. I
would like to expand and strengthen these relationships. In addition, I would like to ask researchers towork together with us in making regional contributions by fostering new technologies in edge computing and energy too.
When our engineers make contact with customers,
the customers are relieved to hear comments such as,“We have research laboratories within the Group andmany technologies like this one.” This results in afavorable response from our customers, which underscores just how important a role the NTT laboratories
play. This is why we are committed to delivering the

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

output from the laboratories to our customers.
—Mr. Kobayashi, can you leave us with a message for
all NTT WEST employees?
I would be happy to. In the end, you cannot excel inyour work if you don’t enjoy it. I believe that having
one’s existence as a company or individual recognized as being useful to society is what makes work
pleasurable. There is truly nothing more enjoyablethan having yourself or your company and the valueof your work appreciated as expressed by the words“Glad you’re here!” or “Glad you could do this forme!” To this end, you must take pleasure in yourwork. I believe that having joyful and exciting times
is very important. The role of managers is to create an
exciting company and implement stimulating work
procedures, and we are working hard to create suchan environment, so I would like to ask everyone to be
excited about your work even when times are difficult. If you are excited, you will be inspired, and then
you will be brilliant! If you do your work with a senseof joy, good ideas will come to you. And if good ideas
come your way, you can excel in your work andshine, which then fosters a virtuous cycle of having
the world around you appreciate the work that you
do.

Interviewee profile
Career highlights
Mitsuyoshi Kobayashi joined Nippon Telegraphand Telephone Public Corporation (now NTT) in
1982. He served as General Manager of the
Okayama Branch of NTT WEST beginning in
2006, General Manager of the Service
Management Department of NTT WEST beginning in 2008, Senior Vice President and General
Manager of the Service Management Departmentof NTT WEST beginning in 2010, Senior Vice
President and Director of the Technology Planning
Department of NTT beginning in 2012, andExecutive Vice President and Director of theTechnology Planning Department of NTT beginning in 2014. He took up his present position inJune 2018.

5

Feature Articles: Ultra-realistic Communication with
Kirari! Technology

Kirari! Ultra-realistic
Communication Technology:
Beyond 2020
Akihito Akutsu, Kenichi Minami, and Kota Hidaka
Abstract

The NTT Group is creating new value to overcome the limitations of space through research anddevelopment of Kirari! ultra-realistic communication technology. Kirari! goes beyond simply improving
sound and image quality; it is aimed at achieving a sense of realism, as though the objects or people being
viewed were actually there in front of the viewer. This article gives an overview of Kirari! technology,
introduces some application examples, including live public viewings and other unprecedented productions, and reviews trends for beyond 2020.
Keywords: ultra-realism, media, telepresence

1. Introduction
The 5th Science and Technology Basic Plan [1] wasestablished in a cabinet decision on January 22, 2016
and defines the new concept of Society 5.0 [2]. The
fifth society, or age, in human history, Society 5.0
follows the hunting and gathering society (1.0), theagrarian society (2.0), the industrial society (3.0), andthe information society (4.0) (Fig. 1). It is defined asfollows in the integrated science and technology
innovation strategy set in the cabinet decision of May
24, 2016 [3].
“A human-centered society able to deliver highquality lifestyles, rich in energy and comfort, throughadvanced integration of cyberspace and physicalspace, achieving both economic development andsolutions to social issues by providing goods andservices to meet the detailed, various and latent needsof all, without disparity by region, age, gender, language, or other factors.”
With advances in information and communication
technology (ICT), voice and video communication
with people in distant locations has been achieved
and advanced, increasing in sound and image quality.
It has also become possible to sense the physicalspace we are in, with advances in various types of
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

Fig. 1. Image of Society 5.0.

6Feature Articles

Cyberspace

Imaging equipment
Physical space: London, UK
Sound capturing
equipment

Position sensor

Physical space: Fukuoka, Japan

Fig. 2. Kirari! overcomes boundaries of physical space and cyberspace.

sensing technologies. Media processing of such sensor data in cyberspace and the subsequent transmission and reproduction in another physical space—creating a sense that the person is right before youreyes—may well become common in Society 5.0(Fig. 2). Such reproduction can be called high realism. The Kirari! system being developed by the NTT
Group is the future of media transmission, going
beyond improvements in sound and image quality, toimplement ultra-realistic communication.
With Kirari!, distance can be overcome, and information and images of people and spaces can be transmitted from distant locations in real time. With Kirari!, people can experience a sporting event without
having to travel to the venue, or a speaker can participate in an event from a location far from thevenue. We believe Kirari! will contribute to overcoming the limitations of space in the coming Society 5.0.
Examples using Kirari! are shown in Fig. 3. In theexamples of local live viewing of overseas sports
competitions, which would not be easy to see otherwise, illusion effects have been used to create near
three-dimensional (3D) viewing experiences or toprovide a wider field of view than ever before possiNTT Technical Review

Vol. 16 No. 12 Dec. 2018ble. Astonishing performances not previously possible can also be produced. For example, by sensing theactual performer and applying media processing,another performer can be recreated and overlaid
using illusion effects. By controlling the timing of theoverlay, a joint performance featuring a performer inreal time and an image of that same performer from a
few minutes earlier can be implemented, which
would not otherwise be possible. Through this sort ofinitiative, we hope to provide ultra-realistic experiences that overcome the limitations of space for various types of content.
2. Overview of Kirari! technology
An overview of Kirari! technology is shown in
Fig. 4. It is divided into the three aspects of the sportsor performance venue, communication, and the viewing venue. Kirari! ultra-realistic communication technology is in the communication area. Information
extracted using cameras, microphones, and sensors ishandled by media control, media processing, andsynchronization and transmitted to the viewing
venue.
7

Feature Articles

Example: local live viewing of overseas sports competitions
Pseudo-3D viewing experience by using illusion effects
A wider image viewing field beyond a
16:9 frame
16:9 frame

Example: experiencing a performance using unprecedented stage effects
Joint performance with one’s old self

13:02:20:10(Current)

13:01:00:00(Past)

Fig. 3. Application examples of Kirari!.

Media control consists of spatial information,
which includes position data obtained from sensors,
people in camera imagery, and associated lighting
information, and time control, which is used for controlling the distribution of people in absolute time.
Media processing includes object extraction, in
which people in the captured video can be extracted
from the background, and audio wave-field synthesis
technology [4] for highly realistic sound. For sports
coverage and other performances as shown in Fig. 3,
these features use illusion effects to display people in
3D and perform the processing needed to reproduce
audio that seems to come from the image to theviewer’s position. In the other wide-angle viewing
experience in Fig. 3, images from multiple cameras
are synthesized to produce an ultrawide image [5].
Then encoding is done to transmit the content efficiently.
We have created our own extension to the MPEG*Media Transport (MMT) standard, called Advanced
MMT, which we use for synchronized transmission[6]. It is capable of synchronization with absolutetime, so synchronized transmission to any location in
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

the world with the same timing is possible. Advanced
MMT has the role of providing a design plan forachieving ultra-realism based on video, audio, lighting, and other information. At the viewing venue,
projection, speakers, lighting, and other elements are
set up according to the Advanced MMT design plan.
Some of the technologies used in Kirari! areexplained in the Feature Articles in this issue [7–10].
3. Future prospects
To allow as many users as possible to experience
the ultra-realism achieved by Kirari!, we have conducted proof-of-concept and live viewing trials,
including sports events that seem to be happening
before the viewer’s eyes, unprecedented performing
arts using ICT, dance performances featuring bothcurrent and past performers, and coverage of lectures*

MPEG: Moving Picture Experts Group, a working group of ISO(International Organization for Standardization) and IEC (International Electrotechnical Commission) in charge of developing
international standards for compression of audio and video data.

8

Feature Articles

Communication
Media control
Sports and performance venue

Spatial information

Viewing venue

Time control

CamerasProjection
Media processing

Microphones

Object extraction

Highly realistic
sound

Sensors

Ultrawide video
stitching

Encoding

Speakers

Field reproduction
Lighting effects

Lighting information
Synchronized transmission (Advanced MMT)

Kirari! ultra-realistic communication technology
MMT: MPEG Media Transport

Fig. 4. Overview of Kirari! technology.

Beyond 2020
Promotion of current use examples by advancing and economizing Kirari!
SportsTheatrical performance

Maintenance andmonitoring
Ultrawide video
stitching

RoadwaysTrainstationLive viewing as if athletes were rightin front of the viewer

Traditional performing art at theaterwith effects never before possible

Dance

LectureLive viewing at a theater
Joint performance with past dancers

Remote lectures and seminars
at a seminar hall

Telemedicine

Reception areas
and teleworking

Fig. 5. Further development of Kirari!.

from remote locations (Fig. 5). One advancement toKirari! that is needed to promote these measures is
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

more accurate extraction of the object being viewed,
even in environments with people coming and going.
9Feature Articles

Another issue is to create ultrawide compositions ofhigh-definition video.
Measures to reduce costs are also an important
issue for implementation in society. For example, weare studying ways to further compress high-definition
video, and to reduce the number of media processing
servers required. Also, with more advanced media
control, it will become possible to automatically andaccurately overlay video of performers from the pastnext to real performers. This could contribute to
reducing the personnel costs for productions.
In addition to promoting Kirari! in society withsports and other performances, we are looking at possibilities beyond 2020. For example, in fields such as
maintenance and surveillance, roadways and train
stations could be monitored using ultrawide composed video, giving an experience in which items
being monitored seem to be right before the viewer’seyes. Increasingly, applications are being studied in
medicine for remote examinations, in business situations to replace receptionists, and to promote telecommuting. With these initiatives, we will continue
to create value by overcoming the limitations of spacein Society 5.0.
References[1][2]

Government of Japan, “The 5th Science and Technology Basic Plan,”
Jan. 2016.
http://www8.cao.go.jp/cstp/english/basic/5thbasicplan.pdf
K. Hidaka, Y. Hasegawa, and H. Fuseda, “A Platform for Realizing a
New Economy and Society: Society 5.0,” Operations Research, Vol.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

61, No. 9, pp. 551–555, 2016 (in Japanese).
Government of Japan, “Comprehensive Strategy on Science, Technology and Innovation 2016 (Excerpt),” May 2016.
http://www8.cao.go.jp/cstp/english/doc/2016stistrategy_main.pdf[4] K. Tsutsumi and H. Takada, “Powerful Sound Effects at Audience
Seats by Wave Field Synthesis,” NTT Technical Review, Vol. 15, No.
12, 2017.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201712fa5.html[5] T. Sato, K. Namba, M. Ono, Y. Kikuchi, T. Yamaguchi, and A. Ono,
“Surround Video Stitching and Synchronous Transmission Technology for Immersive Live Broadcasting of Entire Sports Venues,” NTT
Technical Review, Vol. 15, No. 12, 2017.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201712fa4.html[6] Y. Tonomura, H. Imanaka, K. Tanaka, T. Morizumi, and K. Suzuki,“Standardization Activity for Immersive Live Experience (ILE),” ITU
Journal, Vol. 47, No. 5, pp. 14–17, 2017 (in Japanese).
[7] H. Kakinuma, J. Nagao, H. Miyashita, Y. Tonomura, H. Nagata, and
K. Hidaka, “Real-time Extraction of Objects from Any Background
Using Machine Learning,” NTT Technical Review, Vol. 16, No. 12,
pp. 12–18, 2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201812fa2.html[8] T. Isaka, M. Makiguchi, and H. Takada, “‘Kirari! for Arena’—Highly
Realistic Public Viewing from Multiple Directions,” NTT Technical
Review, Vol. 16, No. 12, pp. 19–23, 2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201812fa3.html[9] M. Makiguchi and H. Takada, “360-degree Tabletop Glassless 3D
Screen System,” NTT Technical Review, Vol. 16, No. 12, pp. 24–28,
2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201812fa4.html[10] M. Isogai, K. Okami, M. Matsumura, M. Date, A. Kameda, H. Noto,and H. Kimata, “Video Processing/Display Technology for Reconstructing the Playing Field in Sports Viewing Service Using VR/AR,”NTT Technical Review, Vol. 16, No. 12, pp. 29–35, 2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201812fa5.html[3]10Feature Articles

Akihito Akutsu

Vice President, NTT Service Evolution Laboratories.
He received an M.E. in engineering from Chiba
University in 1990 and a Ph.D. in natural scienceand technology from Kanazawa University in
2001. Since joining NTT in 1990, he has been
engaged in research and development (R&D) ofvideo indexing technology based on image/video
processing, and man-machine interface architecture design. From 2003 to 2006, he was withNTT EAST, where he was involved in managing
a joint venture between NTT EAST and Japanese
broadcasters. In 2008, he was appointed Directorof NTT Cyber Solutions Laboratories (now NTT
Service Evolution Laboratories), where heworked on an R&D project focused on broadband and broadcast services. In October 2013, he
was appointed Executive Producer of 4K/8K
HEVC (High Efficiency Video Coding) at NTT
Media Intelligence Laboratories. He received theYoung Engineer Award and Best Paper Award
from the Institute of Electronics, Information andCommunication Engineers (IEICE) in 1993 and2000, respectively. He is a member of IEICE.

Kota Hidaka

Senior Research Engineer, Supervisor, Group
Leader, NTT Service Evolution Laboratories.
He received an M.E. in applied physics from
Kyushu University, Fukuoka, in 1998, and a
Ph.D. in media and governance from Keio University, Tokyo, in 2009. He joined NTT in 1998.
His research interests include speech signal processing, image processing, and immersive tele­presence. He was a Senior Researcher at theCouncil for Science, Technology and Innovation,
Cabinet Office, Government of Japan, from 2015
to 2017.

Kenichi Minami

Executive Research Engineer, Natural Communication Project, NTT Service Evolution
Laboratories.
He received a B.E. in electronic engineeringand an M.S. in biomedical engineering from
Keio University, Kanagawa in 1991 and 1993. Hereceived an MBA from Thunderbird, Global
School of Management, Arizona, USA, in 2002.
He has been engaged in R&D management in thedevelopment of “Kirari!” immersive telepresence technology since 2016. During 2012–2014,
he was responsible for the development ofmobile application services at NTT DOCOMO.
His research interests include image and audio
processing, user interfaces, and telepresencetechnologies. He is a member of IEICE.

NTT Technical Review

Vol. 16 No. 12 Dec. 201811

Feature Articles: Ultra-realistic Communication with
Kirari! Technology

Real-time Extraction of Objects from
Any Background Using Machine
Learning
Hirokazu Kakinuma, Jiro Nagao, Hiromu Miyashita,
Yoshihide Tonomura, Hidenobu Nagata, and Kota Hidaka
Abstract

NTT Service Evolution Laboratories is conducting research and development on the Kirari! ultrarealistic communication system, which can make an athlete or performer in a remote location seem tobe right before the viewer’s eyes. This article describes a system that can extract objects from an arbitrary
background in real time, which is essential for realistic remote presentation of athletes and performers
using pseudo-three-dimensional video and other techniques.
Keywords: image segmentation, machine learning, ultra-realism

1. Introduction
The ability to accurately identify the region of aperson or object within an image is an essential technology for performing high-quality image editing andcomposition, and it is therefore a major research
theme in computer vision. Selecting the object regionis also essential for achieving realism in the Kirari!
ultra-realistic communication system when performing pseudo-three-dimensional video display. In thearticle “Real-time Extraction of Objects with Arbitrary Backgrounds” [1], NTT Service Evolution
Laboratories proposed a system able to extract only
the object region in real time from video of a sports
venue or performance stage, without using studioequipment such as a green screen. This article introduces a system that can extract object regions withgreater accuracy. It was developed by introducing
machine learning to distinguish more subtle differences in feature values that have been indistinguishable earlier and generating feature values of theobject being extracted using infrared light.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

2. Framework for real-time object extraction
using machine learning
Background subtraction is a common method forextracting objects in real time. Background subtraction involves finding the differences between the
input image and a background image and applying athreshold to identify changes as the object area. Thismethod is fast and requires little preparation, so it is
widely used. However, there are challenges with thismethod, including the difficulty in deciding an appropriate threshold value and the inability to support
backgrounds that change.
We have developed an object extraction methodthat uses a neural network (NN) to convert input feature vectors to a different feature space and performthe discrimination. With the NN, we hope to derive afeature space for making discriminations within the
NN using training data provided beforehand and toautomatically convert to this more suitable feature
space. Inputs other than the image being detected, forexample, reference images with features of theobject, images from different times, region information, and infrared images, can also be included andwill also be converted to suitable feature spaces in the12Feature Articles

Object extraction image

Input image
Composition

Object (foreground)
Background
Unclassified (unknown)
Boundaryarea
correction

Roughcapturing(Images photographedin advance)
Background image

Rough mask image(trimap)

Accurate mask image(bimap)1 pixel offoreground image

1 pixel of
input image

Fig. 1. Framework for real-time object extraction using machine learning.

Posteriorprobability offoregroundregions

Rt

FG

Gt
Bt

Combinations of quantizedsix-order feature vectors

0
0
0
0

0
0
0
0

0
0
0
0

0
0
0
0

0 0
0 8
0 16
0 24

0.0
0.0
0.1
0.2

Rb
Gb

BG
Posteriorprobability ofbackgroundregions

Bb(a) NN model that discriminates foreground andbackground regions

255 255 255 255 255 232
255 255 255 255 255 240
255 255 255 255 255 248
255 255 255 255 255 255

1.0
0.5
0.0
0.0

(b) NN output combinations are stored
as look-up table (LUT)

Fig. 2. Training process.

NN in order to perform object extraction using higher-order feature values of the background and object.
This should make operations such as changing thebackground more robust.
The workflow for the system we developed isshown in Fig. 1. Object extraction is done in two
steps. In the first step, the object region is selectedusing a rough mask image (a trimap*1), and in thesecond step, a matting process*2 dependent on thistrimap creates a more accurate object region. Machinelearning is used to capture trimap in the first step.
Machine learning is divided broadly into training
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

and application processes. In the training process,parameters in the NN model are learned from training
data. This process is shown in Fig. 2. Training data
are first prepared. Background images that do notcontain the object, and sample images that do contain*1 Trimap: A region map indicating known and unknown regions ofan image. Known foreground regions are set to white, known
background regions to black, and unknown regions to grey.
*2 Matting process: A process that derives an alpha mask for extracting the object. The alpha mask has values ranging from 0 to
1, and the extracted image is obtained by multiplying the input
image by the mask.

13

Feature Articles

 Index values

0.0
0.0
0.1
0.2

(b) Posterior probability
image of foregroundregions

Input image

1.0
0.5
0.0
0.0
Background image…Foreground…Background…Unclassified

LUT(a) Conversion of input feature vectors byreferencing LUT in all coordinates(c) Trimap

Fig. 3. Application process.

the object are obtained, and correct mask images are
created. Then, input feature vectors are created.
These feature vectors combine target pixels from thesample image corresponding to the foreground areain the mask, with the corresponding pixels from thebackground image. These combinations are used totrain for the foreground region. Similarly, input feature vectors combining the background pixels fromthe sample image and corresponding pixels from thebackground image are created, and these combinations are used to train for the background region.
In this way, we obtained an NN model able to discriminate the foreground from the backgroundregions for combinations of input target pixels andbackground image pixels (Fig. 2(a)). Generally, NN
processing requires large computing resources, so we
increased speed by implementing processing using alook-up table (LUT). We reduced the number of gradations in the input feature vectors by quantizingthem and stored all combinations of quantized input
feature vectors and NN outputs as an LUT (Fig. 2(b)).
Note that in Fig. 2, we describe this process in termsof RGB (red, green, and blue color model) pixels for
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

simplicity, but the input vectors can include morethan color information, for example, image position.
The process for applying the LUT to generate thetrimap is shown in Fig. 3. A quantized input feature
vector is derived in a process similar to that for themachine learning process, and this derived feature
vector is used to reference the LUT, rapidly determining a posterior probability that the pixel in question isin the foreground (Fig. 3(b)). The trimap is generated
from the derived foreground-posterior-probability
image by setting regions that are not clearly foreground or background to the unclassified region (Fig.
3(c)). The unclassified region in the trimap is discriminated by using a nearest-neighbor search withinformation about whether pixels neighboring the
pixel in question and having similar feature vectors
were classified as foreground or background.
The details of this nearest-neighbor search are
shown in Fig. 4. For each pixel in the unclassified
region, a spiral search in the local neighborhood isdone to determine whether the pixel is more similar
to a foreground pixel or a background pixel, and thisis used to derive an alpha value for the pixel. The
14

Feature Articles

Comparison map
Input image

Mask image after boundary correction

Unclassified
regionin trimap

Background
image
Object extraction image
For each pixel in unclassified region, surrounding pixels(of input and background images) are searched todetermine the pixel similar to a target pixel and copy it.

Fig. 4. Boundary correction by nearest-neighbor search, and masking process.

object is then extracted using the derived alpha values. By introducing this boundary correction process,we can extract objects based on information from thepixel itself and also from surrounding pixels. We also
provide a framework for optimizing this process suchas by processing the rough mask on a low-resolution
image, or by performing only the magnification process on all pixels.
3. Real-time extraction of objects frombackgrounds of similar color using infrared
Even though machine learning is used to automatically convert input feature vectors to a high-orderfeature space within the NN, we are still extracting
objects based on color and shape information, so it istheoretically impossible to separate cases when the
input feature vectors are the same. For this reason, wedeveloped an object extraction system using RGB
and infrared (IR) cameras, utilizing IR light that is
invisible to the naked eye in an attempt to add newfeatures.
The object extraction photography environment
using the RGB and IR cameras is shown in Fig. 5. An
RGB camera and an IR camera were placed side byside, respectively capturing visible light and IR
images. The background was illuminated with IR,
while deliberately preventing IR light from reflectingfrom the object. In this way, the background appearedbrighter in the IR image, while the object was relatively dark.
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

In the RGB camera image in Fig. 5, the backgroundand the person are of the same color, so it is difficult
to distinguish them based on color, but the person’s
silhouette can be obtained from the IR camera image.
Then, to extract the object accurately from the IR
camera image, we correct for the parallax between
the IR camera and the RGB camera. To perform thecorrection, photos of a calibration board were taken
beforehand, a projection transform from the IR image
was derived so that the same feature points from theIR camera and the RGB camera aligned, and thistransform was applied. By adding the IR image
obtained in this way to the input of machine learning,
we were able to extract objects using new featuresother than color and shape.
4. Evaluation experiments
We developed a real-time object extraction systemusing machine learning and used it in the Cho-Kabuki play called “Tsumoru Omoi Hana no Kaomise,” atthe Niconico Chokaigi event held in April 2018 at
Makuhari Messe, Chiba, Japan. The system developed is capable of 3840 × 2160 resolution video at aframe rate of 60 fps, but for this trial, a resolution of1920 × 1080 and a frame rate of 59.94 fps were used
for compatibility with other systems. In the climax
scene of the Cho-Kabuki main production, Koretakashinno, played by Shido Nakamura II, emerges fromthe screen in real time as the stage video backgroundchanges, while the confrontation with Princess
15Feature Articles

IR camera
RGB camera

Object extraction
equipment
Image using
RGB camera
ObjectIR light

Background

Image using IR camera

Fig. 5. Object extraction photography environment using RGB and IR cameras.

(a) Input image(b) Object extraction image using
background subtraction(c) Object extraction image using machine learning

Fig. 6. Real-time object extraction using machine learning on kabuki stage.

Hatsune Ono, played by virtual personality, Miku
Hatsune, comes to its peak. Samples of the video atthat point are shown in Fig. 6. The board behind
Koretaka-shinno is being held by stage-hands, so it is
not steady, and the image cannot be extracted wellusing background subtraction (Fig. 6(b)). We showedthat it can be extracted more accurately using oursystem using machine learning (Fig. 6(c)).
We also checked the effect of using the IR camera
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

(Fig. 7). We were able to confirm that objects can beextracted accurately, even in cases where it was difficult using an RGB camera. In this case, we did not
do the automatic conversion of input feature vectors
to a high order feature space or the threshold processing in the NN training, but we evaluated how robust
the method using the IR camera would be by applying
background subtraction and changing the thresholdvalues (Fig. 8). We confirmed that using the IR camera
16Feature Articles(a) Image using RGB camera(b) Image using IR camera(c) Result of object extraction

Fig. 7. Object extraction with machine learning using RGB and IR cameras.

1.0

0.9

0.9

0.8

0.8

0.7

Precision
RecallDecrease

1.0

Recall

0.6
Threshold value

0.7

Precision

Increase

Decrease(a) When using RGB camera only

0.6
Threshold value

Increase(b) When using both RGB and IR cameras

Fig. 8. Precision and recall using background subtraction.

enabled us to achieve very reproducible results aswell as stable operation, even when the thresholdvalues changed considerably.
We have introduced cases using an IR camera, butother characteristics suitable for extracting objects,
depending on the conditions, can be input to the system, for example, depth maps generated using stereo
cameras or LiDAR (light detection and ranging).

the use of features based on IR light, which is not visible to the naked eye, to handle use cases where thisextraction is difficult using only an RGB camera.
In the future, to consider semantics when performing object extraction, we will work to perform object
extraction using deep learning in real time and also
study methods for extracting objects accurately when
there are occlusions.

5. Future prospects

Reference

This article introduced a highly accurate object
extraction method based on high-order feature valuesof objects, using machine learning to convert input
feature vectors to a new feature space and performing
the discrimination within an NN. We also introduced

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

[1]

H. Nagata, H. Miyashita, H. Kakinuma, and M. Yamaguchi, “Realtime Extraction of Objects with Arbitrary Backgrounds,” NTT Technical Review, Vol. 15, No. 12, 2017.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201712fa7.html

17Feature Articles

Hirokazu Kakinuma

Engineer, Natural Communication Project,
NTT Service Evolution Laboratories.
He received an M.E. in information sciencefrom the Graduate School of Advanced Integration Science, Chiba University, in 2010. He
joined NTT in 2010 and studied database management systems for Internet protocol television(IPTV). From 2013 to 2016, he worked at NTT
Plala, where he developed set-top box web applications and released the Hikari TV 4K service for4K digital TV. He moved to NTT in 2016 and iscurrently researching real-time image segmentation technology based on machine learning.

Jiro Nagao

Research Engineer, NTT Service Evolution
Laboratories.
He received a Ph.D. in information sciencefrom Nagoya University, Aichi, in 2007. He
joined NTT in 2007. His research interests
include image processing for computer recognition and presentation.

Yoshihide Tonomura

Senior Research Engineer, Natural Communication Project, NTT Service Evolution Laboratories.
He received an M.S. in electronics engineeringfrom Nagaoka University of Technology, Niigata, in 2004 and a Ph.D. from Tokyo Metropolitan
University in 2010. Since joining NTT, he has
been researching and developing multimedia
data transmission technologies. From 2011 to2012, he was a visiting scientist at MIT MediaLab, Massachusetts, USA. He received the
SUEMATSU-Yasuharu Award from the Instituteof Electronics, Information and Communication
Engineers in 2016 and the 26th TAF TELECOM
System Technology Award from the Telecommunications Advancement Foundation in 2011.

Hidenobu Nagata

Senior Research Engineer, Natural Communication Project, NTT Service Evolution Laboratories.
He received an M.E. in systems and information engineering from Hokkaido University in
2001. He joined NTT in 2001 and studied video
technologies including video indexing and automatic summarization. From 2008 to 2014, heworked at NTT Electronics and developed transcoders and embedded audio IP. He is currently
researching ultra-realistic communication technology including the immersive telepresencetechnology called “Kirari!”.

Hiromu Miyashita

Engineer, Natural Communication Project,
NTT Service Evolution Laboratories.
He received a B.E. and M.E. from Keio University, Kanagawa, in 2008 and 2010. He joined
NTT Service Evolution Laboratories in 2010,
where he worked on human computer interaction, image and video processing, and ultrahighpresence telecommunication services.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

Kota Hidaka

Senior Research Engineer, Supervisor, Group
Leader, NTT Service Evolution Laboratories.
He received an M.E. in applied physics from
Kyushu University, Fukuoka, in 1998, and a
Ph.D. in media and governance from Keio University, Tokyo, in 2009. He joined NTT in 1998.
His research interests include speech signal processing, image processing, and immersive tele­presence. He was a Senior Researcher at theCouncil for Science, Technology and Innovation,
Cabinet Office, Government of Japan, from 2015
to 2017.

18

Feature Articles: Ultra-realistic Communication with
Kirari! Technology“Kirari! for Arena”—Highly Realistic
Public Viewing from Multiple
Directions
Takeru Isaka, Motohiro Makiguchi, and Hideaki Takada
Abstract

NTT Service Evolution Laboratories has developed a display system that achieves highly realistic
public viewing surrounding the venues of sporting events. We utilize human visual perception to makeviewers perceive the subject in space, including a motion in the depth direction, in order to achieve a
presentation that appears as though the subject was actually in front of the viewer. This article gives an
overview of an approach to present motion in the depth direction by using visual perception, and a display system that can be implemented with a simple configuration and enables viewing from anywhere
surrounding the display.
Keywords: high realistic, depth perception, floating image

1. Highly realistic public viewing
From television (TV) screens placed where passers-by on the street can watch them, to public viewingof various events, media that enable viewers to meetin one place and enjoy sports events have made it possible for viewers to share in the excitement of theevent by emphasizing the connection between them.
We expect that with the spread of 4K and 8K broadcasts in 2020, many more sporting events fromaround the world will be covered on TV, and such
excitement will be shared much more widely, regardless of the location or time, through various viewing
styles such as live distribution on the Internet andinformal public events in the street.
NTT Service Evolution Laboratories is taking thisdiversification in viewing styles around the world
into account while conducting research and development on the “Kirari!” immersive telepresence technology, which implements highly realistic public
viewing in real time from any location, giving viewers the sense of actually being at an event taking placeelsewhere.
Research is in progress on various imaging methods for “Kirari!”; one method uses two-dimensional
NTT Technical Review

Vol. 16 No. 12 Dec. 2018(2D) floating images and can display life-sized images as though they were actually on the stage in frontof the viewer. The method uses virtual images andinvolves a simple configuration combining an ordinary 2D display with an optical element such as ahalf-mirror, which transmits and reflects part of theincident light (Fig. 1). The method presents the image
in a position separated from the display, so it appears
that the image is floating in the same space as theviewer, and subjects in the image are perceived as real
objects [1]. This method can be used for public viewing of sports events, giving a sense of reality as
though the event was actually happening in front of
the viewer.
2. Greater reality using floating images
For most cases of public viewing, many viewers
usually gather at the same location to watch the content at the same time, so the content must be presented accurately in space for a wide range of viewpoints. However, until now, images from the same
viewpoint have been presented on TV or on screens
to all viewers, so it has not been possible to correctly
reproduce different positions and distances in the19

Feature Articles

ViewerHalf-mirror

2D floating image(virtual image)Stage

2D display

Fig. 1. Setup of 2D floating image.

content. This issue also occurs when using 2D floating images.
Many methods have been developed to present spatial positions of subjects for a wide range of viewpoints [2, 3]. However, most of these methods haverequired many projectors or special optical systems,with large and complex equipment that makes it difficult to present the content as life-sized. Therefore,
these methods have been difficult to apply for public
viewing.
We have instead focused on a method with a simple
configuration that uses 2D floating images. It can
expand the size of the content and can be easily
applied to smartphones and tablets [4].
With 2D floating images, the virtual image plane is
at a fixed position in space, so it is not possible toreproduce motion in the depth direction. Prior
research has been done on adding motion in the depth
direction to 2D floating images [5], but it was limited
to a single viewer and required the use of special
glasses. To date, there have been no attempts to reproduce the motion of subjects in the depth direction for
multiple viewers over a wide range of viewpoints.
In this article, we propose a method with a simple
configuration to present motion in the depth direction
to multiple viewers by implementing psychologicaldepth-perception effects, and we introduce an application of this method to a display system that enables
viewing from all around.
3. Perspective and depth perception
The concept of perspective has long been known to
be a way to express depth based on psychological facNTT Technical Review

Vol. 16 No. 12 Dec. 2018

tors. When the positions and sizes of subjects are
drawn as they would appear to the eye, the viewer isable to recognize significant distances, even whendrawn in a picture or other 2D fixed surface. It is also
possible to present motion in the depth direction by
applying perspective to 2D floating images.
The position and size of the subject drawn using
perspective converge to a point as the distance to thesubject increases. Such a point is called a vanishingpoint. As an example, if the left and right edges of afloor are extended infinitely in the depth direction,
they will cross at the vanishing point in an image. If a
subject moves straight in the depth direction, its
motion will also be toward this vanishing point. If
there are multiple viewpoints, the vanishing points
will be different for each viewpoint, so subjects mustalso move in a direction determined for each viewpoint. When 2D floating images are displayed, there
are correct vanishing points for each viewpoint,depending on the stage and other real objects, but
only one vanishing point for one viewpoint can be set
for subjects displayed as floating images. When such
images are viewed from different viewpoints, motionin the correct depth direction cannot be represented.
4. Method to present motion in thedepth direction
As shown in Fig. 2, even when there is error, andthe direction of the subject motion (shown with thesquare and the dotted line connected to it) does notmatch the vanishing point of the floor plane (shownwith the circle), people can empirically perceive
motion in the depth direction. We have experimentally
20

Feature Articles

Vanishing point
Range of allowable error

Direction of thesubject motion

Subject
Floor

Fig. 2. Range of allowable error in which motion in the depth direction is recognized.

verified and defined a range of allowable error within
which this motion in the depth direction is recognized[6]. We have also proposed a method for representing
motion in the depth direction for multiple viewpoints
by setting this permissible range for each viewpoint
and restricting the motion of subjects to within arange common to all viewpoints.
The permissible range of error within which motionin the depth direction is recognized depends strongly
on the positional relationship between the subject andthe floor plane. If the motion of the subject is within
the triangular region formed by the feet of the subject
and the left and right endpoints at the back of the floor
plane, any perceived strangeness in motion in thedepth direction will be reduced. With this in mind, we
derived a method for representing motion in the depth
direction with 2D floating images as an algorithm
that determines the subject position in each frame
using information on the subject position and theviewing range. We can also adjust the representationof subjects in the virtual image so that they have contact with the real-space floor surfaces.
With this method, we can represent the motion ofsubjects in the depth direction without any apparent
inconsistencies and ensure that they remain connected to the floor. The paths of motion for subjects
before and after applying this depth assignment
method are shown in Fig. 3. The movement of thesubject is shown with the solid line, while the permitted range of motion is the region bounded by the dotted line and the floor surface. In this figure, before the
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

proposed method was applied, the path of motion
goes out of the permitted region, but after the method
was applied, it stayed within the region. This showsthat for life-sized subjects moving within a floor
range of 10 m × 10 m, motion in the depth direction
with a range of 7.5 to 9.0 m can be represented.
5. Prototype of “Kirari! for Arena”
We built a prototype “Kirari! for Arena” system thatenables viewing from all around by combining four
optical systems consisting of a 2D display and a halfmirror, all sharing a single floor surface. Images ofthe subjects captured from four directions are displayed on the virtual image planes (Fig. 4). The system takes images of just the subjects, extracted fromtheir respective backgrounds, as well as subject distance information measured using LiDAR (lightdetection and ranging). The method described here isused to process and display the images in real time.
Compared to existing display systems requiring
many projectors and special optical devices, this system is capable of enabling viewing from all aroundusing a simple configuration of ordinary displays andhalf-mirrors. Because it is so practical, we hope toapply it for public viewing of sports and other events,
where many people surround the venue while watching.

21

Feature Articles

Boundary of permitted
range of motion

After applying the method

Before applying the method

Path of subject’s motion

Subject
Floor

End time of motion

Start time of motion

Time

Fig. 3. Path of subject’s motion before and after applying proposed method.

2D display

Subject in 2D
floating image

Half-mirror

ViewerFloor
Appearance of actual observation

System configuration

Fig. 4. Prototype of “Kirari! for Arena”.

6. Future work
In this study, we focused on 2D floating images
implemented using a simple configuration combining
NTT Technical Review

Vol. 16 No. 12 Dec. 2018a display and half-mirror. We also proposed a method
for representing depth to multiple viewers when subjects being viewed move in the depth direction, byusing psychological depth-perception effects. We
22

Feature Articles

applied this method in prototyping “Kirari! for
Arena,” which provides viewing of content from allaround, including its position in space.
In the future, we plan to represent the position ofsubjects more precisely and expand the viewing area
by applying shadows or other depth cues.
References[1][2]

D. Miyazaki, “Current Status and Future Prospects of Floating 3D
Display Technology,” Japanese Journal of Optics, Vol. 40, No. 12, pp.
608–615, 2011 (in Japanese).
S. Yoshida, “fVisiOn: 360-degree Viewable Glasses-free Tabletop 3D
Display Composed of Conical Screen and Modular Projector Arrays,”

Takeru Isaka

Member, NTT Service Evolution Laboratories.
He received an M.E. in engineering from Keio
University, Kanagawa, in 2016. He joined NTT
in 2016. He currently works at NTT Service
Evolution Laboratories. He has been engaged in
the research of 3D visual perception.

[3]

[4]

[5]
[6]

Opt. Exp., Vol. 24, No. 12, pp. 13194–13203, 2016.
A. Butler, O. Hilliges, S. Izadi, S. Hodges, D. Molyneaux, D. Kim,and D. Kong, “Vermeer: Direct Interaction with a 360° Viewable 3D
Display,” Proc. of the 24th ACM Symposium on User Interface Software and Technology (UIST 2011), pp. 569–576, Santa Barbara,USA, Oct. 2011.
M. Makiguchi, H. Takada, A. Niijima, T. Matsui, and M. Yokoyama,“Proposal and Implementation of Compact Multilayer Floating Imaging Projection System Using Mobile Device,” IPSJ Journal, Vol. 58,
No. 11, pp. 1787–1798, 2017.
O. Bimber, B. Fröhlich, D. Schmalstieg, and L. M. Encarnação, “The
Virtual Showcase,” ACM SIGGRAPH 2005 Courses, No. 3, Los
Angeles, USA, July/Aug. 2005.
T. Isaka, M. Makiguchi, H. Takada, and A. Ono, “A Study of Perceived Depth Motion in 2D Aerial Images,” Proc. of the 2017 ITE
Annual Convention, 34D-5, Tokyo, Japan, Aug./Sept. 2017 (in Japanese).

Hideaki Takada

Senior Research Engineer, Supervisor, NTT
Service Evolution Laboratories.
He received an M.E. in information systems
from the University of Electro-Communications,Tokyo, in 1997. He received a D.S. in global
information and telecommunication studies fromWaseda University, Tokyo, in 2007. He joined
NTT in 1997. He is engaged in research of 3D
visual perception and 3D display technology. He
received an Achievement Award from the Institute of Electronics, Information and Communication Engineers in 2003 and a commendationfrom the Minister of Education, Culture, Sports,
Science and Technology Japan in 2006.

Motohiro Makiguchi

Researcher, Natural Communication Project,
NTT Service Evolution Laboratories.
He received a Master of Information Technology from the Graduate School of Information
Science and Technology, Hokkaido University,
in 2012. He joined NTT Service Evolution Laboratories in 2012. He has been researching 3D
display technology and virtual reality technology.

NTT Technical Review

Vol. 16 No. 12 Dec. 201823

Feature Articles: Ultra-realistic Communication with
Kirari! Technology

360-degree Tabletop Glassless 3D
Screen System
Motohiro Makiguchi and Hideaki Takada
Abstract

NTT Service Evolution Laboratories has developed 360-degree tabletop glassless three-dimensional(3D) screen technology, which produces 3D images that appear on a table and provides experiences ofviewing from the full perimeter of sports venues and other scenes. This article deals mainly with thebasic visual mechanism of depth perception and the optical configuration for projector placement used
for this technology.
Keywords: autostereoscopic 3D, linear blending, smooth motion parallax

1. Introduction
Expectations are rising for autostereoscopic threedimensional (3D) screens that can achieve smooth
motion parallax and natural 3D viewing without
requiring 3D glasses or other such mechanisms as theultimate display system of the future. In particular,tabletop 3D screen technologies, which can display
items as though they were actually on the table, areanticipated for a wide range of applications such as
viewing live sporting events with an overhead view ofthe entire field, or for modeling manufactured products. With this system, several viewers can standaround the table and watch the sports event from
whatever direction they want. This could dramatically improve the communication environment among
viewers. Autostereoscopic 3D screen technology isnecessary to create this sort of experience, as it supports 360-degree motion parallax and does not
require the use of 3D glasses or other special mechanisms that become an obstacle to eye contact or seeing the facial expressions of other viewers.
Autostereoscopic 3D screen technologies with
360-degree motion parallax have been proposed
before, including a system that uses several hundredprojectors placed at very small intervals that project
images onto a special cone-shaped screen [1]. The
use of multiple projectors to project viewpoint images in this way is advantageous in that multiple people
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

can view the 3D content without glasses at the sametime. However, to switch smoothly between video
sources as the viewpoint moves, many projectors atvery close intervals are needed, which increases
equipment costs and the scale and complexity of the
facility.
We previously proposed the basic technology for an
autostereoscopic 3D screen system that supports
horizontal motion parallax using a 50-inch diagonal
screen and 13 projectors [2]. This system achieves
smooth viewpoint motion requiring just one-fourth to
one-tenth the number of projectors of earlier systems
by using a perceptual mechanism of the visual systemcalled linear blending, in which the luminance of theimage from neighboring viewpoints is composed
based on the viewing position, so that the intermediate viewpoint is perceived as visually interpolated.
Linear blending is achieved optically using a spatially imaged iris plane optical screen. We are continuing to study the implementation of a tabletop
screen by applying this screen in the horizontal direction and have implemented a prototype that is effective in reducing the number of projectors using linear
blending [3]. However, in our investigation so far, ascreen and projection system developed for wall
mounting were simply extended to a tabletop form, so
optical design constraints limited the field of view toapproximately ±65 degrees of center, which did notachieve 360-degree viewing.
24

Feature Articles

Luminance
ratio
Left: Right(a) Retinal image
Left Right(b) Perceived image

10: 0
9: 1
8: 2

Perceived
edge

7: 3
6: 4
5: 5
4: 6
3: 7
2: 8
1: 9
0:10

Fig. 1. Image edge perception mechanism with linear blending.

In this article, we describe a new optical configuration to expand the tabletop linear-blending autostereoscopic 3D screen system for 360-degree viewing,
and a new tabletop type prototype using 60 projectors.
2. Utilization of perception mechanisms of thevisual system
For two images that overlap with only a small offset
to the left or right, if the relative brightness of each
image is varied, an image with double edges to theleft and right is projected onto the human retina, as
shown in Fig. 1(a). However, in the human visualsystem, if the distance between these two edges in theimage is sufficiently small, it is perceived as one edge
instead of two, with the position of the edge smoothly
changing according to the relative brightness, as
shown in Fig. 1(b) [3]. This principle, based on using
this image edge perception mechanism to blend viewpoint images from adjacent projectors according to
the relative luminance, is called linear blending inthis article.
Earlier autostereoscopic 3D screens using multiple
projectors had projectors positioned at intervals narrower than the distance between viewers’ eyes, so thatbinocular and motion parallax would be displayedsmoothly. This required large numbers of projectors
as the viewing range was increased, as shown in
Fig. 2(a). In contrast, with linear blending, parallax ispresented by blending images from two viewpoints
NTT Technical Review

Vol. 16 No. 12 Dec. 2018that have a disparity less than the fusion limit angle,
using relative luminance depending on the viewing
position. Intermediate viewpoints are perceived asinterpolated, so projectors are not needed for thoseviewpoints (Fig. 2(b)). This enables 3D images to be
displayed with smooth binocular and motion parallax, even when using projector intervals that are
wider than the spacing between viewers’ eyes.
3. Optimization of the tabletop
optical configuration
The spatially imaged iris plane screen [2] that uses
linear blending is composed of a reflective layer, afresnel lens layer, and a diffusion layer. Light fromthe projectors is reflected on the screen, and it focuses in the opposing space on the side of the screen. In
this way, the projector iris plane (the part equivalent
to the iris in the projector optical system) forms in thespace, and the viewer can only perceive the projected
image within that range. In tabletop configurations
studied earlier [4], a straight-line projector array wasused, with projectors in a straight horizontal line facing the spatially imaged iris plane. However, with thisstraight-line projector array, the farther the viewpoint
was horizontally from the center of the array, themore the projected viewpoint images were distorted,so simply extending this by combining linear projector arrays in four directions to create a full perimetersystem would not enable presentation of 3D images
to all viewpoints.
25

Feature Articles

ScreenProjector array
Luminance
distribution

Viewing
positions

Viewers(a) Conventional method

Projector array
Luminance
distribution

Viewing
positions(b) Proposed method (linear blending)Fig. 2. Comparison of conventional and proposed methods.

Projector array

ViewerSpatially imaged
iris plane screen

3D image

Spatially imaged
iris plane screen

Projector array

Viewer(a) Top view(b) Side viewFig. 3. Optical system with a circular projector array for 360-degree viewing.

To expand the viewing range to the entire perimeter, we designed an optical system with a circular
projector array as shown in Fig. 3; a side view of thepositional relationship between the viewer and theiris plane is shown in Fig. 4. The properties of the
diffusion layer cause the luminance distribution of
the iris plane to peak in the center and also result in
attenuation with distance from the center. The intermediate viewpoint images can be interpolated withNTT Technical Review

Vol. 16 No. 12 Dec. 2018the linear blending effect by optimizing the design ofthe diffusion angle θ2 of the diffusion layer and thehorizontal interval of projector placement, so thatadjacent viewpoint images are composed with luminance corresponding to the viewpoint position.

26

Feature Articles

Projector

Vertical distanceto projector: aVertical distanceto iris plane: b

Diffusionangle: 2

Incidenceangle: 1

Spatially imaged iris plane screen

Floor

Fig. 4. Positional relationship between viewer and iris plane.

Circular projector array (60 projectors)

Spatially imaged iris plane screen

Fig. 5. Prototype of 360-degree tabletop autostereoscopic 3D screen system.

4. Full-range 360-degree tabletop
autostereoscopic 3D screen system prototype
We implemented a prototype full 360-degree,
autostereoscopic 3D screen using the circular projecNTT Technical Review

Vol. 16 No. 12 Dec. 2018

tor array optical system described above (Fig. 5). The
spatially imaged iris plane screen had a diameter of
110 cm and was placed 70 cm from the floor. Additionally, 60 projectors with a resolution of 1280 x 800
pixels and brightness of 600 lm were placed at27

Feature Articles

Fig. 6. 3D screen photographed from five viewpoints.

6-degree intervals. The viewpoint images projectedfrom each projector were rendered with real-timesynchronization on 60 client personal computers(PCs), one for each projector. Each client PC shared
the same computer graphics space and rendered theimage from a virtual camera in the position of thecorresponding projector. Time synchronization of the
viewpoint images was done by sending UDP (userdatagram protocol) packets from a server PC withinformation for the 3D model position and the start ofanimation.
Photographs from five viewpoints around the prototype are shown in Fig. 6. The photographs show
how changes in motion parallax with changes inviewpoint are reproduced.

We plan to improve image quality in the future by
optimizing the optical characteristics of the screen.
We will also work toward future implementation ofhighly realistic viewing of sports competitions andother events around a table for venues such as sports
bars.
References
[1]
[2]

[3]

5. Future prospects
We have demonstrated the feasibility of a naturalautostereoscopic 3D screen technology with smooth,full-perimeter-viewpoint motion parallax using a prototype system. However, with the current prototype,there is some degradation in image quality caused bymulti-edges from neighboring projectors and someother projectors. We are studying how to reduce theadverse effect of multi-edges using visual effects [5];Motohiro Makiguchi

Researcher, Natural Communication Project,
NTT Service Evolution Laboratories.
He received a Master of Information Technology from the Graduate School of Information
Science and Technology, Hokkaido University,
in 2012. He joined NTT Service Evolution Laboratories in 2012. He has been researching 3D
display technology and virtual reality technology.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

[4]

[5]

S. Yoshida, “fVisiOn: 360-degree Viewable Glasses-free Tabletop 3D
Display Composed of Conical Screen and Modular Projector Arrays,”
Opt. Exp., Vol. 24, No. 12, pp. 13194–13203, 2016.
M. Makiguchi, T. Kawakami, M. Sasai, and H. Takada, “Smooth
Motion Parallax Glassless 3D Screen System Using Linear Blendingof Viewing Zones and Spatially Imaged Iris Plane,” SID (Society for
Information Display) 2017 Digest of Technical Papers, 61-2, Vol. 48,
No. 1, pp. 903–906, 2017.
S. Suyama, S. Ohtsuka, H. Takada, K. Uehira, and S. Sakai, “Apparent
3-D Image Perceived from Luminance-modulated Two 2-D Images
Displayed at Different Depths,” Vision Res., Vol. 44, No. 8, pp. 785–
793, 2004.
M. Makiguchi and H. Takada, “Proposal and Basic Study of a Tabletop Autostereoscopic 3D Screen System Using Optical Linear Blending,” Proc. of the 2018 Multimedia, Distributed, Cooperative, andMobile Symposium (DICOMO2018), pp. 509–513, Fukui, Japan,
July 2018.
M. Makiguchi, H. Takada, T. Fukiage, and S. Nishida, “Reducing
Image Quality Variation with Motion Parallax for Glassless 3D
Screens using Linear Blending Technology,” SID 2018 Digest ofTechnical Papers, 21-1, Vol. 49, No. 1, pp. 251–254, 2018.

Hideaki Takada

Senior Research Engineer, Supervisor, NTT
Service Evolution Laboratories.
He received an M.E. in information systems
from the University of Electro-Communications,Tokyo, in 1997. He received a D.S. in global
information and telecommunication studies fromWaseda University, Tokyo, in 2007. He joined
NTT in 1997. He is engaged in research of 3D
visual perception and 3D display technology. He
received an Achievement Award from the Institute of Electronics, Information and Communication Engineers in 2003 and a commendationfrom the Minister of Education, Culture, Sports,
Science and Technology Japan in 2006.

28

Feature Articles: Ultra-realistic Communication with
Kirari! Technology

Video Processing/Display Technology
for Reconstructing the Playing Fieldin Sports Viewing Service Using
VR/AR
Megumi Isogai, Kazuki Okami, Masaaki Matsumura,
Munekazu Date, Akio Kameda, Hajime Noto,and Hideaki Kimata
Abstract

Sports viewing services using virtual reality (VR) and augmented reality (AR) technology have been
introduced recently due to the development of sensor devices and video production/delivery technology.
NTT Media Intelligence Laboratories aims to provide a sports viewing service using VR/AR to display
reconstructed playing field videos without user operation. In this article, we present a video processing
technology to reconstruct the playing field for VR/AR displays and a three-dimensional display technology for representing a playing field three-dimensionally on a table.
Keywords: virtual reality/augmented reality, video processing technology, 3D display technology

1. Introduction
Advances in sensor devices and video production
and delivery technology have led to the introductionof sports viewing services using virtual reality (VR)and augmented reality (AR) technologies. NTT
DOCOMO demonstrated an AR sports viewing service for rugby games that displays player informationand video in a high visibility area through a user’ssmart glass [1]. KDDI has already launched a VR
sports watching service that provides VR views whileswitching among viewpoints from five cameras in a
baseball stadium [2].
However, neither AR nor VR can cover all of thegame scenes in a stadium because suitable scenes for
VR/AR change in response to the constantly varying
game state. To enable viewers to have a good gamewatching experience, it should be possible to change
VR/AR viewing modes without any need for user
operation. NTT Media Intelligence Laboratories thus
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

aims to give the crowd in the sports stadium an additional element of excitement by using VR/AR to display reconstructed playing field videos without user
operation. To achieve this, we have been studying
video processing technology to generate common
content for VR and AR, as well as video display technology to create a feeling of physical presence, as if
real objects were actually there.
In this article, we first give an overview of a sports
viewing system we developed that uses VR/AR. We
then present a three-dimensional (3D) reconstruction
technology we developed as a means for processing
videos to reconstruct the playing field for VR/AR
displays. It generates arbitrary viewpoint images
from places where cameras cannot be placed on theplaying field. We also describe our diminished reality
technology that focuses on particular players byremoving everything except the players from thevideo. Finally, we introduce a novel visually equivalent light field 3D display technology that we propose
29

Feature Articles

Playing field reconstruction
Scene analysis

Directional
techniqueselection

3D
reconstruction
technology

Diminishedrealitytechnology

Display

Sensing

Playing field

Fig. 1. Overview of sports viewing system using VR/AR technologies.

as a future AR method for representing a playing field
three-dimensionally on a table.
2. Overview of sports viewing system
Our system for watching sports in a stadium using
VR/AR technologies is shown in Fig. 1. It features
multiple cameras that surround the stadium and capture videos of the playing field. The videos are delivered to a server on a cloud. The server uses the videos
to analyze the event that occurred on the playing fieldand selects the videos to provide to the users from theanalysis results. The playing field is then reconstructed from the videos based on preprocessing results anddistributed to the user viewing devices.
3. 3D reconstruction technology forthe playing field
One of the video processing technologies used toreconstruct the playing field is 3D reconstruction
technology. It enables viewpoint images to be generated from places inside the playing field where camNTT Technical Review

Vol. 16 No. 12 Dec. 2018

eras cannot be installed. Since the spectators’ seats
are fixed in stadiums and arenas, the spectators canonly watch the event going on from a limited direction. They cannot see players if their seats are faraway from the playing field, and often cannot watch
the event from the viewpoint they would like in orderto get the best view of how the event is developing.
A method has been developed to generate arbitrary
viewpoint images from multiple cameras that surround the playing field. However, it requires theinstallation of a very large number of cameras, which
is difficult to do in a stadium. Also, the method cannot
generate high quality viewpoint images of what ishappening in front of the camera because of insufficient video resolution and because 3D information
cannot do estimations in occluded areas.
To address these problems, we propose a new formof 3D reconstruction technology based on computer
graphic (CG) characters. It estimates a player’s
motions from videos and applies the estimation
results to a preprepared high quality CG model of theplayer. With the notably improved CG quality formovies and games achieved in recent years, our30Feature Articles

Silhouette images

Proposed method
Estimation
targetCNN 1

Motion estimation result

CNN 2
Color images

deconv 4

deconv 3

deconv 2

deconv 1

res 15-16

res 9 -14

res 5 - 8

res 1 - 4

CNN 1

Out 1: depth images

conv

ResNet50

conv

Input: color images

Out 2: silhouette images
CNN: convolution neural network
conv: convolution

deconv: deconvolution
ResNet: Residual Network

Fig. 2. Overview of proposed method.

method generates high quality viewpoint images thatmake users feel as if they were watching scenes in thereal world.
To achieve this, it is necessary to estimate player
motions from videos. While conventional methods
have been proposed to estimate human motions fromvideos using deep learning [3], the pose estimation
accuracy they provide is substantially reduced when
the people being videoed are partly occluded by
objects. An important task is to improve robustness in
occluded areas, because in sports scenes many athletes are frequently intertwined in one scene and caneasily shield each other.
Thus, we propose the idea of estimating athletes’
motions not from color images directly but fromcolor images and silhouette images of people, including those in occluded areas (Fig. 2). By using silhouette images to limit the area for searching human
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

motions, our method achieves higher human motion
estimation accuracy than conventional ones can provide.
4. Diminished reality technology forthe playing field
Here, we describe the diminished reality technology we developed that removes unnecessary players
and objects from videos in order to focus on target
players.
In movie and cartoon scenes, only the persons to befocused on remain in the scene, and effects such as a
spotlight are added. We anticipate that applying thisto sports events held in stadiums and arenas will
ensure that increased attention is directed towards
players involved in decisive moments of the game,
which will make watching the game more exciting.
31

Feature Articles

Technical feature (1)Technical feature (2)Selecting cameras capturing backgroundimage without obstructing target region

Selecting the nearest camera with user
view direction and position

Background

TargetregionCamera 3
Camera 1

Camera 4

Camera 2
Selected camera image

Camera 2
Camera 3

User’s viewpoint image

BackgroundregionUser’s viewpoint image after
removing target region

TargetregionProjective transformation

Fig. 3. Technical features of proposed method.

Diminished reality (DR) technology removes
unnecessary target regions from video images andreconstructs the missing regions with backgroundimages. To apply this to sports events, DR technology
must reconstruct only target regions while considering the relationship between players in 3D space.
Furthermore, since spectators watch sports events invarious positions and postures, it needs to present
images in which the target region does not stand out
above all others for each spectator’s viewpoint.
To address these points, we propose a DR methodthat selects the optimal camera capturing the background of the target region from multiple cameras
installed in a stadium and overlays the transformed
image so that it matches that seen from the user viewpoint. This prevents the target region from becoming
overly conspicuous [4].
The technical features of our proposed method areshown in Fig. 3. The first feature is that the playing
field is treated as a multilayered plane, and 3D information is estimated on that basis, since a wide range
NTT Technical Review

Vol. 16 No. 12 Dec. 2018of depth is unnecessary to reconstruct the playing
field. This enables it to select the group of cameras
capturing the background of the target region withless calculation than that needed in methods estimating 3D information of the entire playing field.
The second feature is that the method automaticallydetermines the optimum camera—one that is near theuser’s location and thus catches the action from a
direction and position similar to that of the user. This
makes it possible to provide images that are naturalfrom a user’s viewpoint by simply applying projective transformation to selected camera images.
5. Visually equivalent light field
display technology
Finally, we introduce a novel visually equivalent
light field 3D display technology, which we propose
as a future AR method that three-dimensionally represents a playing field on a table. We believe thatshowing game highlights at the box office, lobby, or32

Feature Articles

Intermediate viewpoint image
generated by blending two
Blending ratio
Viewpoint
viewpoint images
A

Backlight

LCD
panel

B

Barrier

Image for viewpoint A
Image for viewpoint B
Image for viewpoint CC

LCD panel

Barrier
NTT R&D Forum 2018
Display size: 15.6 inches
Resolution: 4K (2160 x 3840)
Frame rate: 60 Hz
Content: CG (5 viewpoints)

15.6-inch prototype display

Visually equivalent light field display

Fig. 4. Overview of visually equivalent light field display.

some other place outside the playing venue will
increase people’s interest in certain players and teams
so they will come to see them again and again. To
provide spectators with a more attractive viewing
experience, it is important to show them the players
in a way that makes them feel as if the players wereactually there in front of them.
Objects such as players, fields, and balls emit rays
with different colors and brightness in each direction
by reflecting light or by emitting light themselves.
Fields generated with these rays, called light fields,
can be made to seem highly realistic by accurate ray
reproduction. However, to display different rays
depending on the direction, we allocate pixels inaccordance with the number of directions. For example, generating rays in 100 directions requires 100
times more pixels than needed for a 2D display withthe same resolution.
Human vision perceives objects from incident rays
entering the pupils of the eyes. This perception
involves not only acquiring simple images like pictures taken by a camera, but also using differences inrays between the left and right eyes and small temporal variations of rays due to viewing position changes
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

induced by fluctuations of posture or eye movement.
This perception seems to be high at first glance but isinsensitive to elements that are not necessary. Therefore, a visually equivalent light field should exist.
Though it is different from the light field of an actual
object, human vision perceives it as the same as thatof an actual object.
This is the concept of a visually equivalent lightfield display. With this technology, light rays to anintermediate viewpoint are interpolated with visualequivalence. The interpolation is a weighted averageof rays to discrete viewpoints that is optically createdin the display [5].
An overview of a visually equivalent light field
display is given in Fig. 4. As shown in the top left
corner, the display consists of an LCD (liquid crystal
display) panel, a stripe-pattern barrier, and a backlight, in order from the rear. When an observer’s
pupils are at viewpoint A, rays from the barrier spacing illuminate the pixels of the viewpoint A image
only, and that image can be seen. When an observer’s
pupils are at viewpoint B, the image for viewpoint B
can be seen. When an observer’s pupils are midway
between viewpoints A and B, the pixels for both33

Feature Articles

images A and B are partially illuminated, and aweighted average is achieved. Since the weights
depend on the distance between pupil position andviewpoints A and B, an intermediate viewpoint image
is perceived as expected. Because the image quality ishigh when a displayed object is close to the display
panel, we placed the panel horizontally as shown at
the bottom left in the figure. This made it possible todisplay and maintain high resolution, high quality
images in every corner of the court.
Reproducing the light field in this way enables thecorrect depth to be perceived even if the distance
between the left and right eyes is different (e.g., for an
adult and a child). We believe the efficacy of interpolation can not only improve pixel usage efficiency but
also improve feelings of object existence or reality.

selection of video effects and VR/AR displays thatwill accord with game situations. The development ofvideo processing technology will facilitate the reconstruction of playing fields and visually equivalent
light field 3D displays. This will enable us to contribute to the development of innovative ways to provide
people with a new and enhanced sports watchingexperience.
References
[1]

[2]
[3]

6. Future work
We have improved VR/AR technologies and expectthat sports viewing services using them will increase
as interest in sports increases with the approach of the
major international sports event in 2020. The researchon scene analysis technology NTT Media Intelligence Laboratories is conducting will facilitate the

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

[4]

[5]

Press release issued by NTT DOCOMO on December 21, 2017 (inJapanese).
https://www.nttdocomo.co.jp/binary/pdf/info/news_release/topics/
topics_171221_01.pdf
Press release issued by KDDI on July 25, 2018 (in Japanese).
http://news.kddi.com/kddi/corporate/newsrelease/2018/07/25/3281.
html
Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh, “Realtime Multi-person
2D Pose Estimation Using Part Affinity Fields,” Proc. of the 30th
IEEE Conference on Computer Vision and Pattern Recognition(CVPR 2017), pp. 1302–1310, Honolulu, HI, USA, July 2017.
M. Isogai, D. Ochi, and H. Kimata, “Diminished Reality Using Planebased Reconstruction Method,” The 17th International Meeting on
Information Display (IMID 2017), F38-4, 3DSA, Busan, Korea, Aug.
2017.
M. Date, D. Ochi, and H. Kimata, “Visually Equivalent Light Field
Flat Panel 3D Display,” Proc. of the 22nd Annual Conference of the
Virtual Reality Society of Japan, Tokushima, Japan, Sept. 2017 (inJapanese).

34

Feature Articles

Megumi Isogai

Senior Research Engineer, Visual Media Project, NTT Media Intelligence Laboratories.
She received a B.E., M.E., and Ph.D. in communication network engineering from Okayama
University in 2004, 2005, and 2010. She joined
NTT Cyber Space Laboratories (now, NTT
Media Intelligence Laboratories) in 2006 and has
been studying 3D video processing and highreality communication technology.

Kazuki Okami

Researcher, Visual Media Project, NTT MediaIntelligence Laboratories.
He received a B.E. and M.E. from Waseda
University, Tokyo, in 2012 and 2014. He joined
NTT Media Intelligence Laboratories in 2014,
where he has been engaged in research anddevelopment (R&D) of free-viewpoint video
synthesis.

Masaaki Matsumura

Research Engineer, Visual Media Project, NTT
Media Intelligence Laboratories.
He received a B.S. in design from Kyushu
Institute of Design, Fukuoka, in 2005, and an
M.E. in design from Kyushu University, Fukuoka, in 2007. He joined NTT Cyber Space Laboratories (now, NTT Media Intelligence Laboratories) in 2007 and has been researching a videocoding algorithm and GPU (graphics processingunit)-accelerated high-performance computing.

Akio Kameda

Research Engineer, Visual Media Project, NTT
Media Intelligence Laboratories.
He received a B.E and M.E in electrical engineering from Tokyo University of Science,
Chiba, in 1993 and 1995. In 1995, he joined NTT
Human Interface Laboratories and has been
engaged in R&D of video coding, communication, and distribution systems.

Hajime Noto

Senior Research Engineer, Visual Media Project, NTT Media Intelligence Laboratories.
He received a B.E. and M.E. in industrial engineering from Kansai University, Osaka, in 1997
and 1999. In 1999 he joined NTT Cyber Space
Laboratories (now, NTT Media Intelligence
Laboratories), where he researched 3D input
systems. He is a member of the Institute of Image
Information and Television Engineers and is currently involved in R&D of high-reality systems.

Hideaki Kimata

Senior Research Engineer, Supervisor, HighReality Visual Communication Group Leader,
NTT Media Intelligence Laboratories.
He received a B.E. and M.E. in applied physics
in 1993 and 1995 and a Ph.D. in electrical engineering in 2006 from Nagoya University, Aichi.
He joined NTT in 1995 and has been researchingand developing a video coding algorithm, visualcommunication systems, and machine learning.
He is a Chief Examiner of the Information Processing Society of Japan (IPSJ) Special Interest
Group on Audio Visual and Multimedia Information Processing.

Munekazu Date

Research Engineer, Visual Media Project, NTT
Media Intelligence Laboratories.
He received a B.S. in physics from Gakushuin
University, Tokyo, in 1990, an M.E. in applied
electronics from Tokyo Institute of Technology
in 1992, and a Ph.D. in chemistry from Tokyo
University of Science in 2003. He has been withNTT since 1992, where he has been researchingholographic optical devices using polymer/LC
composites and 3D displays. He joined NTT
COMWARE in 2010 and rejoined NTT in 2012.
Dr. Date is a member of the Institute of Electricaland Electronics Engineers, the Society for Information Display, the Japanese Liquid Crystal
Society, the Institute of Electrical Engineers ofJapan, and the Holographic Display Artists andEngineers Club.

NTT Technical Review

Vol. 16 No. 12 Dec. 201835

Global Standardization Activities

Forum Survey Activities by TTC(The Telecommunication Technology
Committee) Technology Research
Advisory Group
Hideyuki Iwata
Abstract

The Technology Research Advisory Group of The Telecommunication Technology Committee (TTC)of Japan surveys and analyzes trends in the standardization activities of Japanese and world forums in
the field of information and communications on a yearly basis. These efforts help to establish direction
in TTC standardization activities. The results of this survey and analysis can be accessed on the TTC
website. These results provide valuable information for individual companies that wish to grasp thelatest trends in standardization and evaluate the need to join a particular forum. This article reports inparticular on the activities of forums recently added to the survey and on forums focusing on current
technology topics.
Keywords: forum, consortium, standardization

1. Overview of Technology Research
Advisory Group
Forum surveys [1] at the Technology Research
Advisory Group of The Telecommunication Technology Committee (TTC) commenced in 1994, making
the survey scheduled for fiscal year (FY) 2018 the25th version. The forums targeted by these surveys
consist of organizations, voluntary bodies, andgroups aiming to promote or promulgate standardization, but exclude de jure standardization bodies suchas the International Telecommunication Union (ITU),
International Electrotechnical Commission (IEC),
International Organization for Standardization (ISO),and the ISO/IEC Joint Technical Committee 1(JTC1). The names of these bodies commonly
include the words forum, consortium, alliance, or
project. The survey team consists of eight TTCmember operator/vendor companies. Their activities
involve (1) determining the latest trends in the information and communications field and narrowing
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

down the forums to be surveyed, (2) collecting information on the selected forums such as membershipfees, target fields, and member makeup from publicly
released information on the web, and (3) examining
the temporal change in member makeup, the types ofactivities carried out, and other details from survey
data.
2. FY2017 surveyed forums
After the review was completed of the 58 forums
targeted in FY2016 version 23 of the forum survey, it
was decided to add 13 bodies and to terminate thesurveying of 4 bodies whose activities had ended orthat had merged with other bodies, or for which information could no longer be obtained.
The following provides a brief introduction to the
13 forums added for FY2017 version 24 of the forum
survey.

36Global Standardization Activities

2.1 5G Automotive Association (5GAA)
The 5GAA was founded in September 2016 to promote the cooperative development of connected car
services using fifth-generation mobile communications systems (5G). Although the founding membersof 5GAA consist mainly of European companies,
today a total of 48 companies from the United States,
Japan, China, Korea, and other countries around theworld are participating in 5GAA activities. These
companies include telecommunications equipmentvendors, telecommunications operators, automotive
manufacturers, and parts suppliers. The mission of
5GAA is to jointly develop, test, and promote communications solutions, support the standardization of
those solutions, and encourage their commercialization and global penetration.
2.2 OpenStack Foundation
The OpenStack Foundation is a non-profit organization established in September 2012. It aims to provide an open and vendor-agnostic cloud-computing
environment deployed as an infrastructure as a service (IaaS) and to develop and disseminate open
source software (OSS). The foundation consists ofindividual members and corporate members, the latter of which consist primarily of telecommunications
equipment vendors and telecommunications operators.
2.3 OpenAPI Initiative (OAI)The OAI is a Linux Foundation Project establishedon November 5, 2015 under the leadership of SmartBear Software as a consortium focused on promoting
a standard format for describing RESTful APIs (representational state transfer application programming
interfaces). As of August 2017, a total of 28 companies had become OAI members. The Swagger open
source API development tool developed by SmartBear and adopted by OAI for describing APIs isalready finding widespread use. The initial version of
the OpenAPI Version 3.0 Implementer’s Draft basedon the Swagger Specification was released on March
1, 2017. The OAI has adopted OpenStack as an IaaS
platform.
2.4 FIWARE Foundation
Based in Germany, the FIWARE Foundation is a
non-profit organization established as a private initiative to promote the FIWARE smart application platform developed by FI-PPP (Future Internet PublicPrivate Partnership), a project of the European
Union’s 7th Framework Programme. FIWARE is
NTT Technical Review

Vol. 16 No. 12 Dec. 2018attracting attention as a set of APIs that facilitate data
usage thanks to its key feature of treating data as context information and its handling of cross-industry
data in common. It is taking hold in Europe as a common platform for smart cities. As of September 2017,
The FIWARE Foundation had 28 member companies
located mainly in Europe, with one Japanese company participating.
2.5 Hypercat Alliance
The Hypercat Alliance is an organization established in September 2014 by 42 British companies
and municipalities using funds from the IoT Demonstrator Phase I Clusters of Innovate UK, an organization under the Department for Business, Innovationand Skills. Hypercat is a technology that is releasedin the form of an online catalog that tags the specifications needed to design Internet of Things (IoT)
devices with metadata and that enables automaticinterconnectivity between devices. Its purpose is toachieve high functionality, high-speed operation, andinteroperability in IoT devices by combining necessary objects and functions in the catalog to generateand share IoT devices. The Hypercat Alliance had 70
members as of September 2017, including one Japanese company.
2.6 OpenID Foundation (OIDF)
OIDF is a non-profit organization established inJune 2007 in Oregon, USA. In addition to formulating digital identifier (ID) standards, OIDF promotesinter-site ID linking, access to web services via
smartphone applications and API linking by using
OpenID Connect and other ID-authentication technologies and is engaged in the dissemination, protection, and training of OpenID technologies. As of
September 2017, 40 companies were OIDF members,
with some of the major members including OTT(over-the-top) telecommunications operators.
2.7 Spring Framework Project
The Spring Framework is an open-source application framework targeting Java platforms. It addresses
various problems common to conventional web
application frameworks such as weakness withrespect to specifications changes, difficulty of performing tests in program units, and difficulty ofmaintenance and reuse. The Spring Framework caninteract with existing protocols and products such as
HTTP (Hypertext Transfer Protocol), SOAP (Simple
Object Access Protocol), and Enterprise JavaBeans.
While there are only six main developers, the37

Global Standardization Activities

development is open, and anyone can participate. The
Japan Spring Framework User Group is a Spring
Framework community with a presence on social
networking services.
2.8 Trusted Computing Group (TCG)
TCG is an industry organization established in
April 2003 with headquarters in Oregon, USA, withthe aim of formulating standard technologies toimprove reliability and safety in the use of personal
computers. It formulated the specifications for the
Trusted Platform Module (TPM) security chipequipped with a hardware-based encryption function.
The most recent version of this specification is version 2.0 released in 2014. Almost all personal computers manufactured today are equipped with a TPM
chip to safely store and manage personal authentication information and encryption keys. The application of this technology to IoT devices has come intoview with the aim of enhancing terminal security.
2.9 Zero Outage Industry Standard Association
The Zero Outage Industry Standard Association
was established in November 2016 with the aim of
maximizing customer satisfaction and value by offering best practices and a standard framework to provide secure, high-reliability, and high-availability
information technology services and solutions. It had
12 member companies as of September 2017.
2.10 Wi-Fi Alliance
The Wi-Fi Alliance is an industry organization
founded in 2000 with the aim of promoting the spreadof wireless local area network products. It formulatesinterconnectivity test methods, certifies products, andengages in promotional activities toward the widespread use of the Wi-Fi brand. It had a total of 790
member companies as of August 2017. Although themain Wi-Fi standard is 802.11ac at present, the Wi-Fi
Alliance plans to provide products based on new standards such as 802.11ad, 802.11ax, and 802.11ah andon the Wi-Fi Agile Multiband that enables instantaneous access to the most optimal wireless environment.
2.11 Z-Wave Alliance
The Z-Wave Alliance was established to ensureinteroperability between devices and equipment
equipped with the Z-Wave wireless communications
protocol for smart homes. It had a total of 365 member companies as of August 2017. Z-Wave is a lowpower technology for smart homes centered on the
NTT Technical Review

Vol. 16 No. 12 Dec. 2018technology developed by the Danish company Zensys (acquired by the US company Sigma Designs in2009). It operates in the sub-gigahertz band. Devices
equipped with Z-Wave have found widespread use inEurope and other regions around the world.
2.12 EnOcean Alliance
EnOcean and other corporate groups establishedthe EnOcean Alliance in April 2008 to promote thespread of EnOcean technology and ensure interoperability between EnOcean products. EnOcean is anenergy harvesting technology developed by EnOcean
GmbH in Germany. It converts weak forms of energy
such as light, heat, and vibration into electric power
to perform low-power wireless communications.
EnOcean switches and sensor modules are being used
in offices, factories, and industrial equipment
throughout the world. It adopts the lower three layers
specified in ISO/IEC 14543-3-10 as wireless specifications and prescribes an application protocol called
EEP (EnOcean Equipment Profiles). As of August
2017, 430 companies had joined the EnOcean Alliance, and more than 1200 products had been certified.
2.13 Digital Stationery Consortium (DSC)
DSC was established in October 2016 in Delaware,
USA, to promote the use of digital ink and achieve
mutual compatibility between digital ink products. It
already provides SDKs (software development kits)
to partner companies for the WILL (Wacom Ink
Layer Language), which is a technology that canshare digital writing tools such as pen tablets in acloud environment and ensure mutual compatibility.
DSC seeks to achieve even closer collaborations
going forward. It has nine participating members
from Japan, Korea, China, and Europe including
Montblanc, a manufacturer of stationery goods, plus
equipment vendors and telecommunications operators.
3. Forums related to current technology topics
The following highlights forums classified by technology topics that have been attracting interest in
recent years, namely, software-defined networking(SDN)/network functions virtualization (NFV), big
data/IoT/machine to machine (M2M), smart cities,
5G, and intelligent transportation system (ITS)/connected cars. Note that a few of these forums wereintroduced in section 2.

38

Global Standardization Activities

3.1 SDN/NFV•	Open Networking Foundation (ONF) (established 2011): ONF released the second versionof Atrium, an SDN software distribution, in February 2016. ONF has made improvements toversion 1 of the Open Network Operating System(ONOS) and developed extensions to OpenDaylight platform. ONF merged with Open Networking Lab (ONOS/CORD (Central OfficeRe-architected as a Datacenter)) in October 2017and added a use-case discussion forum.
•	Broadband Forum (BBF) (established 1994): As
of December 2017, BBF had taken up a varietyof virtualization themes such as SDN accessnode architecture, a Yet Another Next Generation (YANG) model for passive optical networks,
and a YANG model for access nodes.
•	OpenDaylight Project (established 2013): This
group develops SDN controller software toachieve SDN/NFV, providing it as OSS. It
released Carbon as the sixth version of its SDN
platform in June 2017 and is developing extensions for IoT applications.
•	Open Platform for NFV (OPNFV) (established2014): OPNFV released Danube 1.0 as its fourth
OSS release in April 2017 and released Danube
2.0 and Danube 3.0 in May and July, respectively, of the same year. OPNFV Summit 2017 washeld in June 2017.
•	TM Forum (established 1988): The ZOOM
(Zero-touch Orchestration, Operations and Management) project team is collaborating with the
NFV Industry Specification Group of the European Telecommunications Standards Institute(ETSI) to develop an NFV management andimplementation model.
3.2 Big data/IoT/M2M•	Object Management Group (OMG) (established1989): OMG announced a supplementary provision to the Industrial Internet Connectivity
Framework standard of IIoT (Industrial Internetof Things) for application to the energy field at
the Work in Oil and Gas event in September
2017. OMG has held two webinars (web-based
seminars).
•	Industrial Internet Consortium (IIC) (established2014): IIC has been quite active in testbed development with 25 testbeds in total as of September
2017. It holds many events and issues many
types of documents including white papers.
•	Open Connectivity Foundation (OCF) (estabNTT Technical Review

Vol. 16 No. 12 Dec. 2018lished 2016): The parent organization of OCF is
the Open Interconnect Consortium, which previously merged with the Universal Plug and Play
Forum. In October 2016, OCF and the AllSeen
Alliance merged under the OCF name, and it was
decided to make IoTivity and Alljoyn frameworks compatible. In June 2017, OCF announced
that six of its IoT specifications were put to avote at ISO/IEC JTC1.
•	THREAD Group (established 2014): Since therelease of the Thread Wireless Networking Protocol version 1.1 in the summer of 2016, The
THREAD Group has been active in equipment
certification and interconnectivity testing. Seamless connectivity was exhibited by 17 companies
at CES (Consumer Electronics Show) 2017, andthe group completed its first certification of version 1.1 products.
•	OpenFog Consortium (established 2015): The
OpenFog Consortium collaborates with the IoT
Acceleration Consortium in Japan to develop
technologies and testbeds and promote standardization. On September 25, 2017, it announced
that it had concluded a memorandum of understanding with ETSI to cooperate in the development of fog and edge applications.
•	Hypercat Alliance (established 2014): The
Hypercat Alliance has presented use cases in 14
fields including smart buildings and smart energy. It released “PAS 212:2016 Automatic
resource discovery for the Internet of Things Specification” as a British standard in August
2016.
3.3 Smart cities•	Japan Smart Community Alliance (JSCA)(established 2010): JSCA held the Smart Community Summit in June 2017 and conductedactive discussions on issues surrounding the dissemination of solar power generation and entry
into emerging energy markets.
•	Smart Grid Interoperability Panel (SGIP) (established 2009): SGIP is an organization that supports the National Institute of Standards andTechnology in the development of smart grid
standards. SGIP merged with SEPA (Smart Electric Power Alliance) in April 2017, becoming
part of its organization.
•	ECHONET Consortium (established 1997): As
of January 2018, the ECHONET Consortium
had certified 530 devices under the ECHONET
Lite specification, 296 devices under the AIF
39

Global Standardization Activities(application communication interface) specification, and 19 devices under the ECHONET specification. It held the 1st Plugfest event in July
2018.
•	OpenADR Alliance (established 2010): This
organization released the OpenADR 2.0 Program
Guide for the OpenADR smart grid standard in
February 2016. As of May 2017, it had certified
120 devices.
•	FIWARE Foundation (established 2011): The
FIWARE Foundation has released OSS and APIs
as a common platform for smart cities. It has also
released DSEs (Domain-Specific Enablers) as
field-specific sets of resources.
3.4 5G•	NGMN (Next Generation Mobile Networks)
Alliance (established 2006): This group beganby conducting studies of Super 3G and LongTerm Evolution (LTE) in 2006 but is presently
focused on 5G. It published the NGMN 5G
White Paper in March 2015. The NGMN Alliance has published 12 technical documents todate.
•	5G Automotive Association (5GAA) (established 2016): 5GAA has published a variety ofwhite papers including “Edge computing foradvanced automotive communications” and “Anassessment of LTE-V2X (PC5) and 802.11p
direct communications technologies for improved
road safety in the EU” in December 2017.
3.5 ITS/connected cars
•	ITS Forum (established 1991): ITS Forum has
published many technical guidelines such as“700 MHz Band Intelligent Transport Systems Experimental Guideline for Roadside-to-roadside Communications Ver. 1.1,” “700 MHz Band
Intelligent Transport Systems - Experimental
Guideline for Inter-vehicle Communication
Messages Ver. 1.1,” and “700 MHz Band Intelligent Transport Systems - Test Items and Conditions For Mobile Station Interoperability Verifi-

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

cation Guideline Ver. 1.2” in October 2017.
•	ITS America (established 1991): ITS America
published a white paper titled “The Impact of a
Vehicle-to-Vehicle Communications Rulemaking on Growth in the DSRC Automotive Aftermarket” in October 2016.
4. Trends in number of members
Twelve forums showed an increase in members
from FY2016 to FY2017. A case in point is the LoRa
Alliance, which has already expanded its membership to more than 400 member companies despite
being established less than three years ago. Its membership increased in FY2017 by more than 70% overthe previous year, reflecting a growth trend. In addition, Hyperleger, which promotes standardization inblockchain technology, also experienced a jump in
membership of more than 70% over the previous
year. In terms of medium-term trends, 10 forums
showed an increase in members for two consecutive
years from 2015, and 6 forums—OPEN Alliance
SIG, Wi-SUN Alliance, The Open Group, IIC,OpenADR Alliance, and ECHONET Consortium—
showed an increase in members for three or moreconsecutive years. These forums tend to be in fields
such as smart cities, IoT, cloud computing, and connected cars, reflecting heightened interest in thosetechnologies.
5. Future plans
The release of version 25 of the forum survey reportis planned for the end of FY2018.
Reference[1]

Forum Survey Report (Japanese version only), http://www.ttc.or.jp/e/inv/

Trademark notes

All brand names, product names, and company/organization names thatappear in this article are trademarks or registered trademarks of theirrespective owners.

40

Global Standardization Activities

Hideyuki Iwata

Senior Research Engineer, Supervisor,
Research and Development Planning Department, NTT.
He received a Ph.D. in electrical engineeringfrom Yamagata University in 2011. From 1993 to
2000, he conducted research on high-density andaerial optical fiber cables at NTT Access Network Service Systems Laboratories. Since 2000,
he has been responsible for standardization strategy planning for NTT research and development.
He has been a delegate of IEC Subcommittee
86A (optical fiber and cable) since 1998 and of
the ITU-T (Telecommunication Standardization
Sector) Telecommunication Standardization
Advisory Group since 2003. He is a vice-chair of
the Expert Group on Bridging the Standardization Gap in the Asia-Pacific Telecommunity
Standardization Program Forum. In 2004, hereceived an award from the IEC Activities Promotion Committee of Japan for his contributions
to standardization work in IEC.

NTT Technical Review

Vol. 16 No. 12 Dec. 201841

Practical Field Information about
Telecommunication Technologies
Practical Field Information about Telecommunication Technologies

Initiatives Targeting Snow Damage toCommunication Facilities
Technical Assistance and Support Center, NTT EAST
Abstract

Many communication facilities installed outdoors are continuously exposed to harsh weather including snowfall. This article presents a countermeasure to prevent the bands that secure signboards to utility poles from breaking and a countermeasure to prevent snow from accumulating on overhead equipment. This is the forty-ninth article in a series on telecommunication technologies.
Keywords: snow damage, utility pole, closure

1. Introduction
Many communication facilities installed outdoors
are continuously exposed to harsh weather including
wind, rain, and snowfall. Consequently, many countermeasures to prevent damage caused by such conditions have been implemented. Moreover, in the winter of 2017/18, snowfall in Japan was heavier thanthat in normal years, and damage due to the heavy
snowfall was reported, particularly in northern Japan.
Accordingly, concerns about snow damage haveincreased.
In regions with heavy snowfall, NTT facilities arealso affected by snow damage, which leads to failures
and accidents. The Technical Assistance and Support
Center is working on countermeasures to prevent the
bands that secure signboards to utility poles frombreaking and countermeasures to prevent snow fromaccumulating on overhead equipment. Both of these
phenomena contribute to increases in repair operations and accidents in snowy areas.
1.1 Breakage of bands
Utility pole signboards are installed on utility poles
along roads across the country for the purpose of
preventing accidents such as vehicles accidentally
hitting the utility poles (Fig. 1). The signboards are
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

affixed to the poles with vinyl bands. A common phenomenon that occurs is that one or more of the vinyl
bands breaks due to the load of accumulated snow,
and the band(s) and/or signboard falls off as a result.
This band breakage occurs over a wide region andthus requires a large operation to replace the broken
bands. Moreover, if the bands and/or signboards falloff, they might affect passing pedestrians or vehicles,
possibly leading to accidents.
1.2 Snow accumulation
In snowy areas, snow accumulates on overheadequipment such as cable closures and cable intersections (Fig. 2). If the snow continues to fall for several
days, repeated melting and resolidification will generate lumps of ice, and the risk of those ice lumps
falling is a very dangerous one that may lead to traffic
accidents resulting in injury or property damage(Fig. 3). Accordingly, maintenance to remove accumulated snow is carried out periodically to prevent
such accidents. When workers clear snow in this kindof operation, they must use an aerial platform, the useof which is a major operation.
Details about certain countermeasures against snow
damage to utility poles devised by the Technical
Assistance and Support Center have previously been
introduced [1]. In this issue, two of our initiatives—
42

Practical Field Information about Telecommunication Technologies

Fig. 1. Breakage of vinyl bands securing utility pole signboards.

Fig. 2. Snow accretion on overhead equipment.

2. Measures to prevent breakage of vinyl bands
securing utility pole signboards
In this section, we describe the problem in detailand explain the proposed countermeasures and theresults of field trials.

Damaged car windscreen due tofalling snow

Fig. 3. Property damage accident caused by snow fallingfrom overhead equipment.

namely, a countermeasure to prevent breakage of thevinyl bands securing utility pole signboards and acountermeasure to prevent snow from accumulatingon overhead equipment—are introduced.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

2.1 Mechanism by which bands break
We conducted a field trial to confirm the mechanism by which the bands break. In this trial, two patterns of load application by snow accumulation wereassumed: (i) the case in which the bands break due toapplication of a load to the utility pole signboard and(ii) the case in which a load is applied only to theband itself, causing it to break. The aim of the trial
was to verify by which pattern (or both) the bands
break. During the trial, lasting from December 2015
to April 2016, a countermeasure was applied thatinvolved closing the gap between the utility pole andthe utility pole signboard (Fig. 4). This was done toverify the case in which the band broke due to snow
43

Practical Field Information about Telecommunication Technologies

Closing the gap between thesignboard and the utility polewith self-adhesive tape

Snow cover in layers

Fig. 5. Layers of accumulated snow in trial site.
Fig. 4. Verification of state of band breakage.

Sedimentation pressure

Twisting causes theband to distort outwards.
(1) Icing on the band(2) Twisting of the band

(3) Breakage of the band

Fig. 6. Mechanism by which the band breaks.

loading on the band itself. A countermeasure involving protecting the band with sealants was also applied
to verify the case in which the band broke due to snow
loading on the utility pole signboard. The trial siteused a utility pole placed in a residential area locatedat the foot of a mountain in a region that receives
heavy snowfall. The amount of snowfall in the year of
the trial (2016) was about 2.5 m, and the trial site wasa location where snowfall and ploughed snow regularly pile up significantly (Fig. 5).
We checked the state of the signboard and band
during snowfall and after snow thawing. In the casewhere the gap between the utility pole and the utility
pole signboard was closed by attaching tape, icing
and twisting of the band’s fastener and the left/rightends of the band still occurred. These phenomena
caused the band to break. Since breaking, sagging,and twisting of the band was confirmed even if thegap between the utility pole and the signboard wasclosed, it was assumed that sedimentation pressure(pressure caused when snow accumulates and settles
in one place) acts directly on the band. The mechanism leading to the band breaking was considered tohave three stages: (1) ice building up on the band, (2)generation of sedimentation pressure causing twisting of the band so it distorts outward, and (3) breakNTT Technical Review

Vol. 16 No. 12 Dec. 2018

age of the band (Fig. 6).
In contrast, in the case where the band was protected with sealants, snow only accumulated on thesignboard and the band did not break.
2.2 Proposed countermeasures and field trials
It became clear that the load is directly applied tothe band by sedimentation pressure, so three proposed countermeasures against band breakage(Table 1) were tested in trials from December 2016
to April 2017 (Fig. 7).
When proposed countermeasure (1) was applied,certain effects of reducing sedimentation pressure
were confirmed, and no band breakage was observedin any of the five samples. However, loosening andtwisting of the band as well as peeling of part of thestandard sealant were observed. This result is thought
to be due to the fact that working with standard sealant is somewhat difficult, and thus it is not easy toapply it evenly. It is therefore conceivable that in the
future, the standard sealant will peel off, and the band
will subsequently break.
In the case of proposed countermeasure (2),although no twisting or breaking of the band wasseen, the band did loosen. It is therefore also conceivable that the band might break in the future.
44

Practical Field Information about Telecommunication Technologies

Table 1. Proposed countermeasures against loading of the band due to snowfall.
Proposedcountermeasure

Name

(1)

Standard sealantReduce the effect of sedimentation pressure by protecting the entire band.

(2)

Power band

Improve durability and resistance to sedimentation pressure by using a band
resistant to twisting.

(3)

Signboard with adhesive backng*

Principle

Remove the need for a band by bonding signboard directly to the utility pole withadhesive sheet.

*A product satisfying JIS Z 9117 (Japanese Industrial Standard for Reflectivity) for visibility was used for the signboard with adhesive
backing.

Fastener

Fig. 7. Countermeasures: (1) standard sealant; (2) power band; and (3) reflective sheets (signboard with adhesive backing).

Slight peeling at corners of sheet
Four corners
cut off and color
changed (fromwhite to yellow).

Installing multiple
signboards andchanging colors(white to yellow) to
improve visibility

Fig. 8. Result of applying proposed countermeasure (3).
Fig. 9. Additional trial of signboard with adhesive backing.

With proposed countermeasure (3), some of thetrial samples showed slight peeling at the four cornersof the sheet (Fig. 8), but the adhesiveness of the sheet
was found to be basically satisfactory, and it was easy
to apply the sheet evenly, with the correct alignment
and without puckering or air bubbles.
Since no problem concerning adhesiveness wasfound, and the sheet was easy to apply, an additionalverification of countermeasure (3) was conducted.
We are currently evaluating whether cutting off thefour corners of the sheet before applying it will prevent the peeling at the corners. In addition, to achieve
stable sheet visibility and improve adhesion quality,
NTT Technical Review

Vol. 16 No. 12 Dec. 2018

an on-site trial of an improved version of this countermeasure (3), namely, a signboard with adhesive backing is continuing. In this trial, in addition to the adhesiveness and visibility of the signboard, the size andcolor suitable for installation are being evaluated(Fig. 9). Similar trials were also carried out on steelpipe columns.

45

Practical Field Information about Telecommunication Technologies

Mar. 24, 2017

Mar. 24, 2017

Fig. 10. Test to prevent snow accumulation on closures.

3. Countermeasure to prevent snow
accumulation on overhead equipment

factors in order to verify the effectiveness of thiscountermeasure.

It is difficult to avoid snow accumulation on overhead equipment in snowy areas, but it is thought that
if the snow falls off harmlessly before it turns into alump of ice, accidental damage can be prevented.
Therefore, an initial test was carried out to evaluate amethod of prompting snow to fall from overheadequipment after it accumulates during snowfall. A
water-repellent sheet was attached to the top of acable closure, and the closure was observed to determine if the sheet caused the snow to fall off rather
than to accumulate (Fig. 10). The test result revealed
that by attaching the water-repellent sheet to the topof the closure, the snow that accumulates on that closure was eliminated in a short time (seen in the photoon the right), so that snow was prevented frombecoming a lump of ice. Investigations will continue
to monitor aging deterioration of the sheet and other4. Concluding remarks

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

These trials were conducted with the assistance of
the NTT EAST-Tohoku, Aomori Branch, Facilities
Division, Aomori Service Center; NTT EASTTohoku, Yamagata Branch, Facilities Division,
Yamagata Service Center; and NTT EAST-Kanshin­etsu, Nagano Branch, Facilities Division. More information will be reported as it becomes available.
Reference[1]

Technical Assistance and Support Center, NTT EAST, “Snow-stopper
Pole as a Countermeasure for Preventing Snow Damage to Utility
Poles,” NTT Technical Review, Vol. 16, No. 10, pp. 83–86, 2018.
https://www.ntt-review.jp/archive/ntttechnical.php?contents=
ntr201810pf1.html

46External Awards
Eco-ICT Award 2017
Winner: Hideki Maeda, Daisaku Shimazaki, Masaaki Inami,
Takeshi Seki, Takafumi Fukatani, Masatoshi Namiki, and Yuji
Minato, NTT Network Service Systems Laboratories
Date: February 21, 2018
Organization: Telecommunications Carriers Association

For “Evanescent Coupling between an Optical Microbottle and an
Electromechanical Resonator.”
Published as: M. Asano, R. Ohta, T. Yamamoto, H. Okamoto, and H.
Yamaguchi, “Evanescent Coupling between an Optical Microbottle
and an Electromechanical Resonator,” Proc. of the 65th JSAP Spring
Meeting, 18p-C301-5, Tokyo, Japan, Mar. 2018.

For the development of optical packet multifunctional system“100G-PTS.”

Young Scientist Presentation Award
Winner: Masaaki Ono, NTT Basic Research Laboratories
Date: September 18, 2018
Organization: JSAP

Excellent Technology Award
Winner: Motohiro Makiguchi and Hideaki Takada, NTT Service
Evolution Laboratories; Tohru Kawakami and Mutsumi Sasai,
Tohoku University
Date: June 22, 2018
Organization: The Institute of Image Electronics Engineers of Japan(IIEEJ)
For the development of “glassless motion parallax 3D screen system using visual perception.”
1906 Award
Winner: Yohei Sakamaki, NTT Device Technology Laboratories
Date: July 31, 2018
Organization: The International Electrotechnical Commission(IEC)

For “Enhancement of Graphene Absorption in Plasmonic Waveguides with 30 x 20 nm2 Core.”
Published as: M. Ono, M. Hata, K. Nozaki, H. Sumikura, and M.
Notomi, “Enhancement of Graphene Absorption in Plasmonic Waveguides with 30 x 20 nm2 Core,” Proc. of the 65th JSAP Spring Meeting, 20p-C301-13, Tokyo, Japan, Mar. 2018.
JSAP Poster Award
Winner: Makoto Takamura, Norio Kumada, Shengnan Wang,
Kazuhide Kumakura, and Yoshitaka Taniyasu, NTT Basic ResearchLaboratories
Date: September 18, 2018
Organization: JSAP

For his achievement and contribution as a key project leader in IEC
Subcommittee 86C/Working Group 5, with very good results indeveloping IEC 62343-3-4, a performance specification template forthe multi-cast optical switch—an important component in reconfigurable optical add-drop multiplexer (ROADM) systems.

For “Tuning of Plasmonic Reflection in Graphene by Carrier Density Modulation.”
Published as: M. Takamura, N. Kumada, S. Wang, K. Kumakura,
and Y. Taniyasu, “Tuning of Plasmonic Reflection in Graphene by
Carrier Density Modulation,” Proc. of the 79th JSAP Autumn Meeting, 18p-PB3-58, Nagoya, Aichi, Japan, Sept. 2018.

Communications Society: Distinguished Contributions
Award
Winner: Hideki Maeda, NTT Network Service Systems Laboratories
Date: September 12, 2018
Organization: The Institute of Electronics, Information and Communication Engineers (IEICE)Best Paper Award
Winner: Hiroshi Ishikawa, Salah Ibrahim, Tatsushi Nakahara,
Hiroki Sugiyama, and Toshikazu Hashimoto, NTT Device Technology Laboratories
Date: September 20, 2018
Organization: International Conference on Photonics in Switching
and Computing 2018 (PSC 2018)For his contributions as secretary of the Technical Committee on
Network Systems.

For “A Novel Optoelectronic Parallel-to-Serial Converter for 100Gbps Optical Packets.”
Published as: H. Ishikawa, S. Ibrahim, T. Nakahara, H. Sugiyama,
and T. Hashimoto, “A Novel Optoelectronic Parallel-to-Serial Converter for 100-Gbps Optical Packets,” PSC 2018, Th3B.3, Limassol,
Cyprus, Sept. 2018.

Young Scientist Presentation Award
Winner: Motoki Asano, NTT Basic Research Laboratories
Date: September 18, 2018
Organization: The Japan Society of Applied Physics (JSAP)NTT Technical Review

Vol. 16 No. 12 Dec. 201847

Papers Published in Technical Journals andConference Proceedings
Wideband Slow Short-pulse Propagation in One-thousand
Slantingly Coupled L3 Photonic Crystal Nanocavities
E. Kuramochi, N. Matsuda, K. Nozaki, A. H. K. Park, H. Takesue,and M. Notomi
Optics Express, Vol. 26, No. 8, pp. 9552–9564, April 2018.
Coupled cavities have been used previously to realize on-chip lowdispersion slow-light waveguides, but the bandwidth was usually
narrower than 10 nm and the total length was much shorter than 1
mm. Here we report long (0.05–2.5 mm) slow-light coupled cavity
waveguides formed by using 50, 200, and 1000 L3 photonic crystal
nanocavities with an optical volume smaller than (λ/n)3, slanted fromΓ-K orientation. We demonstrate experimentally the formation of asingle-mode wideband coupled cavity mode with a bandwidth of up
to 32 nm (4 THz) in telecom C-band, generated from the ultra-narrow-band (~300 MHz) fundamental mode of each L3 nanocavity, bycontrolling the cavity array orientation. Thanks to the ultrahigh-Q
nanocavity design, coupled cavity waveguides longer than 1 mm
exhibited low loss and allowed time-of-flight dispersion measurement over a bandwidth up to 22 nm by propagating a short pulse over1000 coupled L3 nanocavities. The highly dense slanted array of L3
nanocavity demonstrated unprecedentedly high cavity coupling
among the nanocavities. The scheme we describe provides controllable planar dispersion-managed waveguides as an alternative to
W1-based waveguides on a photonic crystal chip.

Experimental Investigation of Laser Linewidth Toleranceof 32-GBaud DP-256QAM Optical Coherent System
T. Sasai, F. Hamaoka, A. Matsushita, M. Nakamura, H. Kawakami,
and Y. Kisaka
Proc. of the 23rd OptoElectronics and Communications Conference (OECC 2018), 6D2-1, Jeju, Korea, July 2018.
We experimentally investigate the laser linewidth tolerance of
32-GBaud DP-256QAM system for back-to-back and 50-km transmission. Results show that laser linewidth (0.1–550 kHz) has only aslight effect on the signal quality even for higher-order QAM.

Room Temperature Continuous-wave Nanolaser Diode
Utilized by Ultrahigh-Q Few-cell Photonic Crystal Nanocavities
E. Kuramochi, H. Duprez, J. Kim, M. Takiguchi, K. Takeda, T.
Fujii, K. Nozaki, A. Shinya, H. Sumikura, H. Taniyama, S. Matsuo,
and M. Notomi
Optics Express, Vol. 26, No. 20, pp. 26598–26617, September
2018.
Few-cell point-defect photonic crystal (PhC) nanocavities (such as
LX and H1 type cavities), have several unique characteristics including an ultrasmall mode volume (Vm), a small device footprint advantageous for dense integration, and a large mode spacing advantageousfor high spontaneous-emission coupling coefficient (β), which arepromising for energy-efficient densely integratable on-chip laser
light sources enhanced by the cavity quantum electrodynamics(QED) effect. To achieve this goal, a high quality factor (Q) is essential, but conventional few-cell point-defect cavities do not have asufficiently high Q. Here we adopt a series of modified designs of LX

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

cavities with a buried heterostructure (BH) multi-quantum-well(MQW) active region that can achieve a high Q while maintaining
their original advantages and fabricate current-injection laser devices. We have successfully observed continuous-wave (CW) lasing in
InP-based L1, L2, L3 and L5 PhC nanocavities at 23°C with a DC
current injection lower than 10 μA and a bias voltage lower than 0.9
V. The active volume is ultrasmall while maintaining a sufficiently
high confinement factor, which is as low as ~10−15 cm3 for a singlecell (L1) nanocavity. This is the first room-temperature currentinjection CW lasing from any type of few-cell point-defect PhC
nanocavities (LX or H1 types). Our report marks an important step
towards realizing a nanolaser diode with a high cavity-QED effect,
which is promising for use with on-chip densely integrated laser
sources in photonic networks-on-chip combined with CMOS (complementary metal-oxide semiconductor) processors.

Adaptive Compensation for SOA-induced Nonlinear Distortion with Training-based Estimation of SOA Device
Parameters
F. Hamaoka, S. Okamoto, M. Nakamura, A. Matsushita, and Y.
KisakaProc. of the 44th European Conference on Optical Communication(ECOC 2018), We2.31, Rome, Italy, September 2018.
The semiconductor optical amplifier (SOA)-induced nonlinear
distortion is compensated solving a time-evolving equation for thegain exponent using SOA device parameters estimated by our proposed training-based technique. In experiments for 32-GBaud PDM16QAM (polarization division multiplexed 16 quadrature amplitude
modulation), a maximum 2.52-dB Q-factor improvement was
obtained by adaptive nonlinear compensation.

150.3-Tb/s Ultra-wideband (S, C, and L bands) Singlemode Fibre Transmission over 40-km Using >519Gb/s/λ
PDM-128QAM Signals
F. Hamaoka, K. Minoguchi, T. Sasai, A. Matsushita, M. Nakamura, S. Okamoto, E. Yamazaki, and Y. Kisaka
Proc. of ECOC 2018, Mo4G. 1, Rome, Italy, September 2018.
A record capacity of 150.3-Tb/s SMF (single-mode fibre) transmission over 40 km is achieved with 13.6-THz ultra-wideband WDM(wavelength division multiplexed) 45-GBaud PDM-128QAM signals with 11.05-b/s/Hz spectral efficiency by using our proposed
optimization scheme of fibre input powers considering the nonlinear
interaction by stimulated Raman scattering.

Novel ODU4 Path Protection without Bit Disruption for
Low Latency and Resilient Core/Metro Network
K. Kitamura, F. Inuzuka, T. Tanaka, and A. Hirano
Proc. of ECOC 2018, Mo4D. 4, Rome, Italy, September 2018.
We demonstrate novel 100Gbit/s ODU4 path switching without bit
disruption at a relative delay difference up to 400 km experimentally.
We confirmed ITU-T (International Telecommunication Union Telecommunication Standardization Sector) standard-compliant48

Papers Published in Technical Journals and Conference Proceedings

jitter characteristics and applicability for core and metro networks byevaluating the accommodation rate in a European network.

Underground Infrastructure Management System Using
Internet of Things Wireless Transmission Technology
Y. Yamaguchi, Y. Fujino, H. Katsuda, M. Nakano, H. Fukumoto, S.
Teruhi, K. Akabane, and S. Yoshino
IEICE Transactions on Electronics, Vol. E101-C, No. 10, pp.
727–733, October 2018.
This paper presents a water leakage monitoring system that gathers
acoustic data of water pipes using wireless communication technology and identifies the sound of water leakage using machine leaning
technology. To collect acoustic data effectively, this system combines
three types of data-collection methods: drive-by, walk-by, and static.
To design this system, it is important to ascertain the wireless communication distance that can be achieved with sensors installed in abasement. This paper also reports on radio propagation from underground manholes made from reinforced concrete and resin concrete
in residential and commercial areas using the 920 MHz band. We
reveal that it is possible to design a practical system that uses radio
communication from underground sensors.

NTT Technical Review

Vol. 16 No. 12 Dec. 2018

Incremental Environmental Monitoring for Revealing the
Ecology of Endangered Fish
Y. Shirai, Y. Kishino, S. Mizutani, Y. Yanagisawa, T. Suyama, T.
Otsuka, T. Kitagawa, and F. Naya
IEICE Transactions on Communications, Vol. E101-B, No. 10, pp.
2070–2082, October 2018.
This paper proposes a novel environmental monitoring strategy,
incremental environmental monitoring, that enables scientists toreveal the ecology of wild animals in the field. We applied this strategy to the habitat of endangered freshwater fish. Specifically, we
designed and implemented a network-based system using distributed
sensors to continuously monitor and record the habitat of endangered
fish. Moreover, we developed a set of analytical tools to exploit avariety of sensor data, including environmental time-series data suchas the amount of dissolved oxygen, as well as underwater video capturing the interaction of fish and their environment. We also describe
the current state of monitoring the behavior and habitat of endangered fish and discuss solutions for making such environmental
monitoring more efficient in the field.

49