NTT Communication Science Laboratories

OPEN HOUSE 2019
30Thr 31Fri

May 12:00～17:30

9:30～16:00

Venue: NTT Keihanna BuildingScience of Machine Learning
01 Learning and finding congestion-free routes～Online shortest path algorithm with binary decision diagrams～

02 Efficient and comfortable AC control by AI～Environment reproduction and control optimization system～

03 Recover urban people flow from population data
～People flow estimation from spatiotemporal population data～

04 Improving the accuracy of deep learning～Larger capacity output function for deep learning～

05 Which is cause? Which is effect? Learn from data!
～Causal inference in time series via supervised learning～

06 Forecasting future data for unobserved locations～Tensor factorization for spatio-temporal data analysis～

07 Search suitable for various viewpoints～“Pitarie”: Picture book search with graph index based search～

Science of Communication and Computation
08 We can transmit messages to the efficiency limit～Error correcting code achieving the Shannon limit～

09 New secrets threaten past secrets～Vulnerability assessment of quantum secret sharing～

10 Analyzing the discourse structure behind the text
～Hierarchical top-down RST parsing based on neural networks～

11 When children begin to understand hiragana～Emergent literacy development in Japanese～

12 Measuring emotional response and emotion sharing～Quantitative assessment of empathic communication～

13 Touch, enhance, and measure the empathy in crowd～Towards tactile enhanced crowd empathetic communication～

14 Robot understands events in your story～Chat-oriented dialogue system based on event understanding～Science of Media Information
15 Voice command and speech communication in car～World’s best voice capture and recognition technologies～

16 Learning speech recognition from small paired data
～Semi-supervised end-to-end training with text-to-speech～

17 Who spoke when & what? How many people were there?
～All-neural source separation, counting and diarization model～

18 Changing your voice and speaking style～Voice and prosody conversion with sequence-to-sequence model ～19 Face-to-voice conversion and voice-to-face conversion～Crossmodal voice conversion with deep generative models～

20 Learning unknown objects from speech and vision～Crossmodal audio-visual concept discovery～

21 Neural audio captioning～Generating text describing non-speech audio～

22 Recognizing types and shapes of objects from sound～Crossmodal audio-visual analysis for scene understanding～

Science of Human
23 Speech of chirping birds, music of bubbling water～Sound texture conversion with an auditory model～

24 Danswing papers～An illusion to give motion impressions to papers～

25 Measuring visual abilities in a delightful manner～Self eye-check system using video games and tablet PCs～

26 How do winners control their mental states?
～Physiological states and sports performance in real games～

27 Split-second brain function at baseball hitting～Instantaneous cooperation between vision and action～

28 Designing technologies for mindful inclusion～How sharing caregiving data affects family communication～

29 Real-world motion that the body sees～Distinct visuomotor control revealed by natural statistics～

30 Creating a walking sensation for the seated～A sensation of pseudo-walking expands peripersonal space～

01

Learning and finding congestion-free routes- Online shortest path algorithm with binary decision diagrams -Abstract
We consider adaptively finding congestion-free routes connecting specified two locations on a network. In many
practical scenarios, congestion on a network, or transmission time taken to send messages, changes
dynamically. Therefore, we need to effectively learn congestion using past congestion data and efficiently find acongestion-free route each time we send a message. While there exist learning algorithms that can be used forpredicting congestion, they incur too much computation cost due to the presence of a huge number of possible
routes. We overcome this difficulty by using the zero-suppressed binary decision diagram (ZDD), which is acompact representation of all possible routes. We develop a learning algorithm that can work on ZDDs without
examining all possible routes explicitly, which enbles us to find congestion-free routes far more efficiently than theexisting algorithms.

Problem Setting

Efficient Algorithm with ZDDs

We choose a route every time we send a message,
where congestion on the network changes dynamically.

Our algorithm first compactly represents the set of all
possible routes using the zero-suppressed binary
decision diagram (ZDD), and then performs learningalgorithm [2] on the ZDD without examining all routes.

time3

congestion-freecongested

We aim to find a congestion-free route at each time.

Difficulties of the Problem

We have achieved to find congestion-free routes
adaptively on a network with dozens of nodes for the
first time.

1

4
3

First, we cannot see how congested each edge iswhen sending a message.
For example, cyberattacks may cause sudden congestion,
which is sometimes hard to observe without sending amessage and getting a feedback.

Second, since there are a huge number of possible
routes, predicting congestion for each route is too costly.
Existing methods (e.g., [2]) learns and predicts congestion
by examining all possible routes, which takes too long time.

5

2
1

1
2
3

3

5

4

4
3
5

2
．．．

time2

Set of all routes

⊤

time1

⊤

Not a route

Route

ZDD

Point 1. Can learn congestion-free routes efficiently.
Existing learning algorithm

All operations are performed on compact ZDDs, and thus
our algorithm can run faster than existing algorithms.

Point 2. Need not reconstruct ZDDs at each time.
．．．

Once a ZDD is constructed, we can reuse it at each time.
This makes our algorithm so efficient as to deal withsudden congestion.

References[1] S. Sakaue, M. Ishihata, S. Minato, “Efficient bandit combinatorial optimization algorithm with zero-suppressed binary decision diagrams,” inProc. 21st International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
[2] N. Cesa-Bianchi, G. Lugosi, “Combinatorial bandits,” Journal of Computer and System Sciences, Vol. 78, No. 5, pp. 1404—1422, 2012.

Contact
Shinsaku Sakaue Email: cs-liaison-ml at hco.ntt.co.jp
Linguistic Intelligence Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

02

Efficient and comfortable AC control by AI- Environment reproduction and control optimization system -Abstract
We propose an air-conditioning control system by AI to save more enegy and to be more comfortable. In a lergescaled facility, it takes serveral time to stabilize temperature. Traditional and typical way of control system,
commonly known as feed-back control, makes sometimes uncomfortable and consumes extra enegy by the timedelay. On the other hand, feed-forward control determines suitable control with predicting environment status of
the facility. For example, if congestion is predicted, the air-flow could be increased or decreased in advance,
which would make the facility’s temperature suitable. We developed AI consisting of environment reproduction
system and control optimization system to calculate the optimal operation schedule for multiple air-conditioning
flows, and demonstrated the importance of feed-forward control through field trial at “COREDO Muromachi”,
which is one of the largest-scale commercial facilities, with NTT-Facilities and MITSUI FUDOSAN.

Background: feed-back or feed-forward

11:00

12:00

13:00

Problem1: complex environment
Temperature

DisturbanceFeed-backcontrol

Air control(ex. cooling/heating,CO2 exhaust)(ex. incoming outside air,human radiant heat)

It is hard to build prediction model.
comfortable

TOO hot

TOO cold

Problem2: insufficient data variation

: ON

: OFF

All air flows are synchronously controlledt (00:00-24:00)
AC1
AC2
…

Feed-forward
control

There are several missing teacher data like when airconditioning flows are partially or periodically turned off,
Prediction is needed to keep temperature comfortable. because normally all air flows are synchronously controlled.
comfortablecomfortablecomfortable

Proposed Method
Data Augmentation by Slicing(DAS)

BEMS※

Combined mask

Single mask

WeatherPedestrian
traffic

Temperature

Domain knowledge

AC1
AC2
…

…
…
…

AC1
AC2
…

…
…
…

(1)Learning within area(2)Learning between areas⇒ Learning within and between area by stochastic generation maskcorresponding to the area to cover missing control pattern.

Condition Control Curriculum Design(C3D)
Formulating comfort and energy saving⇒ Maximize conflicting indicators by deep reinforcement learning
to calculate optimal operation schedule.

Environment
reproduction system(Environment)

Deepreinforcement
learning
Control
optimization system(Agent)schedule

operation※: BEMS: Building Energy Management System

References[1] I. Shake, K. Kawase, Y. Suzuki, “NSRI×NTT×MITSUI FUDOUSAN Collaboration results: Commenced joint experiments to utilize urban
big data and AI in Nihonbashi Muromachi area,” NTT technical journal, Vol. 29, No. 11, pp. 63-65, 2017.

Contact
Nobuhiko Matsuura Email: cs-liaison-ml at hco.ntt.co.jp
Network Innovation Laboratories
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

03

Recover urban people flow from population data
- People flow estimation from spatiotemporal population data -Abstract
Real-time spatiotemporal population data is attracting a great deal of attention for understanding crowd movements
in cities. The data is the aggregation of personal location information and consists of just areas and the number ofpeople in each area at certain time instants. Accordingly, it does not explicitly represent crowd movement.
We propose a probabilistic collective graphical models that can estimate crowd movement from spatiotemporal
population data. There are two technical challenges: (i) poor estimation accuracy as the traditional approachmeans the model would have too many degrees of freedom, (ii) excessive computation cost. Our key idea is tomodel the transition probability between areas by using three factors: departure probability of areas, gathering
score of areas, and geographical distance between areas. These advances enable us to reduce the degrees of
freedom of the model appropriately and derive an efficient estimation algorithm.

What we are working on?

8:00 AM

7:00 AM

9:00 AM It is difficult to utilize people movement
data across various services andenterprises because of privacy issues.
 In many practical situations, only
aggregate information is available.

8:00 – 9:00 AM

7:00 – 8:00 AM
9:00 AM

Our technology
Input: population of each area at each time
Output: # of people who moved between areas

Technical points
Estimation reflects nature of human movements•••

Population fluctuations

Basic idea: Allocate people flow from decreasing area
to increasing area
Point 1: Consider area characteristics

increase- Gathering area ( areas where people are likely to gather )- Emissive area ( areas where people are likely to leave )decrease

Point 2: Consider distance between areas

No change

Transition probability from area 𝑖 to 𝑗𝜃𝑖𝑗 ∝ 𝜋𝑖 × 𝑠𝑗 × exp(−𝛽 ⋅ 𝑑𝑖𝑠𝑡(𝑖, 𝑗))
Point 1••Tokyo
stationShinagawastationEstimation

Basic idea

Basic + Point 1 + Point 2

Point 2

Departure probability 𝜋𝑖determines whether the person leaves area 𝑖 or stay
If the person is deemed to leave 𝑖,(probability of next area 𝑗 to be chosen) ∝( gathering factor of 𝑗 ) × (distance between 𝑖 and 𝑗 )incorrect

correct

Most people
move tocloser area

People move to “gathering area”
Shinagawa ( As the results, flow from
Shinagawa to Tokyo can be recovered. )References[1] Y. Akagi, T. Nishimura, T. Kurashima, H. Toda, “A fast and accurate method for estimating people flow from spatiotemporal population
data,” in Proc. the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence(IJCAI-ECAI-18), pp. 3293-3300, 2018.

Contact
Yasunori Akagi Email: cs-liaison-ml at hco.ntt.co.jp
Proactive Navigation Project, NTT Service Evolution Laboratories
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

04

Improving the accuracy of deep learning- Larger capacity output function for deep learning -Abstract
Deep learning is used in a lot of applications, e.g., image recognition, speech recognition, and machinetranslation. In many applications of deep learning, softmax is used as an output activation function for modelingcategorical probability distributions. To represent various probabilities, models should output various patterns, i.e.,models should have sufficient representation capacity. However, softmax can be a bottleneck of representational
capacity (the softmax bottleneck) under a certain condition. In order to break the softmax bottleneck, we propose
a novel output activation function: sigsoftmax. To break the softmax bottleneck, sigsoftmax is composed of
sigmoid and exponential functions. Sigsoftmax can output more various patterns than softmax without additionalparameters and additional computation costs. As a result, the model with sigsoftmax can be more accurate thanthat with softmax.

Deep Learning

Problem

Deep learning is used in a lot of applications.
(e.g., image recognition or machine translation)
To represent the probability, softmax is used as an
output function.
“Olympics in”“I have a” Example:
next word prediction

We assume the number of outputs 𝑀 is 3.

WeightUnit𝒛softmax

exp(𝑧𝑖 )𝑓𝑖 𝒛 =∑exp(𝑧𝑗 )softmax
softmax
dog Tokyo bird dog Tokyo bird
0.05 0.9 0.05
0.5 0.1 0.4
For accurate prediction, models should output a lot of
patterns of probabilities, i.e., models require sufficient
representation capacity.

Probability of outputs： (𝑃dog, 𝑃Tokyo, 𝑃bird )→ It can be represented as a point on the triangle.
Models connect these points and inputs.
We assume the blue line represents the true patterns ofprobabilities (output set of various inputs).
→ The range of outputs of models should fit this line .

𝑃bird(0,0,1)

Inputs(0,1,0) 𝑃Tokyo
True patterns(1,0,0)“I have a”

Model

𝑃dog

“Olympics in”

Various Inputs

Bottleneck of Representation

Novel output function: sigsoftmax

If the number of units in the last layer 𝑑 < thenumber of outputs 𝑀 -1, softmax does nothave sufficient representation capacity.

Multiplying sigmoid to have larger
representation capacity

𝑃bird

sigsoftmax
𝜎(𝑧𝑖 )exp(𝑧𝑖 )𝑓𝑖 𝒛 =∑𝜎(𝑧𝑗 )exp(𝑧𝑗 )𝑃TokyoLast layer
softmax

𝑃dog

If 𝑑 = 1, softmax cannot
output the true patterns.

In natural language processing, 𝑀 = thenumber of vocabulary is very large, and a lot ofparameters are required if we set 𝑑 = 𝑀-1.

𝜎 𝑧 =

𝑃bird

𝑃Tokyo𝑃dog

1：sigmoid
1−exp(−𝑧)monotonically increasing to 1 from 0

Sigsoftmax has a larger representation capacity
than softmax[1] without additional parameters andcomputation costs.

References[1] S. Kanai, Y. Fujiwara, Y. Yamanaka, S. Adachi, “Sigsoftmax: Reanalysis of the softmax bottleneck,” in Proc. 32nd Conference on Neural
Information Processing Systems (NeurIPS), 2018.

Contact
Sekitoshi Kanai Email: cs-liaison-ml at hco.ntt.co.jp
NTT Software Innovation Center
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

05

Which is cause? Which is effect? Learn from data!
Causal inference in time series via supervised learning

Abstract
Our goal is to automatically discover “causal relationships” from time series data, i.e., a sequence of datameasured over time. Discovering causal relationships has key applications in various fields: e.g., finding that
“R&D expenditure influences sales” is useful for decision making in companies; discovering gene regulatory
relationships provides a key insight for drug discovery researches.
To infer causal relationships, existing methods require us to select an appropriate mathematical expression (i.e.,
auto-regressive model) for each time series data, which is difficult without expertise in data analysis. For thisproblem, we build a novel approach that trains a machine learning model by using various data. Our method does
not require a deep understanding of data analysis and therefore will help us to effectively make an important
decision making in several situations.
(e.g.,)
causal relationship between
R&D expenditure & sales

Problem setup: causal inference in time series
Time series data

Causal relationship

X
Y

X→Y
X←Y

Time

Y

What is causality?

YY

prediction errors

Predictive model(Only Y )If past values of X are
helpful to predict future
values of Y, theneffectR&D exp.
sales
X：R&D expenditure
Y: salesR&D expenditure influences sales

No Causation

Existing methods

cause

X →Y

predicted values

Y
Predictive modelX( X and Y )Xcurrent time

X→Y

If errors are (significantly)
reduced by using past values of X

(Granger causality)

Difficulty
Require to appropriately selectpredictive models• Difficult without expertise indata analysis•

Feature vectors
X →Y X ←Y No Causation

Learn causal relationships from data
Our method

No Causation

can infer causal
Sampling
X →Y
relationships without
LinearX ←Yexpertise in data analysis model

...

X →Y
X ←Y

...

Nonlinearmodel
Various probabilistic models

X→Y
X←Y

Point 1.
Provide feature vectors to
identify causal relationships

Point 2.
Learn how todetermine causality
from training data

Training data

References[1] Y. Chikahara, A. Fujino, “Causal Inference in Time Series via Supervised Learning,” in Proc. 27th International Joint Conference on Artificial
Intelligence (IJCAI), 2018.

Contact
Yoichi Chikahara Email: cs-liaison-ml at hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

06

Forecasting future data for unobserved locations- Tensor factorization for spatio-temporal data analysis -Abstract
Analysis of spatio-temporal data is a common research topic that requires the interpolations of unobserved
locations and the predictions of feature observations by utilizing information about where and when the data wereobserved. One of the most difficult problems is to make future predictions of unobserved locations. Tensor
factorization methods are popular in this field because of their capability of handling multiple types of spatiotemporal data, dealing with missing values, and providing computationally efficient parameter estimation
procedures. We propose a new tensor factorization method that estimates low-rank latent factors by
simultaneously learning the spatial and temporal correlations. We introduce new spatial autoregressive
regularizers based on existing spatial autoregressive models and provide an efficient estimation procedure.

Spatio-Temporal Regression Problem
Our tensor factorization method estimates factors of unobserved locations (blue) with a spatial
regression and employ it as a spatial regularizer. By combining it with future actors (green) obtained
from an autoregression model, we enable to get predictions of unobserved locations (red).
TRMF: a temporal autoregression
method [Yu+, NIPS2016]

Sensortypes

Get predictions of unobserved
locations by multiplying blueand green blocks.

Location

Unobserved
locationA spatial regression method
Time stamp

New time stamp[ours]

Our spatial regression method can deal with both grid and non-grid sensor locations by assigning
the same coefficients based on the angle between a source and a target sensor locations.
Our angle dependent coefficient learning enables(1)

to get factors of unobserved locations 𝑢𝑝,𝑘 .
Lean a spatial
correlations of latent
factors

Spatial regression regularizer

A regression coefficient 𝑏𝑘,𝑛𝑝′ is assigned by theangle between p and p’ (red and blue arrows)← Positive coefficient← Negative coefficient𝑝𝑏𝑘,𝑛𝑝′
𝑤𝑝,𝑝′𝑝′

References[1] K. Takeuchi, H. Kashima and N. Ueda, “Autoregressive Tensor Factorization for Spatio-Temporal Predictions,” in Proc. of 2017 IEEE International
Conference on Data Mining (ICDM), 2017.
[2] 竹内孝, 鹿島久嗣, 上田修功, “自己回帰テンソル分解による時空間データ予測,” 2018年度人工知能学会全国大会（第32回）, 2018.

Contact
Koh Takeuchi Email: cs-liaison-ml at hco.ntt.co.jp
Ueda Research Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

07

Search suitable for various viewpoints- “Pitarie”: Picture book search with graph index based search -Abstract
We propose a similarity search method for finding similar objects in a large-scale database. The search method isbased on a graph index, where each vertex corresponds to a object and two vertices are connected by an edge
when they satisfy a certain similarity condition. The graph index shows small-world behavior, that is, vertices canbe reached from every other vertex by a small number of steps. Hence, searching the graph results in quick
termination of the search process. Furthermore, since the graph index is constructed based on similarity between
two objects, the search method is versatile and can be applied to wide variety of media such as text, images andaudio. When applied to complex objects that are more than two media combined, such as picture books which
consists of text and illustration, users can search from various viewpoints; users can find picture picture books
that are not only similar in content but also similar in style of illustration.

Picture book search system “Pitarie”

Preparation of graph index in advance of search (Off line)

Picture books

Graph index construction
Construct a graph by connecting
similar objects, and use it as a index
for search.
✔ “Small-world behavior” ：Search picture books from various viewpoints.

Graph index

Case 1: Input a summary and search✔ Since the graph construction isCase 2: Search for books with similar contents

independent of media’s characteristic,
the proposed method is applicable tovarious media.

Case 3: Search for books with similar style ofillustration (Example below)

Various applications

Fast search by utilizing graph index
ExampleAny two objects are within a small
number of edge hops.

You can find books with similar styleof picture by various illustrators.

ExampleSearch books suitable forarticulation disorder training
I (わたち（私）) like princesses!

input Graph search output
Query
Search for similar books bycontinuously traversing
from a vertex (object) to its
neighbors.

Similar bookby a different
illustrator.

Cited picture books：

I’m looking for a book that contains
many lisp concerning words with
Lisping more than 3 light syllables.
girl

Search for picture books that
fulfill both requests.

Speechtherapist

Joint Researchwith a hospital.

しろくまちゃんのほっとけーき、わかやまけん作、こぐま社、1972
うさこちゃんとうみ、ディックブルーナ作、福音館書店、1964(The cove-front-like illustration of “The snow queen” was in-house illustrated.)

The SnowQueenphonetic
histogram

Words inthe book

References[1] T. Hattori, T. Kobayashi, S. Fujita, Y. Okumura, K. Aoyama, “Pitarie: Picture Book Search with Interdisciplinary Approach,” NTT Technical
Review, vol.14, no.7, pp. 1-8, 2016.
[2] K. Aoyama, K. Saito, H. Sawada, N. Ueda, “Fast approximate similarity search based on degree-reduced neighborhood graphs,” in Proc.
The 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1055-1063, 2011.

Contact
Takashi Hattori Email: cs-liaison-ml at hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

08

We can transmit messages to the efficiency limit- Error correcting code achieving the Shannon limit -Abstract
For the realization of high-speed digital communication, it is necessary transmitting messages reliably with highefficiency under noisy environment. The limit of efficiency is derived by a computer scientist C. E. Shannon and it
is called the Shannon limit. It is known that we can achieve the limit for a paticular class of channels with LDPC
(Low Density Parity Check) codes or the Polar codes, which are used in the 5G mobile communication
technology. However, it is impossible to achieve the limit for a general class of channels with these codes. We
propose a novel technology called CoCoNuTS (Code based on Constrained Numbers Theoretically-achieving the
Shannon limit). With this technology, we can constuct a code achieving the Shannon limit for a general class ofchannels. Our goal is realizing future high-speed digital communication by establishing related peripheraltechnologies.

Error Correcting Code
Reproducing message
by eliminating noise

Adding informationfor error correction

Channel output

Channel input

Encoder𝑋

Noisy
environment

𝑌

Decoder

SenderReceiverEfficiency＝（no. of message symbols)/（no. of transmitted signals）
We want to maximize efficiency to the Shannon limit.
Difficulty in encoding and decoding

Essence of proposed technology•

To achieve the Shannon limit, it is necessary to place channel
inputs 𝒙𝑖 efficiently (as far as possible in the left figure).

•

When the optimum distribution of a channel input 𝑋 is uniform, we
can achieve the limit with LDPC codes or the Polar codes.
However, when the optimal input distribution is not uniform, it is
impossible to achieve the limit by using these codes.

•

In the naïve decoding method, we have to guess a channel input
from a channel output (𝒙4 from 𝒚 in the left figure) by using thebrute-force search, which is impractical.

Geometric illustration

𝒙2

𝒙3

𝒙1

𝒙5

𝒙4
Channel input𝒚Channel output (noisy)

Proposed technology•

We can realize an ideal layout of channel inputs by using theconstrained-random-number generator.

•

We can avoid the brute-force search by using the constrainedrandom-number-generator, which provides a practical decoding
method.

References[1] J. Muramatsu, “Channel coding and lossy source coding using a generator of constrained random numbers,” IEEE Transactions on
Information Theory, Vol. IT-60, No. 5, pp. 2667-2686, May 2014.
[2] J. Muramatsu, S. Miyake, “Channel code using constrained-random-number generator revisited,” IEEE Transactions on Information Theory,
Vol. IT-65, No. 1, pp. 500-508, Jan. 2019.

Contact
Jun Muramatsu Email: cs-liaison-ml at hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

09

New secrets threaten past secrets- Vulnerability assessment of quantum secret sharing-

Abstract
We investigate a counter-intuitive phenomenon of quantum state discrimination that the success probability ofidentifying all the unknown quantum states increases even when the number of unknown states increases. The
phenomenon is known for vulnerability of quantum secret sharing (QSS), which enables one to distribute a secretamongst untrusted participants securely, however, the necessary and sufficient condition for the phenomenon
was unknown. We show the condition for a specific discrimination task and construct a practical method to realize
the phenomenon. These results advance the analysis of the phenomenon and reveal the vulnerability of QSS.
Since quantum state discrimination lies at the heart of many quantum information processing tasks, our research
widely contributes to the future information society based on quantum technologies, where people would obtain
the benefits from genuine quantum information processing.

Quantum state discrimination

Discrimination of independent Bell states

1. Distribute a randomly chosen quantum state to
two users

1. Distribute randomly chosen Bell states to two users

2. The users try to identify the state by using an
ordinary communication channel

2. The users try to identify all the states 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑁 by using
an ordinary communication channel
Bell 𝑥1
Bell 𝑥2

user 1user 1user 2…

unknown quantum stateuser 2

Bell 𝑥𝑁

ordinary communication channel
Main Results(i) Entropy of the randomness> 1 (ii) Entropy of the randomness < 1
Suppose the quantum state is randomly chosenfrom four Bell states (standard and useful quantum
states)Bell 1
Bell 2
Bell 3
Bell 4

Bell 𝑥

Success probability of the identification< 1

Vulnerability assessment of QSS
1. Distribute quantum states each of which encodes
secret 𝑥 to two users
Bell 1 if 𝑥 = 1
Bell 2 if 𝑥 = 2
Bell 3 if 𝑥 = 3
Bell 4 if 𝑥 = 4

Fail the identification no
matter how well one identify

Succeed the identification
with a practical method

2. Attacking users try to read secret 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑁 byidentifying the distributed quantum states

attacker

attacker…

quantum state

Upper bound of the success probability Lower bound of the success probability
1
1
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0
0
50
300
550
800
100
600
1100 1600
Number of Bell states 𝑁
Number of Bell states 𝑁

Main results → The more secrets one distribute to attacking users, the more vulnerable the secrets become

References[1] S. Akibue, G. Kato, “Bipartite discrimination of independently prepared quantum states as a counterexample to a parallel repetition
conjecture,” Physical Review A, Vol. 97, No. 10, 042309, 2018.

Contact
Seiseki Akiube Email: cs-liaison-ml at hco.ntt.co.jp
Computing Theory Research Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

10

Analyzing the discourse structure behind the text
- Hierarchical top-down RST parsing based on neural networks -Abstract
Analyzing a discourse structure behind the document is crucial for context aware Natural Language Processing(NLP) tasks including machine translation and automatic summarization. We propose a neural discourse parsing
method based on Rhetorical Structure Theory (RST) that regards a document as a constituent tree. Our parser
builds RST trees at different levels of granularity in a document and then replace leaves of upper-level RST trees
with lower-level RST trees that were already constructed. The parsing is performed in a top-down manner foreach granularity level by recursively splitting a larger text span into two smaller ones while predicting nuclearity
labels and rhetorical relations. Unlike previous discourse parsers, our parser can be fully parallelized at each
granularity in a document and does not require any handcrafted features such as syntactic features obtained fromfull parse trees of sentences.

RST parsing by splitting text spans

Point

Recursively splitting a large span into smaller ones
S11：Snowfall accumulation reached 30cm.
S12：The railway company has been
Document
P2
S１1
S１2
S１3
S１4
S１5
S１6
considered to stop the operation.
Elaboration
P1 S1 to S10 S13：The temperature dropped to -10
degrees.
Satellite
Nucleus
Satellite
Nucleus2 occurred due to the road
Cause
S14：Accidents
S１6
surface freezing.
P2 S11 to S16
Satellite
Satellite
Nucleus
Nucleus
S15：The start of the entrance exam
List
List
S１5
has been delayed for two hours.
Nucleus
Nucleus
Nucleus
Nucleus
S16：The exam has been finished
Cause
Cause
without any accident.
P3 S17 to S19

段落Satellite Nucleus Satellite Nucleus

P4 S20 to S22

S１1

Point

S１2

S１3

Satellite Nucleus SatelliteNucleus

S１4
S11

Document: a span consists of paragraphs
Paragraph: a span consists of sentences
Sentence: a span consists of elementary discourse units

P*: paragraph,
S*: sentence

S12

S13
S14 S15 S16
Red words: relation

Evaluation results on RST-DT corpus

Document
Elaboration
Nucleus

Unlabeled

Nuclearity

Relation

Proposed

72.0

58.6

46.7

w/o
granularity

66.5

53.4

43.3

Previous

68.6

55.9

45.8

Human

78.3

66.8

57.1

Satellite
Elaboration
Nucleus
Elaboration

List

Satellite
List

Satellite
Nucleus Nucleus
Cause
Satellite
List
List
Nucleus
Cause

Nucleus

Nucleus

Nucleus
Cause

Satellite Nucleus Satellite Nucleus
S11
P１

S12

S13
P2

Bracket F-score

S14 S15 S16P3P４This is a collaborative research project between Okumura Lab at Tokyo Institute of Technology and NTT CS labs.

References[1] N. Kobayashi, T. Hirao, M. Okumura, M. Nagata, “Top-down RST Parsing Utilizing Granularity Levels in Documents,” in Proc. of 25th Annual
Meeting of Natural Language Processing, pp. 1002-1005, 2019.

Contact
Tsutomu Hirao Email: cs-liaison-ml at hco.ntt.co.jp
Linguistic Intelligence Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

11

When children begin to understand hiragana- Emergent literacy development in Japanese -Abstract

1

No. of correct response

Although many studies have reported child literacy development, it remains unclear when and how toddlers
acquire letters well before starting formal education. We focus on Japanese hiragana letters to investigate(a) when toddlers begin to understand hiragana, and (b) what kind of letters is easily acquired. This work’s
eye-tracking experiment shows that toddlers at 32–39 months begin to understand hiragana letter-soundmapping. Moreover, our large-scale corpus analysis found that various factors, such as letter frequency in picture
books and visual complexity, contribute to the acquisition of hiragana reading and writing. We aim to extend our
findings to develop an early detection method and letter-learning method for children with reading difficulties.

Hiragana understanding measured by eye-gaze
しー、しはどっち？
/Shi/ which is /shi/?

Although toddlers at 32–39 months could barely
read hiragana letters (Fig. 1), they start tounderstand letter-sound correspondence (Fig. 2).

2

Letterfrequency

1.1 letters

24–31 mo. 32–39 mo. 40–48 mo.
Fig. 1 Hiragana reading task

Factors contributing to hiragana reading
and writing acquisition
Visualcomplexity

0.3 letters

Target-viewing ratio

Participants： 80 toddlers at
24–48 months of age
Method： Measuring eye-gaze
patterns during gazes at thescreenAnalysis：Calculating viewing
ratio of target letters

4.3 lettersmonthsFig. 2 Understanding of letter-soundcorrespondence (32–39 mo.)Order of 50-soundtable

Rank correlation = -0.63

Hiragana writing

Hiragana reading

Letter frequency

Picture book corpus(open dataset by National institute of Japanese language)

While hiragana reading acquisition relates to letter frequency
in picture books, hiragana writing acquisition relates to visualcomplexity as well as letter frequency (Fig. 3).

Hiragana reading accuracy rank
Fig. 3 Letter frequency and reading
accuracy

References[1] H. Higuchi, Y. Okumura, T. Kobayashi, “Acquisition of letter-sound correspondence in Japanese-speaking 2-year-olds: An eye-tracking study”
in Biennial Meeting of Society for Research in Child Development (SRCD), 2019.
[2] H. Higuchi, Y. Okumura, T. Kobayashi “Influence of letter properties for Japanese hiragana reading and writing acquisition” The Japan Journalof Logopedics and Phoniatrics, Vol.60 No.2, 113-120, 2019.
[3] H. Higuchi, Y. Okumura, T. Kobayashi “Influence of letter properties for Japanese katakana reading and writing acquisition” The Japan
Journal of Logopedics and Phoniatrics(in press).

Contact
Hiroki Higuchi Email: cs-liaison-ml at hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

12

Measuring emotional response and emotion sharing- Quantitative assessment of empathic communication -Abstract
Empathy is the basis of people's social lives. However, the mechanism has not yet been fully elucidated because
it is a complex phenomenon consisting of subjectivity, physiology, and behavior. The purpose of this study is toquantify empathy from a multifaceted point of view considering individual differences. We examined how
physiology, behavior and cognition are related in an individual, and how they are shared with other individuals. In
order to deal with the large individual difference in the subjective judgment, we built a computational model that
explains the individual difference from their personality traits, exploiting the wisdom of the group approach which
aggregates the judgment of multiple individuals. This kind of framework for quantitative measurement of empathy
including individual differences will make it possible to assess and predict the effects of interventions that promote
empathy tailored to individuals. We believe this is an essential step toward improving human well-being.
Empathy involves a variety of phenomena,from the low-level phenomenon (e.g.
physiological synchronization) to the higherlevel phenomenon (e.g. cognitive sharing). We
try to understand these from various aspects.

Individual’s physiology and behavior [1]* joint research with Tsukuba University

In the task of distinguishing between posed andspontaneous faces, we directly comparedhumans with machines using video or EMG signal
in terms of accuracy.

Accuracy: human < Video ≒ EMG

Cognition and personality trait on
emotion sharing [2]The higher Systemizing
Quotient, the more accurately
people answer the degree of
emotional similarity with others.
Self

Emotiondimension

People with a higher (lower)
Empathizing Quotient (EQ)
are less (more) likely tohave similar emotions.

High EQ
Self

Long

Short
Opponent→ Dissimilar→ SimilarLong

Low EQ

We found that the physiological response is
synchronized during cooperative work, and that thehigher the physiological synchrony, the more consistent
the impression of the interaction.
Person 1

Person 2

Emotiondimension
Short
Opponent

Physiological synchrony

Based on the similarity judgment theory, we clarified whichtypes of people tend to have emotion similar to others andto recognize their emotional similarities accurately.

Physiological synchrony andcognitive sharing

Time

Finger plethysmogram (amplitude value)
synchronized between a pair of people

Degree of consistency ofimpression among people

References[1] M. Perusquia-Hernandez, S. Ayabe-Kanamura, K. Suzuki, S. Kumano, “The Invisible Potential of Facial Electromyography: a Comparison of
EMG and Computer Vision when Distinguishing Posed from Spontaneous Smiles,” in Proc. Conf. Human Factors in Computing Systems(CHI), 2019.
[2] L. Antaket, M. Matsuda, K. Otsuka, S. Kumano, “Analyzing Generation and Cognition of Emotional Congruence using EmpathizingSystemizing Quotient,” International Journal of Affective Engineering, Vol. 17, No. 3, pp. 183-192, 2018.

Contact
Shiro Kumano Email: cs-liaison-ml at hco.ntt.co.jp
Sensory Resonance Research Group, Human Information Science Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

13

Touch, enhance, and measure the empathy in crowd- Towards tactile enhanced crowd empathetic communication-

Abstract
This study targets the empathetic communication that occur in sharing the same field. To provide quantitativeevaluations, physiological changes are observed using optical camera communication. Vibrotactile stimulation
was presented simultaneously to enhance the viewing experience. Thanks to the vibrotactile communication
technologies, we can record, distribute, and display tactile information in accordance with audiovisual contents andoptical camera communication enabled us to simultaneously observe physiological responses from crowd ofpeople. By combining these interdisciplinary technologies we can run cyclical research processes of senseintervention, measurement, evaluation, and factor analysis to progress the research on empathetic
communication. Based on these research results, we will make a design theory for making field that can enhance
the wellbeing of the people gathered in the field.
Mass measurement by Optical
Camera communication

Vibrotactile communication

LED

Audio
Video

Demodulation

Touch

Recording and creating
contents for vibrotactile.

Meta data(Video, Audio, Touch)

Video

Audio

Touch

Image capture
light

Meditation

dark
Visual codec
H.265/HEVC
H.264.AVC
MPEG-4MPEG-2... etc

Audio codec

Tactile codec

MP3
AAC
ALS
USAC
... etc

ALS... etc

Transfer to
LED Blinking

Vibrotactile display

Tactile communication
combined with video andaudio

Brain activities
β：Attention
α：Meditation

Attention

Descript the brain activity
as character’s face.

Observation of empathetic communication using OCC and vibrotactile
Meditaion

We measure the brain activities of
the users who are experiencing
vibrotactile intervention by using
optical camera communication.
We compare the condition
between alone and together toelucidate the effect of empathetic
communication caused by sharing
a field.

Alone

Attention

Alone

Together

Meditation

Attention

Together

By using the level of brain activities and its amount ofsynchronization, we estimate the effect of empathetic level,
and investigate the effect of imposing vibrotactile stimuli or
the effect of the number sharing the field.

References[1] Y. Shiraki, T. G. Sato, T. Moriya, “Flexible synchronization in optical camera communication with on-off keying,” in Proc. IEEE GLOBECOM
Workshops, pp. 1-6, 2017.
[2] T. G. Sato, Y. Shiraki, T. Moriya, “Audience Excitement Reflected in Respiratory Phase Synchronization,” IEEE Int. Conf. on SMC, pp. 28562860, 2017.

Contact
Takashi G Sato Email: cs-liaison-ml at hco.ntt.co.jp
Moriya Research Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

14

Robot understands events in your story- Chat-oriented dialogue system based on event understanding -Abstract
The proposed chat-oriented dialogue system can make users think the system understands the user’s utterances.
By understanding the user’s utterances as an event structure (a group of time, place, person, etc.), we achieve achat-oriented dialogue system that can sympathize and delve into topics during a chat. To understand a user’s
events from the user’s utterances, a system must understand various words/phrases in user utterances. To tackle
this problem, we focus on general words and phrases that are familiar in a chatting situation but difficult to extract
by conventional methods. Using this technology, systems can extract a user’s utterances by organizing theextracted information. In the future, we aim to foster a world where humans can converse with systems likehumans with mutual understanding by grounding the extracted information to the system and external knowledge.

System utterance generation based on event understanding
The system generates its next utterances corresponding to the user’s event, extracted from the user’sutterances as structured event information, by comparing the event with system knowledge.
Example of an extracted event structure

How was your summer vacation?

Time
Place
PersonActionFeeling

I went to Osaka!
Oh, did you go to USJ?
No, I went to Namba!
Oh, Namba! Did you eat takoyaki?

ate takoyaki

Example of similar events (system knowledge)

Yes, I searched on the internet
and went to a popular shop.

ユーザ
User

Summer vacation
Osaka/Namba

システム
SystemGood! Was it delicious?

Ability to empathize and delve into topics

Time
Place
PersonActionFeelingin vacation
Osaka/USJ
with family
saw turtles
was cute

Time
Place
PersonActionFeeling

in September
Osaka/Nambawith friends
ate takoyaki
was delicious

This system was developed based on the dialogue system that won first prize in a live competition in Japan (2019).

Phrase detection in user utterance
To understand user’s events from his/her utterances, various types of words/phrases must be extracted.
By analyzing such words and phrases in chats, we achieve this extraction.
Example of location words/phrases in chats
Conventional
NE extraction

Proposedextraction

Italy

Italy

We went to the park near
Kyoto station.

Kyoto station

park near Kyoto
Station

I often go to electricity shop.

(none)

electricity shop

User utterance（Red is location phrase）

I went to Italy.

70% chance that phrases that are not Named Entity (NE)appear in chats (in case of location words/phrases)

Extracted phrase
This weekend, I went to the park near Kyoto sta.
When

Where
Extracted general word

Ability to extract general words and phrases

References[1] H. Narimatsu, H. Sugiyama, M. Mizukami, “Detecting Location-Indicating Phrases in User Utterances for Chat-Oriented Dialogue Systems,”
in Proc. The Fourth Linguistic and Cognitive Approaches to Dialog Agents Workshop (LACATODA), 2018.
[2] H. Sugiyama, H. Narimatsu, M. Mizukami, T. Arimoto, “Empirical study on domain-specific conversational dialogue system based oncontext-aware utterance understanding and generation,” JSAI SIG-SLUD, 2018. (in Japanese)[3] M. Mizukami, H. Sugiyama, H. Narimatsu, “Event Data Collection for Recent Personal Questions,” in Proc. LACATODA, 2018.

Contact
Hiromi Narimatsu Email: cs-liaison-ml at hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

15

Voice command and speech communication in car- World’s best voice capture and recognition technologies -Abstract
Our technology can support a speech command and hand-free communication even in noisy environment suchas road noise without any stresses. Clear speech can be picked up from the noise-mixed sound in order to realize
speech command with high accuracy. A lot of computational complexy and memory was required to keep speech
quality and reduce only noise so far. This problem can be solved by using our acoustical knowhow, moreover, lowlatency was able to be also achieved. In addition, a sign of howling was able to be detected rapidly by combining
multiple microphone array. Our goal is to improve an in-car acoustical environment by reducing noises which areroad noise, engiine noise, and any sound from other cars. We will also try to establish an event detection
technology in order to help a driving assistant or an early maintenance by detecting emergency car or anomalous
in sound.
IM : Intelligent Microhpone
ASTER : Anti-distortion Suppression of noise with mask-based TransER function estimation

Differences from conventional

Conventional NTT
technology 1 (IM*)
Abstract

Conventional NTT
technology 2 (ASTER*)

Extracting target sound Reducing noise rom noisefrom noise-mixed sound mixed speech signal while
by combining a linear and minimizing speech
non-linear process.
distortion.

This technology(IM-ASTER)
Achieving low computational
complexity and small amountof memory, and having
advantages of IM and ASTER.

Image of process
Noise
Speech(Input signal)

Speech is degradedby too much noise
reduction

Noise is removed while
Noise is removed while
minimizing speech distortion minimizing speech distortion

Minimize distortion

△

○

○

High level noise

○

△

○

Computational complexity / memory

○

×

○

Demonstration
In car communication

Speech command from target seat

Hand-free communicationMusic
Telephone

Road

Music DVD
Turn on light

Telephone

Telephonenoise

References[1] Y. Hioka, K. Furuya, K. Kobayashi, K. Niwa, Y. Haneda, “Underdetermined sound source separation using power spectrum density
estimated by combination of directivity gain,” IEEE Trans. Audio, Speech, Lang. Process., vol. 21, no. 6, pp. 1240-1250, 2013.
[2] T. Yoshioka, N. Ito, M. Delcroix, A. Ogawa, K. Kinoshita, M. Fujimoto, C. Yu, W. H. Fabian, M. Espi, T. Higuchi, S. Araki, T. Nakatani, “TheNTT CHiME-3 system: Advances in speech enhancement and recognition for mobile multi-microphone devices,” in Proc. IEEE ASRU, pp.
436-443, Dec. 2015.

Contact
Noboru Harada Email: cs-liaison-ml at hco.ntt.co.jp
Media intelligence laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

16

Learning speech recognition from small paired data
- Semi-supervised end-to-end training with text-to-speech -Abstract
We propose a semi-supervised end-to-end method for learning speech recognition from small paired data andlarge unpaired data. This is because preparing the paired data of a speech and its transcription text requires alarge amount of human effort. In our method, we introduce speech and text autoencoders that share encoders
and decoders with an automatic speech recognition (ASR) model to improve ASR performance using speech-only
and text-only training datasets. To build the speech and text autoencoders, we leverage state-of-the-art ASR andtext-to-speech (TTS) encoder-decoder architectures. These autoencoders learn features from speech-only andtext-only datasets by switching the encoders and decoders used in the ASR and TTS models. Simultaneously,
they aim to encode features to be compatible with ASR and TTS models using a multi-task loss.

Speech Recognition• Prepare paired data ofspeech and transcription
text for training• Train speech encoder
and text decoder networks• Perform speech
recognition on given
speech input“HELLO”

Problems• The encoder-decoder model requires a large amountof the paired speech and transcription text dataset.

decoder• Preparation of such dataset needs huge amounts oftime and money
feature• If the networks can learn speech-only and text-only
datasets, the data preparation becomes much easierencoder“HELLO”

encoder

decoder

Semi-supervised training method
Training task

Point 1：Combine speech-to-text task with text-to-speechspeechtotext・Train with paired data: speech-to-text , text-to-speech・Train without paired data: speech-to-speech , text-to-text✔

decoder“HELLO”

encoder

Jointly train
all encoders anddecoders

text
tospeech

char
error
rate
15.0
%

✔

✔
✔

encoderspeechtospeech✔

“HELLO”

decoder

text
to
text

9.0%
✔

✔

✔

8.7%
✔

8.4%

Point2: Training to encode or decodefeaturesthat look similar to each other

References[1] S. Karita, S. Watanabe, T. Iwata, A. Ogawa, M. Delcroix, “Semi-supervised end-to-end speech recognition,” in Proc. of 2018 Interspeech,
pp. 2-6, 2018.
[2] S. Karita, S. Watanabe, T. Iwata, M. Delcroix, A. Ogawa, T. Nakatani, “Semi-supervised end-to-end speech recognition using text-to-speech
and autoencoders,” in Proc. of 2019 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019.

Contact
Shigeki Karita Email: cs-liaison-ml at hco.ntt.co.jp
Signal Processing Research Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

17

Who spoke when & what? How many people were there?
- All-neural source separation, counting and diarization model -Abstract
We propose a method to accurately estimate “who spoke when" based on speaker's voice characteristics. It
works even in a situation where multiple speaker’s speech signals overlap, and accurately counts the number of
speakers in such cases. Conventional methods with the similar functionality works only when the observed signal
satisfies certain a priori (unrealistic) assumptions (e.g. the number of speaker known in advance, speakers neverchange their locations). However, these assumptions cannot be often satisfied in realistic scenarios, which leads
to performance degradation. On the other hand, the proposed method, which is based purely on deep learning,
can theoretically learn and deal with any realistic conversation situations. It is expected to serve as a fundamental
technology for automatic conversation analysis systems, and will contribute to realization of automatic meeting
minutes generation systems and communication robots.

Difficulty of automatically analyzing ‘who spoke when’(Target environment example)

Conversation data is dynamic & diverse（Periods where each speaker speaking)

B

A
B

A

→Difficult for conventional

C

C・The number of speakers is arbitrary
and changes over time・People speaks intermittently
・Voice often overlaps each other・Speaker location changes randomly
Time method to handle

Proposed method- Estimating ‘who spoke when’based on deep learning- Entire process optimized
with training data!

NeuralNetwork

Characteristics of
person A’s voice

Proposed method…

Tracking

Recorded signal
1st proc,
block

NeuralNetwork

NeuralNetwork

2nd proc. 3rd proc. 4th proc.
block
block
block
Characteristics of
person A’s voice

Characteristics of
person B’s voice

Tracking
・Iteratively extracts speakers’ voice
and repeat the process same times
as the number of speakers
・When a new speaker appears, it increase the extraction
iteration by one
・Tracks a speaker over time based on voice characteristics
・Block online processing

NeuralNetwork

Characteristics of
person A’s voice

NeuralNetwork

NeuralNetwork

NeuralNetwork

Silence
NeuralNetwork

Characteristics of
person C’s voice

Tracking

NeuralNetwork

Silence

Advantage of the proposed method in comparison with conventional method
・The proposed method achieves source separation and source number counting simultaneously.
・The proposed method can track speaker’s voice over time based on voice characteristics. It can keep tracking the
speaker even if the speaker changes his/her location.

References[1] K. Kinoshita, L. Drude, M. Delcroix, T. Nakatani, “Listening to each speaker one by one with recurrent selective hearing networks,” in Proc.
IEEE International Conference on Acoustics, Speech & Signal Processing (ICASSP), pp. 5064-5068, 2018.
[2] T. von Neuman, K. Kinoshita, M. Delcroix, S. Araki, T. Nakatani, R. Haeb-Umbach, “All-neural online source separation, counting, anddiarization for meeting analysis,” in Proc. IEEE International Conference on Acoustics, Speech & Signal Processing (ICASSP), 2019.

Contact
Keisuke Kinoshita Email: cs-liaison-ml at hco.ntt.co.jp
Signal Processing Research group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

18

Changing your voice and speaking style- Voice and prosody conversion with sequence-to-sequence model -Abstract
We propose an voice and prosody conversion method for impersonating a desired speaker’s identity and hiding a
speaker’s identity. The conversion method consists of acoustic feature conversion and time-domain neural
postfilter. The acoustic feature conversion is based on a sequence-to-sequence learning with attention
mechanism, which makes it possible to capture the long-range temporal dependencies between source andtarget sequences. The later post filter employs a cyclic model based on adversarial networks, which requires noassumption for the speech waveform modeling. In contrast to current voice conversion techniques, the proposed
method makes it possible to convert not only voice timbre but also prosody and rhythm while achieving highquality speech waveform generation due to the proposed time-domain neural post filter. The remaining challenge
is the real-time voice conversion which is our ongoing work.
Convert a speaker identity

Overview
Input
speechbut, quit synthetic…

Converted
speechAcoustic feature conversion [1](Intonation, voice timbre, and rhythm)

Naturally-sounding!
OutputspeechTime-domain
neural post filter [2](e.g., Impersonating a speaker’s identity and modifying pronunciation)

Acoustic Feature Conversion [1]Speech samples→• Train encoders, decoder, and attention mechanism

Time warping function ofsource context information• Encoder-decoder model makes it possible toContext information
Extraction

1.Handle input and output sequencesof different lengths
2.Convert not only voice timbre
but also rhythm

Attentionmechanism

Acoustic feature prediction of the nexttime frame from the previous output
and the aligned context information

Source acoustic feature
Sourceencoder

Targetdecoder• Attention mechanism has the abilities to

1.Select critical information fromthe encoded representationin accordance with the output
sequence representation

Model training for minimizing
the differences toTarget acoustic feature
Targetdecoder

2.Consider the long-range temporaldependencies for converting intonation

Time-domain Neural Post Filter [2]Training
step

Minimize the prediction error

Conversion step

Use the predicted acoustic feature for predicting the next ones

Minimize the discrimination error

Train 𝐹 for imitating 𝐹(𝐲)1.Train the models with non-parallel
data of synthetic and natural speech

Real/Fake• Train conversion functions 𝐺, 𝐹 between synthetic and natural
speechas natural
speech by 𝐷
• Cyclic model makes it possible to

Minimize the reconstruction error

Train 𝐺, 𝐹 for minimizing
The differences to 𝐲ො

FakeDiscriminator

Minimize the discrimination error

Synthetic-to-natural

Train 𝐺 for imitating 𝐺 𝐱
as natural speech by 𝐷

Real

Fake

Synthetic speech

Discriminator

Natural-tosynthetic

Train 𝐺, 𝐹 for minimizing
The differences to 𝐱ො
Minimize the reconstruction error

Real

Real/Fake

2.Handle the phase information correctly
due to need for the reconstructionof speech waveform• Generative adversarial learning helps
to generate clear speech

Speech samples →(e.g., improving the speech quality)

Natural speech

References[1] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo, ” AttS2S-VC: Sequence-to-Sequence Voice Conversion with Attention and Context Preservation
Mechanisms," in Proc. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2019), May 2019.
[2] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo,“WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation," arXiv:1904.02892,
Apr. 2019, (submitted to INTERSPEECH2019.)

Contact
Kou Tanaka Email: cs-liaison-ml at hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

19Face-to-voice conversion and voice-to-face conversion- Crossmodal voice conversion with deep generative models -Abstract
Humans are able to imagine a person's voice from the person's appearance and imagine the person's
appearance from his/her voice. In this work, we take an information-theoretic approach using deep generativemodels to develop a method that can convert speech into a voice that matches an input face image and generatea face image that matches the voice of the input speech by leveraging the correlation between faces and voices.
We propose a model, consisting of a speech encoder/decoder, a face encoder/decoder and a voice encoder. We
use the latent code of an input face image encoded by the face encoder as the auxiliary input into the speech
decoder and train the speech encoder/decoder so that the original latent code can be recovered from thegenerated speech by the voice encoder. We also train the face decoder along with the face encoder to ensurethat the latent code will contain sufficient information to reconstruct the input face image.

Crossmodal Voice Conversion/Face Image Generation
Leverage underlying correlation between voices and appearances to• convert speech into a voice that matches an input face image, and• generate face image that matches input speech.

Information-theoretic approach using deep generative models
Voice Convertor 𝑮：Neural network (NN) that converts input speech 𝐱 into 𝐱ො = 𝐺(𝐱, 𝐲) by using face image 𝐲 as auxiliary input
Training objective： Train G so that mutual information between 𝐱ො = 𝐺(𝐱, 𝐲) and 𝐲 is maximized
Maximize lower bound w.r.t. 𝐺 and 𝑅

Voice Convertor 𝐺
Input speech
Feature Dim.

Timedependent
SpeechEncoder NN

SpeechDecoder NN

Feature Dim.

NN that approximates posterior 𝑝(𝐲|𝐱)Speech and face image pair

Output speech

Time

Time

Input image
Timeindependent
Face
Encoder NN

Five NNs to be trained simultaneously:

Consistency between𝐜 and 𝐜Ƹ corresponds
to mutual information
between 𝐜 and 𝐱ො

Speech Encoder NN ： Aims to extract linguistic information 𝐳
Face Encoder NN

Converted speechwith voice thatmatches input
face image

Voice
Encoder

Voice2FaceGenerator 𝑅
Face
Decoder NN
Audio/image
samples areavailable here:： Aims to extract facial impression 𝐜

Speech Decoder NN ： Generates speech 𝐱ො from 𝐳 and 𝐜
Voice Encoder NN： Aims to extract voice characteristic 𝐜Ƹ

Face Decoder NN： Generates face image 𝐲ො from 𝐜 or 𝐜Ƹ

Predicted image

References[1] H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, "StarGAN-VC: Non-parallel many-to-many voice conversion using star generative adversarial
networks," in Proc. 2018 IEEE Workshop on Spoken Language Technology (SLT 2018) , pp. 266-273, Dec. 2018.
[2] H. Kameoka, T. Kaneko, K. Tanaka, N. Hojo, “ ACVAE-VC: Non-parallel voice conversion with auxiliary classifier variational autoencoder, ”
arXiv:1808.05092 [stat.ML], 2018.
[3] H. Kameoka, K. Tanaka, A. Valero Puche, Y. Ohishi, T. Kaneko, “Crossmodal Voice Conversion,” arXiv:1904.04540 [cs.SD], 2019.

Contact
Hirokazu Kameoka Email: cs-liaison-ml at hco.ntt.co.jp
Media Recognition Research Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

20

Learning unknown objects from speech and vision- Crossmodal audio-visual concept discovery -Abstract
In order for AI to visually perceive the world around it and to use language to communicate, it needs a dictionarythat associates the visual objects in the world with the spoken words that refers to them. We explore a neuralnetwork models that learn semantic correspondences between the objects and the words given images andmultilingual speech audio captions describing that images. We show that training a trilingual model
simultaneously on English, Hindi, and newly recorded Japanese audio caption data offers improved retrieval
performance over the monolingual models. Further, we demonstrate the trilingual model implicitly learns
meaningful word-level translations based on images. We aim for a future in which AI discovers concepts
autonomously while finding the audio-visual co-occurrences by simply providing media data that exists in theworld such as TV broadcasting. We also consider the application to large-scale archive retrieval and automatic
annotation that involves interactions between different sensory modalities such as vision, audio, and language.
EnglishHindi

Japanese

EnglishAudio-visualconcept
discovery(2) Learn semantic correspondences(1) Collect images and speech
between visual objects and spoken
waveforms describing those images.
words.

Japanese(3) Highlight relevant regions ofimage as they are being described.

Evaluation of embedding spacelearned from image and audio captions

Learning neural network embeddings
for images and spoken audio captions
Paired image and audio captions are more similarin embedding space than mismatched pairs(1) Audio-visual retrieval performance
Recall scores for the top 10 hits (1,000 image/caption pairs)

Image CNN（Pre-trained VGG16）

Monolingual model: 0.45

*CNN: Convolutional Neural Network

Meanpooling

Hindi(2) Cross-lingual audio2audio retrieval performance

Embeddingspace

Recall scores for the top 10 hits (1,000 cross-lingual caption
pairs)

w/ using images: 0.50

w/o using images: 0.01
English CNN

Exploring visually grounded speech-tospeech translation

Mean
pooling

Hindi

Two small children are walking on a dirt

English road in what seems to be a cornfield

Hindi CNN…
Unpooled
embeddings

Meanpooling

JapaneseJapanese CNN

Meanpooling…
Mismatchedaudio captions

Japanese

子供二人組が道を歩いている。
えー、左右には背の低い草木が生い茂っている

children cornfield

EnglishMultilingual model: 0.50

Similarity matrix
between English andJapanese captions

子供

草木*Joint research results with MIT Computer Science and Artificial Intelligence Laboratory

References[1] Y. Ohishi, A. Kimura, T. Kawanishi, K. Kashino, D. Harwath, and J. Glass, “Crossmodal Search using Visually Grounded Multilingual
Speech Signal,” IEICE Technical report on Pattern Recognition and Media Understanding (to appear)[2] D. Harwath, G. Chuang, and J. Glass, “Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech,” In
Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2018), April 2018.

Contact
Yasunori Ohishi Email: cs-liaison-ml at hco.ntt.co.jp
Media Recognition Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

21

Neural audio captioning- Generating text describing non-speech audio -Abstract
Recently, detection and classification of various sounds has attracted many researchers attention. We propose anaudio captioning system that can describe various non-speech audio signals in the form of natural language. Most
existing audio captioning systems have mainly focused on “what the individual sound is,” or classifying sounds tofind object labels or types. In contrast, the proposed system generates (1) an onomatopoeia, i.e. a verbal
simulation of non-speech sounds, and (2) an sentence describing sounds, given an audio signal as an input. This
allows the description to include more information, such as how the sound sounds and how the tone or volume
changes over time. Our approach also enables directly measuring the distance between a sentence and an audio
sample. The potential applications include sound effect search systems that can accept detailed sentence
queries, audio captioning systems for videos, and AI systems that can hear and represent sounds as humans do.

Detailed audio captioning

Experimental results

・Given an audio input, the system describes not only what thesound source is, but also how it is sounding and how it is changing
over time, as a natural language sentence. NEW

The description matches the sound:totally
well
partlypoorlyGagagaGeneration

Metallic soundgraduallygetting louder

Various non-speechsoundsAppropriateness of the output sentence(without specificity conditioning)

MethodWith the specificity control, the sentence is:

A high-pitched fricative noise is …
Decoder（LSTM）zmuch better
somehow betterthe same quality
somehow worse
much worse

Onomatopoeias
mapped in the latent space(Latent space)

Approx.
2/3

Effectiveness of the specificity conditioning

Encoder（LSTM）c(Specificity)

Approx. 3/4

Sound sources

Instrumental sounds
mapped in the latent space

1. Onomatopoeic mode: output a word simulating the sound
2. Sentence mode: output a sentence describing the sound【Point】 No unique correct answer for captioning→ 【Proposal】 Can control the degree of detailby conditioning the decoder by “Specificity” input
Specificity: Sum of the amount of information
contained in the output text.

Examples of description for a base drum sound(English translation from Japanese)c

Generated sentence-

A low sound rings for a moment20

A low sound sounds for a moment50

A low, striking sound sounds as if something is dashed on a mat

80

A very low-pitched drum is played only once

110

A faint, low-pitched sound sounds as if something is hit dully, andit soon disappears

References[1] Shota Ikawa, Kunio Kashino, “Generating sound words from audio signals of acoustic events with sequence-to-sequence model,” In Proc.
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2018), April 2018.
[2] Shota Ikawa, Kunio Kashino, “Acoustic event search with an onomatopoeic query: measuring distance between onomatopoeic words andsounds,” In Proc. Detection and Classification of Acoustic Scenes and Events (DCASE 2018), November 2018.

Contact
Kunio Kashino Email: cs-liaison-ml at hco.ntt.co.jp
Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

22

Recognizing types and shapes of objects from sound- Crossmodal audio-visual analysis for scene understanding -Abstract
Sounds provide us with vast amounts of information about surrounding objects and scenes and can even remind
us visual images of them. Is it possible to implement this noteworthy ability on machines? We addressed this task
and developed a crossmodal scene analysis method that can predict the structures and semantic classes ofobjects/scenes from auditory information alone, i.e., without actually looking at the scene. Our approach uses a
convolutional neural network that is designed to directly output semantic and structural information of objects andscenes by taking low-level audio features as its inputs. An efficient feature fusion scheme is incorporated tomodel underlying higher-order interactions between audio and visual sources. Our method allows users tovisually check the state of the scene even in a case where they cannot or do not want to use a camera. Our
method will contribute to expanding the availability of monitoring applications in various environments.

Crossmodal Scene Understanding
Predicting structures and semantic classes of objects frommulti-channel audio signals

Visualizing scenes where photographing
impossible or prohibited

Demonstrated recognition of limited
number of object classes possible so far

Experimental setup

FeaturesFeature fusion layer for efficiently modeling higher-orderinteractions between audio and visual sources
Typical fusion scheme
has prohibitive number ofparameters!

Recognition of various classes of objects
from real sound sources possible

Our approach reduces the number ofparameters by considering weighted
outer-product of lower-dimensional features

References[1] G. Irie, M. Ostrek, H. Wang, H. Kameoka, A. Kimura, T. Kawanishi, K. Kashino, “Seeing through sounds: predicting visual semantic
segmentation results from multichannel audio signals,” in Proc. International Conference on Acoustics, Speech, and Signal Processing(ICASSP), 2019.
[2] H. Wang, G. Irie, H. Kameoka, A. Kimura, K. Hiramatsu, K. Kashino, “Audio-based Semantic Segmentation based on Bilinear Feature
Fusion,” Meeting on Image Recognition and Understanding (MIRU), 2018.

Contact
Go Irie Email: cs-liaison-ml at hco.ntt.co.jp
Recognition Research Group, Media Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

23

Speech of chirping birds, music of bubbling water- Sound texture conversion with an auditory model -Abstract
Natural scenes such as bubbling water and rustling trees give us specific perception of sound textures. We
developed a method to artificially give such textures to speech and music. Inspired by research on manipulating
image textures, we improved the method so that we can applied it to sound. A computational model that takes
into account our hearing mechanism enabled effective control of sound textures in terms of hearing sensation.
The method is realized in the same framework as the image texture manipulation. This indicates that, in the brain,
seen and heard textures are processed by the similar mechanisms. From a scientific viewpoint, this study leads tounderstanding of the mechanisms of sound texture perception by comparing the model’s internal states with thebrain activities induced by hearing sounds. From an application perspective, the proposed method enables us to
speak in a voice that does not actually exist or to play music with an instrument that does not exist.

Prior study: image texture conversion
By combining contents in a photo and textures in a painting, animage that has both of them is synthesized. (Converting the texturesof the photo to that of the painting while preserving its contents.)

This study: sound texture conversion
A model of the auditory system calculates
the features that represent textures ofenvironmental sounds and the features that
represent contents of speech and music.

Prior study: representation ofsound textures
A model of the auditory system
calculates the features that
represent textures of environmental
sounds.

By combining contents in a speech or music and textures inan environmental sound, a sound that has both of them issynthesized. (Converting the textures of the speech ormusic to that of the environmental sound while preserving
its contents.)

References[1] T. Koumura, H. Terashima, S. Furukawa, “Chimeric sounds with shuffled “texture” and “content” synthesized by a model of the auditory
system,” in Proc. International Symposium on Universal Acoustical Communication, 2018.
[2] T. Koumura, H. Terashima, S. Furukawa, “Sound texture transfer using a model of the auditory system,” in Proc. Annual Conference of theJapanese Society for Artificial Intelligence, 2017.

Contact
Takuya Koumura Email: cs-liaison-ml at hco.ntt.co.jp
Sensory Resonance Research Group, Human Information Science Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

24

Danswing papers- An illusion to give motion impressions to papers -Abstract
We propose a technique to give motion illusions to static paper objects. Previous studies have reported visual
illusions wherein a static “virtual” object apparently moves on the basis of the luminance interaction between
object’s contours and the object’s background. However, no studies have proposed a method to give motion
illusion to a static “real” object. This study found a phenomenon in which a paper objects having bright and dark
contours apparently moved against the background with dynamic luminance modulation. Manipulating the contour
patterns could also produce not only a simple illusory movement such as translation but also relatively complex
illusory movements such as expansion, contraction, and rotation. We call this technique Danswing (Dance +swing) papers. By utilizing the Dancing papers, it is possible to gather customer’s attention towards an actuallystatic, but perceptually dynamic, objects.

How to create Danswing papers●Apparently rotating heart object①We digitally create images of a grayheart-shape
figure,aslightly
anticlockwise rotated white heart-shape
object, and a slightly clockwise rotatedblack heart-shape object.
Anticlockwise②The three heart-shape objects are digitally
synthesized so that the gray object is set atthe most front layer. The synthesized image isprinted out, cut, and placed on the display
with dynamic luminance modulation.

Clockwise

Print
&
Cut

Place iton the display

Visual mechanism for Danswing papers
Proportion correct for detecting
illusory motion●Specific bands of spatial frequency are related to Danswing papers.
Using stimulus clips wherein a specific band of spatial
frequency was extracted, we asked observers to detect
illusory motion, and found that the observers’ performance
was good when the clips contained the specific bands ofspatial frequency.

Effective bands
for Danswing papers

N = 10 (Error bars: SEM)
Original

Please scan the right QR code, and check
our YouTube clip of Danswing papers!

Cut-off spatial frequency (cpi)References[1] T. Kawabe, “Danswing papers,” in Proc. SIGGRAPH Asia 2018 (SA '18) Posters Article No. 4.
[2] T. Kawabe, “Danswing papers,” Top 10 finalist of Best illusion of the year contest. http://illusionoftheyear.com/2018/10/danswing-papers/

Contact
Takahiro Kawabe Email: cs-liaison-ml at hco.ntt.co.jp
Sensory and Representation Research Group, Human Information Science Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

25

Measuring visual abilities in a delightful manner
Self eye-check system using video games and tablet PCs

Abstract
We explore an enjoyable and simple way to measure functions of the eye. We combined know-hows accumulated
through vision-science-experiments with technologies for drawing precise computer graphics on web browsers,and created a system for testing visual functions with a generic tablet device. Our system can be utilized for selfchecking of eye functions in a delightful way like a video game. Previous tests of visual function are often timeconsuming and normally require the help of medical experts. Our system allows users to measure each visualfunction in about 3 minutes. This system can be utilized to self-check users’ eye condition routinely. In addition,by accumulating knowledge through data of many people including patients with eye diseases in simple andshort-time measurement, we can expect an early detection of eye diseases, rehabilitation application, andscientific findings about complex visual processes.

Simple tablet tests

Usage of vision test in various places

Heavy

Symptom

Consultation / examination room•

Few medical expert for one
person•High test sensitivity•

Easy tests with short measuring time•

Run on tablet PCs.

•

Modified conventional measurement
protocols.

Reliable/
Stableassessment

None / Unaware

Light

Hospital waiting room•

Several medical expert for
many person•

Intermediate test sensitivity

Gamified vision tests•

Gamification with good graphicaland protocol designs•

Aiming at repeated use in daily
situations

Daily
•

Medical expert: absent•

Moderate test sensitivity (screening)

Daily
selfchecks

References[1] K. Hosokawa, K. Maruya, S. Nishida, “Testing a novel tool for vision experiments over the internet,” Journal of Vision, Vol. 16, p. 967, 2016.

Contact
Kazushi Maruya Email: cs-liaison-ml at hco.ntt.co.jp
Sensory Representation Research Group, Human Information Science Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

26

How do winners control their mental states?
- Physiological states and sports performance in real games -Abstract
It is said that mental state is an important factor if one is to be a winner in sports. Although mental andphysiological states are related, the relationship between physiological state and sporting performance, especiallyin real games, remains unclear. Here, we investigate this relationship for real competition in esports, baseball, andsnowboarding by focusing on the heart rate (HR) as an indicator of mental state. The results show a strong
relationship between sporting performance and HR, such as the huge variation in HR that occurs when the
opponent is a higher-level player, the stable performance that accompanies a stable heart rate regardless of thesituation, and a top player delivering a good performance with a high HR. Further investigation will reveal thecomponent of the mental state related to performance and will enable us to develop ways of improving athletes’
performance by adjusting their physiological state.

Esports (Fighting game)

Baseball (Pitching)

Snowboard (Jumping)

Relative abilities of opponents
define the HR variation

Stable HR leads to stablehigh-performance pitching

Top players HRs increase to
make a high-scoring jump

Strongerplayer

WeakerplayerPitching error

Simultaneous recordingHighhorizontal
Small error = Good control
0510Minutes

Control： Pitcher A > Pitcher B

Comparable ability pair:
HR synchronization

Good control  Stable HR
Large
Pitcher A
Pitcher B

HR

Pitching error

High

Low

Top players

0510Minutes*The heart rate value was normalized.

Small 130180

Mean HR in an inning (BPM）

Low
Low
High

HR before jumpHigh

Non-top players

Score

Low

Pitcher B

Score

HR

Different ability pair：
Stronger player is stable

Pitcher A
vertical

High

Low

Low

High
HR before jump
*The heart rate value was normalized.

References[1] K. Watanabe, N. Saijo, M. Kashino, “The physiological change reflecting the fight-or-flight response of an esports player correlates strongly with that of the opponent,”in Proc. NEURO2019, 2019.
[2] T. Fukuda, T. Mochida, N. Saijo, M. Kashino, “Game situation affects pitching control - variation of pitching error distribution in real games,” Japan Society of Baseball
Science, the 5th Annual Meeting, 2017.
[3] S. Matsumura, K. Watanabe, T. Kimura, M. Kashino, “Relationship between pre-competitive physiological states and performance in snowboard jumping
competitions,” Japan Society of Ski Science, the 29th Annual Meeting, 2019.

Contact
Ken Watanabe Email: cs-liaison-ml at hco.ntt.co.jp
Sports Brain Science ProjectOpen House 2019
Copyright © 2019 NTT. All Rights Reserved.

Horizontal angl

Horizontal ana ll b a tte r s
10

10

0

0

-10

-10

-20

600

0

100

200

300

400

-20
600 0

500

100

300

400

500

• 両打者と も ， 予測的サッ カ

600

ード が観察さ れた．
Email: kishita.y.aa@m.titech.ac.jp• 打者 1 は顕著な頭部運動が

0
-200
-400
-600
-800
0

600

200

Time after ball-release (ms)200

Angular velocity (deg/s)(deg/s)Angularvelocity (deg/s)
Angularvelocity

Time after ball-release (ms)
200

100

200

300

0

-200

27

400

-400

after
ball-release
TimeTimeafterball
release(ms)
(ms)

s)

Split-second brain function at baseball hitting

-600

-800
600 0

500

100

200

300

400

500

見ら れる が， 打者 2 におい

600

Time
after ballball-release(ms)(ms)Time
after
releasebatter2ては見ら れなかっ た．

Illustration of angular parameters

wn curveballunknown fastballangle (deg)Horizontal
Horizontal angle (deg)

400

500

60

60

40

40

0

100

200

300

400

0
600 0

500

100

Time afterball-release (ms)knownfastball40
unknown fastball 30
20 unknown curveball 20

200

300

400

Horizontal angle (deg)

500

qball

600

ディ スカ ッ シ ョ ンTime after ball-release (ms)40

Horizontal angle (deg)

qeye

20

20

600

after ball-release (ms)2

- Instantaneous cooperation between vision and action -

80

80

0
300

unknown curveball

100

100

30

10
Solid
lines: bat-ball contact 10
Breaklines: swing and miss 0
0

400

500

-10

0

100

200

300

400

400

400

500

0
-200
-400

500

600

700

-600

0

600

100

200

300

400

known fastball
unknown fastball
unknown curveballBatter2
-300

300

400

500

600

• 予測的サッカ ードード が観察された．の着地点の視覚情報を 得て ， それにIn ball games,
it ン
isグ動作をnecessary
to move the body appropriately in reaction to a ball moving at high speed, but the• 打者 スイ1 は顕著な頭部運動が基づき修正する ？予測的サッ カ ード のmechanism
ofれるsuch
is not known. In this study, we examined the brain mechanism that captures a fastが， 打者movement
2 におい
後，見ら
ボールを
見てから
イ ン パク ト ま で の時間差は約
ては見ら
れなかっ
た． time by 1) measuring the eye and body movements of professional baseball players whilemoving ballinalimited
1 0 0 m s以下である ため， スイ ン グ動作へのフ ィ ード バ
they wereッ actually
hitting and 2) performing basic experiments using an optical illusion. By measuring eye and body
ク は時間的に困難．
ディ スカ ッ シ ョ ン
movements
inード
a コscenario
close
to利用．
the 視覚フ
actual• サッ カマ ン ド の遠心性コ
ピ ーをィ game, we succeeded in capturing the sophisticated skills used by top
ードbasic
バッ ク と experiments
比べ， 遅延が遥かに短い．
athletes. In
using the illusion, we clarified how the brain uses visual information to control body
頭部回転と
眼球運動，
ら がよ り よ い戦略か？
movements.
Ourgoal どisちtwofold:to uncover the implicit brain functions for vision and action and to establish a new
なぜ予測的サッ カ ード の開始時刻から イ ン パク ト ま でのtraining method to train people in techniques for optimal body control according to the situation. This will help
な る のか？
improve時間差はほぼ一定と
the motor skillsof a wide range of people, from children to the elderly, as well as top athletes.
-200

-400

-600

-800
600 0

500

比較
-400

200

Time after ball-release (ms)

0

100

after
ball-release
TimeTimeafterball
release(ms)(ms)

fter ballball-release(ms)(ms)terrelease

50

100

200

200

-800
300

-20
600 0

500

Time after ball-release (ms)(deg/s)Angularvelocity (deg/s)
Angularvelocity

300

Abstract
予測的サッ
カもードの機能的意義とは？
• 両打者と， 予測的サッ
カ

-10
-20

600

Angular velocity (deg/s)

300

fter ball-release (ms)

-200

-100
0 contact
Solid lines: bat-ball40

Break lines: swing and miss

200

300

400

500

600

Time
after ballball-release(ms)(ms)Time
after
release

予測的サッ カ ード の機能的意義と は？

30
20
10

• 予測的サッ カ ード の着地点の視覚情報を 得て ， それに

qeye

0

基づき スイ ン グ動作を 修正する ？予測的サッ カ ード の

-10
0

100

200

300

400

500

600

700

50

後， ボールを 見て から イ ン パク ト ま で の時間差は約4030

1 0 0 m s以下で ある ため， ス イ ン グ動作へのフ ィ ード バ

20
10

qeye -300

-400

0

-10
-700
80

-600

-200

-100

ッ ク は時間的に困難．

0

-500

-400

-300

-200

-100

0

-500

-400

-300

-200

-100

0

• サッ カ ード コ マン ド の遠心性コ ピ ーを 利用． 視覚フ ィード バッ ク と 比べ， 遅延が遥かに短い．

60• 予測的サッ カ ード はボールの位置情報によ っ て駆動さ40

頭部回転と 眼球運動， ど ち ら がよ り よ い戦略か？

qhead

20
0
-700
100

-600

な ぜ予測的サッ カ ード の開始時刻から イ ン パク ト ま で のれている ので はない？

時間差はほぼ一定と な る のか？
50

-400

0

-300

qwaist (ms)
time

-200

-100

0

-700

• 予測的サッ カ ード はボールの位置情報によ っ て 駆動さ

予測的サッ カ ード と スイ ン グ動作の間に強固な関係性
How do• professional
baseball players hit a fastball?
れて いる ので はな い？

-50
-600

-500

-400

-300

-200

-100

0

の存在が示唆さ れる ．

イ
ン パク ト ま
始から イ ン パク ト ま定Why can a fastball be hit in a very limited time?

• 予測的サッ カ ード と スイ ン グ動作の間に強固な 関係性

time (ms)の存在が示唆さ れる ．

Experiment Measured eye and body movements using

Experiment Tested whether motor responses to a moving

wearable sensors in a situation close to a real gametarget is driven by position or motion brain information

反射マーカ ー

る 投手の投球を 打撃

ト ラッ カーラ : 1 2 0 f p s, Pu p il-lab s In c.)

打撃

シ ョ ン キャ プ チャ

Motion capturesystemFuzzy edge

Reach for a rotating
moving target

反射マーカ ー

視野カ メ ラ.)(120 fps)

perceived
positionshift*＞n k n o w n ス ト レ ート ， u n k n o w n カ ーブ )につ

n ， u n k n o w n は打者が球種を 事前に知っ て

u p il-lab s In c.)メラ
Field視野カ
camera(200 fps)(120 fps)
fps)
(120

hand

レ ート ， u n k n o w n カ ーブ )につ

は打者が球種をグロ 事前に知っーバルリ てーダー教育院での活動

学)

面着

電

に防

で

Results Good hitters capture a ball by linking eye andモ ロ ッ コ 研修
モ ロ ッ コ を 訪問し ， 現地の学生と のワーク シ ョ ッ プ や， 現地の企業訪問を 実施し た． 文化や国の違いを 体感し ， 国際感覚が滋養し た．

body movements until such a time as no visual
information is reflected in the swing.
験を グロ ーバルリ ーダー教育院での活動

熱し

n of
on

デザイ ン 思考ワーク シ ョ ッ プ
デザイ ン 思考の手法を 学んだ． 対話の中で課題を 十分に見極め， 多
く のア イ ディ ア を 出し ， そこ から ア イ ディ ア を 絞っ て研ぎ澄ま せていく ， と いう 課程は思考の型と し て非常に重要だと 感じ た．

シ ュ タ イ ン バイ ス大学日本研修Time zone wherein visual information
is not reflected in the swing

ド イ ツ のシ ュ タ イ Ball
ン バイ
ス大学の学生と 共に，d日本の中小企業へのrelease
私の特徴
戦略提言を実施． 企業戦略の基本的な考え方を 学び， 国際的なチー• 専門性： 神経科学（ 特に視覚科学） と 機械工学
• 視覚など の人間の感覚に深く 関わる 製品づく り に関心がある ． （ アムでのプロ ジ ェ ク ト を 経験し た．
イ ト ラ ッ キン グ， 運転中のド ラ イ バーの注意， 拡張現実， V R ， 網θpResults Arm movements are first driven by target motion

information and then adjusted by the position
information.
Hand trajectory

膜投影ディ
スプ レ イ ）モロ ッコ 研修
θ= arctan(p/(d-vt))• でき たら 人があま り 手を つけていないテ ーマに取り 組みたい．
国際的な場面で，仕事をのが好き である ． のワーク シ ョ ッ プ や， 現地の企業
モ ロ• ッコ を 訪問し，する
現地の学生と• 変化を 恐れず， 好奇心を も っ て取り 組む． 新し いこ と にチャ レ ン ジ
訪問をた． 文化や国の違いを 体感し ， 国際感覚が滋養し た．
し ，実施し結果を 残す．

Horizontal gaze position θ [deg]バmotionsignals＜*When the edge is fuzzy, the perceived
target position is illusionally shifted
toward the direction of pattern motion.

start
position

Eye tracker

アイ カ メ ラ
シ ュ タ イ ン バイ ス大学日本研修
ド イ ツ のシ ュ タ イ (200ン バイス大学の学生と共に， 日本の中小企業への
fps)
(200
fps)
戦略提言を 実施． 企業戦略の基本的な考え方を 学び， 国際的なチー
ムでのプ ロ ジ ェ ク ト を 経験し た．

Ball position (135 km/h)

デザイ ン 思考ワーク シ ョ ッ プ
Top-team player
デザイ ン 思考の手法を 学んだ．Farm-team
対話の中で課題を十分に見極め， 多playerく のア イ ディ アを 出し ， そこ から ア イ ディ ア を 絞っ て研ぎ澄ま せていく ， と いう 課程は思考の型と し て非常に重要だと 感じ た．

私の特徴

Fuzzy
Sharp

Comparison of responses (“fuzzy” – “sharp”)
Horizontal hand acceleration [m/s2]

チャ
撃アイ カ メ ラ

contrast

る よ う 指示Clear edge

Resp. to “fuzzy” is larger= effect of position info.
response onset

Resp. to “sharp” is larger= effect of motion info.

• 専門性： 神経科学（ 特に視覚科学） と 機械工学
• 視覚など の人間の感覚に深く 関わる 製品づく
り に関心がある．（ アTime to impact[sec]
イ ト ラ ッ キン グ， 運転中のド ラ イ バーの注意， 拡張現実， V R ， 網
Not only visual information but also motor
膜投影ディ スプ レ イ ）commands in the brain are used.
• でき たら 人があま り 手を つけていないテ ーマに取り 組みたい．

Time from start of movement of the target [sec]The bat‘s trajectory starts to change before theball’s position is located.

• 国際的な場面で， 仕事を する のが好き である ．
• 変化を 恐れず， 好奇心を も っ て取り 組む． 新し いこ と にチャ レ ン ジ

References

し ， 結果を 残す．
1] Y. Kishita, H. Ueda, M. Kashino, “Eye movements in real baseball batting by elite players,” in Proc. The 48th Annual Meeting of the Society
for Neuroscience, 2018.
[2] H. Ueda, N. Abekawa, S. Ito, H. Gomi, “Temporal development of an interaction effect between internal motion and contour signals ofdrifting target on reaching adjustment,” in Proc. The 47th Annual Meeting of the Society for Neuroscience, 2017.

Contact
Hiroshi Ueda Email: cs-liaison-ml at hco.ntt.co.jp
Sports Brain Science ProjectOpen House 2019
Copyright © 2019 NTT. All Rights Reserved.

28

Designing technologies for mindful inclusion- How sharing caregiving data affects family communication -Abstract
Previous research has shown that tracking technologies have the potential to help family caregivers optimize theircoping strategies and improve their relationships with care recipients. In this research, we explore how sharing
the tracked data (i.e., caregiving journals and patient’s conditions) with other family caregivers affects home careand family communication. Although previous works suggested that family caregivers may benefit from reading
the records of others, sharing patients’ private information might fuel negative feelings of surveillance andviolation of trust for care recipients. To address this research question, we added a sharing feature to thepreviously developed tracking tool and deployed it for six weeks in the homes of 15 family caregivers who werecaring for a depressed family member. Our findings show how the sharing feature attracted the attention of carerecipients and helped the family caregivers discuss sensitive issues with care recipients.

Development

Goal

Phase 1Development of a tracking tool.
Family caregivers record:
・Mood & activities of the patients・Caregiving activities

Informal CareStressor
Patient

Vicious Cycle

FamilyCaregiver

Reaction

Phase 2・Add a sharing feature to
the tracking tool・Can read others’ records.

Our goal is to design a tool thathelps family caregivers improvetheir care and communication
with their care recipients at home.

Study

To investigate how information sharing aboutcare recipients by family caregivers impacts
family communication
Hypotheses:
1. Learn from other records
2. Gain emotional support
3. Conflict between caregiver-care recipient

Results

Participants (family caregivers):
・14 adults that care for a depressed family member
・11 women, 3 male, average age 43
・6 housewives, 6 full-time employees, 2 self-employed
Patient condition:
・Onset of illness: On average, 8yrs ago, 8 experienced a relapse
・All take antidepressants, all given regular consultation

Posts in phase 2 included morebackground and contextual information

Hypotheses 1 & 2: Supported.
Participants reported improvement of
their coping strategies by learning fromothers. Hypotheses 3: Not supported.

Increased caregiver-care recipient communication- Others’ records triggered communication about depression- Care recipients suggested what to write on caregiving journal- Caregivers indirectly expressed feelings to care recipients
 Importance of supporting indirect communication

References[1] N. Yamashita, H. Kuzuoka, T. Kudo, K. Hirata, E. Aramaki, K. Hattori, “How Information Sharing about Care Recipients by Family Caregivers
Impacts Family Communication,” in Proc. ACM Conference on Human Factors in Computing Systems (CHI), 2018.
[2] N. Yamashita, H. Kuzuoka, K. Hirata, T. Kudo, E. Aramaki, K. Hattori, “Changing Moods: How Manual Tracking by Family Caregivers
Improve Caring and Family Communication,” in Proc. ACM Conference on Human Factors in Computing Systems (CHI), 2017.

Contact
Naomi Yamashita Email: cs-liaison-ml at hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

29Real-world motion that the body sees
- Distinct visuomotor control revealed by natural statistics -Abstract

Movie 𝐼 𝑥, 𝑦, 𝑡
Motion vector
SpatialComput.
integration
model

Self-motion(Angular velocity)

Natural statistics of self-motion& estimation accuracy of proposed model
・ Blurred images provided highest estimation accuracy(center column of scatter plots)
・ Motion signals in low-spatial frequency are important toestimate self-motion

False estimation
Original images

Edge images

Measured angular velocity by gyro sensor (deg/s)

Watching a poster

Distributions of self-motion
Walking

Human studies: implicit motor responses
induced by visual motion[2]Edge image
Blurred image(low-pass filtered） （high-pass filtered）

Manual response

Spatial freq. (cpd)Temporal freq. (Hz)Horizontal angular velocity (deg/s)

Temporal freq. (Hz)・ Spatiotemporal tunings of implicit motor responses
are highly sensitive to low-spatial frequency stimulus

Jogging

Original image

Blur images

Estimated angular velocity (deg/s)

・ We employed a computational model which reproduced
human visual processing properties[1]Jogging

Procedure of self-motion estimation

Walking Watching a poster

Visual motion has critical roles for quickly adjusting posture, eyes, and limbs in dynamic interactions withenvironments. By behavioral experiments and synthetic model simulations, we have tried to reveal fundamental
mechanisms of implicit visuomotor processing. It is difficult to retrieve detailed information about the scene fromhighly blurred image. However, we found that blurred image sequence can provide higher estimation accuracy of
rapid self-motion than the original image sequence. Interestingly, implicit motor responses of hands and eyes arehighly sensitive for low-spatial frequency stimuli. These results suggest that the brain knows the importance of
low-spatial frequency component to code the high-speed self-motion from the statistical relationship between
visual motion and head/posture fluctuation. This type of visuomotor control would be helpful to realize a novel
visual processing for moving robot.

Ocular response
StrongWeak
Spatial freq. (cpd)

The body sees blurred images to estimate rapid
self-motion information

References[1] 中村大樹, 佐藤俊治, “計算論的に最適な速度推定器によってMT野細胞の複雑な反応特性を説明する,” in 第27回日本神経回路学会全国大会, 2017.
[2] H. Gomi, N. Abekawa, S. Nishida, “Spatiotemporal tuning of rapid interactions between visual-motion analysis and reaching movement,” The
Journal of Neuroscience, vol. 26, No. 20, pp. 5301-5308, 2006.
[3] D. Nakamura, H. Gomi, “Statistical analysis of optic flow induced by body motion characterizing OFR and MFR,” in JNNS satellite meeting, 2018.

Contact
Daiki Nakamura Email: cs-liaison-ml at hco.ntt.co.jp
Sensory and Motor Research Group, Human Information Science Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

30

Creating a walking sensation for the seated
A sensation of pseudo-walking expands peripersonal space

Abstract
Body action such as walking is known to extend the subjective boundaries of peripersonal space (PPS; thespace immediately surrounding our body) and to facilitate the processing of audio-tactile multisensory stimuli
presented within the PPS. However, it is unclear whether the boundaries change when a sensation of walking is
induced with no physical body motion. Here, we presented several vibration patterns on the soles of the feet ofseated participants to evoke a sensation of walking, together with a looming sound approaching the body. We
measured reaction times for detecting a vibrotactile stimulus on the chest, which was taken as a behavioral proxy
for the PPS boundary. Results revealed that a cyclic vibration consisting of lowpass-filtered walking sounds
presented at the soles that clearly evoked a sensation of walking decreased the reaction times, indicating that the
PPS boundary was expanded forward by inducing a sensation of walking.
.

A sensation of pseudo-walking created by multisensory stimulation
We have developed a method for creating a sensation ofwalking by combining multisensory information, such as amechanism for moving the upper and lower limbs, a
motorized chair for body sway, and wind and smell
presentation, with the aim of achieving “generating asensation of walking while seated”.

3D video
Arm swing

We focused on the soles of foot and presented avibration to create a sensation of walking.
Wind & smell

PPS expanded by sensation of pseudo-walking
We found that reaction time to a stimulus
approaching toward the body changed when avibration stimulus was applied to the sole of the foot.

Body sway

Lower limb motion

Collaboration with Tokyo Metropolitan University,& Toyohashi Tech University

When a walking sensation was not induced

PPS = Peripersonal Space

Vibrotactile stimuli on soles
Left
sole

Right
sole

When a walking sensation was induced

References[1] Tomohiro Amemiya, “Haptic Interface Technologies Using Perceptual Illusions,” in Proc. of 20th International Conference on HumanComputer Interaction (HCI International 2018), pp.168-174, Las Vegas, NV, July 2018.
[2] Koichi Shimizu, Gaku Sueta, Kentaro Yamaoka, Kazuki Sawamura, Yujin Suzuki, Keisuke Yoshida, Vibol Yem, Yasushi Ikei, Tomohiro
Amemiya, Makoto Sato, Koichi Hirota, Michiteru Kitazaki, “FiveStar VR: shareable travel experience through multisensory stimulation to thewhole body,” in Proc. of SIGGRAPH Asia 2018 Virtual & Augmented Reality, Article 2, Tokyo, Japan, Dec. 2018.

Contact
Tomohiro Amemiya Email: cs-liaison-ml at hco.ntt.co.jp
Sensory and Motor Research Group, Human Information Laboratory
Open House 2019
Copyright © 2019 NTT. All Rights Reserved.

NTT Communication Science Laboratories