Science of Machine Learning
01. People on the WWW, give us your computation each!
- Generating datasets using people and information on the WWW
02. Presenting a quick solution to system failures- Generating recovery-command sequences by neural networks
03. Refinement of spatially aggregated data
- Multivariate Gaussian processes for spatially aggregated data
04. Fast inference of accurate anomaly detectors- Transfer anomaly detection for unseen datasets
05. Anomaly detection with low false positive rate
- Semi-supervised learning for maximizing partial AUC
06. Is the data really biased?
- Testing combinatorial correlation by decision diagrams

Science of Communication and Computation
07. What happens if every player rushes selfishly?
- Equilibrium computation of congestion games
08. Handle a huge quantum world through a tiny window- Investigation of the ability of indirect quantum controls
09. Tuning machine translation with small tuning data
- Domain adaptation with JParaCrawl, a large parallel corpus
10. Assessing children’s emotional development- Investigating developmental changes via multiple cues
11. Creating a personalized picture book
- Support for parent-child picture book interaction

12. How many words do you know?
- Vocabulary size test, Reiwa edition
13. Kyomachi Seika will guide you!
- Training the role-play AI with community cooperation
14. What does he/she think in this situation?
- Sentiment text generation based on personality

Science of Media Information
15. Can you guess the age from this voice?
- Deep speaker attribute estimation with speaker clustering
16. More wireless microphones are available in a room
- BRAVE: Bit-error-robust low-delay audio and voice encoding
17. Pay attention to the speaker you want to listen to (II)- Neural selective hearing with audio-visual speaker clues
18. Controlling voice expression using face expression- Crossmodal voice expression control
19. Learning to search like human- Adaptive spotting for efficient object search
20. Deep learning without data aggregation from nodes- Asynchronous consensus algorithm on distributed networks
21. Cardiac model that makes it heart- Gaussian process with physical laws for 3D cardiac modeling
22. Listening carefully to your heart beat- Cardiohemodynamical analysis based on stethoscopic sounds

Science of Human
23. Make natural-looking illusions by perceptual model- Adaptive motion retargeting for illusion-based projection AR
24. Tiny eye movements reflect cognitive states- Relation of eye-movement dynamics with cognition and pupil
25. Haptic metameric textures- Direct control of perceived texture of 3D printed stimuli
26. What causes emotional change?
- Monitoring emotion in experimental settings and daily life
27. Special cognitive abilities of esports experts- Performance, physiological state, and brain activity
28. Realizing a harmony in rugby scrum- Easy assessment of player coordination with wearable sensors
29. What is a "straight" ball?
- Physical and perceptual attributes of a pitched ball
30. Body representation for quick and skillful action- Uncertainty of hand-state estimate regulates stretch reflex
31. Unconscious is smarter than conscious- Environmental dependency in visuomotor responses

01

People on the WWW, give us your computation each!
Generating datasets using people and information on the WWW

Abstract
Although contents on the WWW are potentially valuable data as training data for machine learning, they are
difficult to use in their current state. Our approach, Browser-based Human Computation (BbHC), offers a costeffective way to extract desirable data from web contents. BbHC enables people to label various web contents
through the web browsers they normally use for web browsing. To accelerate the labeling of data without theinducement of monetary rewards, browser extensions based on BbHC motivate people to continuously engage inlabeling tasks through various human computation techniques. We implemented systems based on BbHC to
explain how it works. Matome supporter helps us to collect labeled images to create an image classifier. Text
monster reduces the cost of annotating word familiarity values for updating a word familiarity database. Multi-voicelabeler’s purpose is to collect writings with speaker information for natural language processing research.

Deep learning requires muchlabeled data
Samples to be labeled Need budget, if you want to buy
them

Building human
computation spaceon the WWW!

World Wide Web （WWW）
Candidates of samples
 A wide variety of data is stored

Potential workers for labeling tasks
 Many people spends much time on
the WWW

Labeling task
 Time-consuming, need efforts ofmany people

Browser-based Human Computation (BbHC)
To collect desirable data from web contents, web browser extensions offer
labeling interfaces to users and motivate users to engage in labeling tasks.
Matome supporter

Text monster

Multi-voice labeler

Users can easily collect images to build
web pages that show a collection ofimages. Collected images are used to
update datasets for image classification.

Users can enjoy collecting Japanesewords which are personified as monsters.
The game results are used to update
word-familiarity database.

Users annotate speaker labels to web
contents so that smartphones can readthem with appropriate voices. Such labels
can be used for natural language
processing research.

Extension shows instructions
for capturing txmon (monster
with Japanese words).

The user simply selects
topic name shown in theright-click context menu.

Web pages associated with
topics are generated.

First, user selects
one of txmons
emerged on the
page.

Then the user
selects sentencesthat the speakerseems to havespoken.

The user then selects fivewords whose wordfamiliarity values ​are close
to the value of txmon words.

First, the user defines
speakers by name,gender, and age.

Finally, the user saves
the labeled data.

References[1] Y. Shirai, Y. Kishino, Y. Yanagisawa, S. Mizutani, T. Suyama, “Building human computation space on the www: labeling web contents
through web browsers,” Proc. The seventh AAAI Conference on Human Computation and Crowdsourcing （HCOMP2019）, 2019.

Contact
Yoshinari Shirai Email: cs-openhouse-ml@hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
オープンハウス 2020

02

Presenting a quick solution to system failures
Generating recovery-command sequences by neural networks

Abstract
We propose a method for automatically generating recovery-command sequences, which is intended to support
quick recovery actions by system operators and to achieve automatic recovery from ICT (information andcommunication technology)-system failures. Our method is based on Seq2Seq (sequence-to-sequence), a neuralnetwork model usually used to solve translation tasks in the field of natural language processing. This model canlearn complex relationships between logs obtained from equipment and recovery commands that operators
executed in the past. When a new failure occurs, our method estimates plausible commands that recover from thefailure on the basis of collected logs. Our method also evaluates the confidence score of the estimated recoverycommand sequences. Operators can use this confidence score as a criterion to determine whether the estimatedrecovery-command sequence should be executed.

Problem： Construction of Recovery rules• Automated recovery requires predefined rules
associating logs/alarms with recovery actions.
• Operators spend a lot of time making
rules and action sequences

Technology： Automatic generationof recovery commands• Input: Logs/Alarms, Output: Recovery commands→ Reducing operation cost
• Confidence score of estimated commands→ Supporting operators’ judgement of commandexecution

Method： Log-Command transformation by Seq2Seq
We constructed a neural network model that converts log/alarm sequences into the corresponding
recovery-command sequences on the basis of historical data by using Seq2Seq , which can learn therelationship between multiple sequences.

Input: Log IDs
・Using a log templater [2] to assign IDs to logs

Output: Words included in recovery commands
・Considering an action such as “Pressing Enterkey” as a word・Evaluating the likelihood of the output sequenceas the confidence score of estimation

References[1] H. Ikeuchi, A. Watanabe, T. Hirao, M. Morishita, M. Nishino, Y. Matsuo, K. Watanabe, “Recovery command generation towards automatic
recovery in ICT systems by Seq2Seq learning,” Proc. of IEEE/IFIP Network Operations and Management Symposium （NOMS）, 2020, toappear.
[2] T. Kimura, A. Watanabe, T. Toyono, K. Ishibashi, “Proactive failure detection learning generation patterns of large-scale network
logs,” IEICE Transactions on Communications, Vol. E102-B, No2, pp. 306–316, 2019.

Contact
Hiroki Ikeuchi Email: cs-openhouse-ml@hco.ntt.co.jp
NTT Network technology laboratories
オープンハウス 2020

03

Refinement of spatially aggregated data
Multivariate Gaussian processes for spatially aggregated data

Abstract
Spatial data collected from cities are often aggregated into geographical partitions (e.g., districts). We propose aprobabilistic model for refining coarse-grained aggregated data by utilizing multiple aggregated data sets withvarious granularities. Our model is based on multivariate Gaussian processes (GPs), in which dependences
between data sets are established by linearly mixing some independent latent GPs. We newly introduce anobservation model with spatial aggregation processes, which allows us to use multiple aggregated data sets forthe refinement task even if they have various granularities. Our model can be used for predicting data values with
arbitrary fine granularity; it is useful for finding key pin-point regions (e.g., poverty area) in a city, efficiently. In the
future, we will extend the model to handle data gathered from multiple cities simultaneously.

Problem : refinement of spatially aggregated data
Coarse-grained data

 TaskFine-grained data

Predicting fine-grained data by utilizing multiple
aggregated data sets with various granularities.

Infer

Ex. Poverty rate IdeaUtilize

Interpolating coarse-grained data by using fine-grained
data that have spatial correlation similar to target data.

 Difficulty
It is not straightforward to evaluate the similarity between
aggregated data sets whose granularity is different.

Various kinds of data sets (Ex. Population, crime rate)

Proposal : spatially aggregated Gaussian processes
We design generative processes of multiple aggregated data sets and train the model from observation data.
Generative
processes

Functions shared by data sets

 Point ① : Spatial interpolation①

Assume the underlying smooth functions (i.e., GPs).

…

 Point ② : Dependences between data sets②

Fine-grained
data (function)

Share the functions gl（x） among multiple data setson the basis of the similarity of spatial patterns.

 Point ③ : Spatial aggregation processes…

③

Average fine-grained data values over regions
Ex. Aggregation in one-dimensional space…
Data 1

Data 2

Data 3Fine-grained data

Observations

Data S
Agg.

Observed aggregated data

References[1] Y. Tanaka, T. Tanaka, T. Iwata, T. Kurashima, M. Okawa, Y. Akagi, H. Toda, “Spatially aggregated Gaussian processes with multivariate
areal outputs,” Proc. 33rd Conference on Neural Information Processing Systems （NeurIPS）, pp. 3000-3010, 2019.

Contact
Yusuke Tanaka Email: cs-openhouse-ml@hco.ntt.co.jp
Service Evolution Laboratory
オープンハウス 2020

04

Fast inference of accurate anomaly detectors
Transfer anomaly detection for unseen datasets

Abstract
We propose a method to improve the anomaly detection performance on target datasets by transferringknowledge on related datasets. Although anomaly labels are valuable to learn anomaly detectors, they are difficult
to obtain due to their rarity. To alleviate this problem, we use anomalous and normal instancesin the related datasets as well as target normal instances. Our method can infer the anomaly detectors for target
datasets without re-training by introducing a novel permutation-invariant neural network. This neural network takes
the set of normal instances as an input and infers the dataset-specific anomaly detector from the set. By learningwith multiple related datasets, our neural network can learn the latent relationship between the anomaly detector
for each dataset and the set of normal instances in the dataset. When target normal instances can be used during
training, our method can also use them for training in a unified framework.

What’s is Anomaly Detection?

Proposed Method

A task to detect anomalous instances
in a dataset.
Anomaly

Normal

•We can detect anomalies accurately
by using normal/anomalous data.

•

We pre-trains this neural network with multiple related datasets.
⇒ It can infer the accurate anomaly detector from normal
instances in the target dataset without re-trainig.
The set of
normal instances

Decisionboundary

NN model１

Related dataset

However, it is difficult to collectanomalies due to their rarity.

We propose a neural network (NN) that infers an appropriate
anomaly detector from the set of normal instances.

Related dataset••

Approach

We use anomalous/normal instances in“related” datasets as well as normal ones
２in the target dataset.

…

Target dataset

Anomalydetector forthe target
datasetTarget Dataset…
Related Dataset 1

Related Dataset 2

We can infer accurate anomaly detector (decision boundary)
without anomalies in the target dataset.

References[1] A. Kumagai, T. Iwata, Y.Fujiwara, “Transfer anomaly detection by inferring latent domain representations,” Proc. 33ｒd Conference on Neural
Information Processing Systems （NeurIPS）, 2019.

Contact
Atsutoshi Kumagai Email: cs-openhouse-ml@hco.ntt.co.jp
Software Innovation Centor
オープンハウス 2020

05

Anomaly detection with low false positive rate
Semi-supervised learning for maximizing partial AUC

Abstract
The partial area under a receiver operating characteristic curve (pAUC) is a performance measurement for binaryclassification problems that summarizes the true positive rate with the specific range of the false positive rate.
Obtaining classifiers that achieve high pAUC is important in a wide variety of applications, such as anomaly detection
and medical diagnosis. Although many methods have been proposed for maximizing the pAUC, existing methods require
many labeled data for training. We propose a semi-supervised learning method for maximizing the pAUC, which trains aclassifier with a small amount of labeled data and a large amount of unlabeled data. To exploit the unlabeled data, wederive two approximations of the pAUC: the first is calculated from positive and unlabeled data, and the second is
calculated from negative and unlabeled data. A classifier is trained by maximizing the weighted sum of the two
approximations of the pAUC and the pAUC that is calculated from positive and negative data.

Partial AUC

True positive rate

Partial Area Under the Receiver Operating Characteristic Curve

PAUC : Area Under the ROC Curve when α≤False positive rate≤β
Evaluation measurement for binary classifiers when positive and negative
sample size is unbalanced
Applications: Anomaly detection and medical diagnosis: reduce costs by
reducing false alarm

1
pAUC（α,β）

𝛽0 𝛼1False positive rate

Existing method（supervised)

Proposed method (semi-supervised)

Inputs labeled data (positive andnegative data), and learns scorefunction 𝑠 by maximizing thepAUC

Derives two approximations of pAUC using unlabeled data, anduses them for learning the score function

Assumes data with high (low) scores as positive (negative) withthe boundary of positive rate 𝜃p
Unlabeleddata

Negative data
Sort byscoresPositive data
0𝛼

𝛽

1

Positive data

0

𝜃P

負例

Sort byscores𝜃P +𝛼𝜃N 𝜃P + 𝛽𝜃N 1Provisionally
positive0𝛼𝛽1Provisionallynegative

𝔼[𝐼（𝑠 𝐱 P > 𝑠 𝐱 N ）]= pAUC1（𝛼, 𝛽）𝔼[𝐼（𝑠 𝐱 P > 𝑠 𝐱 U ）]≈ pAUC2（𝛼, 𝛽）𝜃P ：Positive rate
𝜃N ：Negative rate
𝔼[𝐼（𝑠 𝐱 U > 𝑠 𝐱 N ）]≈ pAUC3（𝛼, 𝛽）pAUC calculated bypositive and negative data

Approximated pAUC
calculated by positive andunlabeled data

Approximated pAUC
calculated by negative andunlabeled data

Objective function
𝐿 = pAUC1 𝛼, 𝛽Objective function: weighted sum of three pAUCs
𝐿 = 𝜆1 pAUC1 𝛼, 𝛽 + 𝜆2 pAUC2 𝛼, 𝛽 + 𝜆3 pAUC3 𝛼, 𝛽References[1] T. Iwata, A. Fujino, N. Ueda, “Semi-supervised learning for maximizing the partial AUC,” Proc AAAI Conference on Artificial Intelligence （AAAI）, 2020.

Contact
Tomoharu Iwata Email: cs-openhouse-ml@hco.ntt.co.jp
Ueda Research Laboratory
オープンハウス 2020

06

Is the data really biased?
Testing combinatorial correlation by decision diagrams

Abstract
We sometimes observe data with structures; population changes of cities on a map, traffic densities of roads on atraffic network, and reactions of sensors on a sensor network. Then, it is a natural question that the observations
depend on the structure or not. Testing combinatorial correlation is a statistical method to answer the question:
however, the test generally requires the exponential time because it considers all possible observations toevaluate the rarity of the current observation. In this research, we propose an efficient testing method using
decision diagrams (DDs) that are a compact representation of a family of sets. We first compress the hypothesis
patterns, which define the structure of the observations, by a DD and then construct another DD that compresses
rare events to evaluate the rarity of the current observations. Our method reduces the testing time from 10^8years to only 1 day in the case of testing binary observations on the Japanese prefecture map.

Combinatorial Correlation

Observations depend on Structure?

Ex2: Sensor Network

Ex1: Hotspot of Diseases

Sensor Reaction

Red areas have so many patients.

1. Intruder?
2. Noise?

White areas have a few patients.
This disease has
1. Hotspot (locality)? Null Hypothesis
2. No Hotspot (Bias)? Alternative Hypothesis※ This observation is just an example.

Difficulty𝑝（𝐾 𝑥 = 𝑘）

Get observations 𝑥 and hypothesis patterns ℱ

2.

Compute Scan statistics 𝐾 𝑥

3.

Compute the P-value of 𝐾(𝑥) by rare events 𝒲

4.

Reject Alt. Hypo. if P-value ≦ Significance level (0.05）ℱ and 𝒲 areNaïve computation takes 10^8 years!!

exponentially huge.

Our method takes only 1 day!!

Combination of A&B is
1. Popular?
2. Coincident?

Rare Event: 𝒘 s.t. 𝑲 𝒘 ≥ 𝑲 𝒙𝑘Histogram of K(x) considering all possible observations

Compute 𝐾(𝑥) and P-value on decision diagrams (DDs) of ℱ and 𝒲.

Scan Statistics
𝑆∈ℱ

Ex３: Shopping History

P-value: the total probability of all rare events.

1.

𝐾（𝑥） = max ෍ 𝑥𝑣

#
3
5
1
1
1

Consider all possible observations to compute the P-value.

Combinatorial Correlation Testing

Proposed method

Hist.
ABC
AB
BC
A
B

Family of Rare Events
Compressed 𝒲 = 𝑤 ∈0,1 𝑉 ∣ 𝐾 𝑤

P-values≥𝐾 𝑥

Compressed 𝑃 = ෍ 𝑝0 𝑤

𝑣∈𝑆

𝑤∈𝒲

Computewithout
Decompression

ComputeConstruct DD from DDℱ

𝒲

Decision Diagram

Decision Diagram

without
Decompression

References[1] M. Ishihata, T. Maehara: “Exact Bernoulli scan statistics by binary decision diagrams,” The 28th International Joint Conference on
Artificial Intelligence （IJCAI-2019）, 2019.

Contact
Masakazu Ishihata Email: cs-openhouse-ml@hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
オープンハウス 2020

07

What happens if every player rushes selfishly?
Equilibrium computation of congestion games

Abstract
In a road or telecommunication network, an edge (link) may be congested and may incur more cost if many
people use it. We can compute how much each edge will be congested if each user of such network chooses apath (route) selfishly, i.e. chooses a path with minimum cost without any guidance or control. The computation ofcongestion requires calculation of probability for each of all available paths, which is generally prohibitive because
there are a great many number of paths. To overcome this problem, we use a data structure called binary
decision diagram to represent all available paths compactly. With this data structure, we can speed up thecomputation dramatically, and can compute such state of a network with realistic size. This study enables us topredict the state of congestion for a road and telecommunication network in a simple way, which is useful for
designing such networks.

Congestion Games and Equilibrium State
Goal

Start
- There are infinite num. of players.
- Each player chooses a path.
- Chooses a path with lower cost.
- Available paths are fixed.
- Edge cost increases w.r.t. mass of using players.

Solution with Decision Diagrams
Using zero-suppressed binary decision diagram (ZDD),
all available paths can be represented in compact form
- e.g. 8 quadrillion paths may be represented in 1MB-> What path is chosen by each player?
Equilibrium state : each player chooses a path with(currently) minimum cost from available paths
Example:Non-equilibrated

At equilibrium

50%
0.5 1
1 0 0.5

1
1

⊥

0

1

1
1 0

1

Key 1: the path with minimum cost among available ones
can easily be computed using ZDD
0

100%

50%

*Blue：edge cost is fixed*Green：edge cost is equal to proportion of players using it

Computing Equilibrium

1

1

Example of restriction：
Only paths within $10 are available

At equilibrium

0.5 1
1 0 0.5

1
0
$10

50%

0
0 0

0

⊥

0
0.8 T

0.8

1

Key 2: equilibrium can be computed by performing thefollowing steps repetitively:- computing path with currently minimum cost
- augmenting players using this path

50%

$10
1

1

Increasing cost w.r.t.
mass of using players- All paths are available -> easy to compute- Available paths are restricted -> difficult to compute
since all available (enormous) paths should be enumerated

0.2

1
0.2

0
0.5

T

0.2

20%

70%

1

1

0
0.8

0.7

0
0.3

40%
0.4

1

1

1

80%

30%

60%

1
0
0.6
…

References[1] K. Nakamura, S. Sakaue, N. Yasuda, “Practical Frank—Wolfe method with decision diagrams for computing Wardrop equilibrium of
combinatorial congestion games,” Proc. 34th AAAI Conference on Artificial Intelligence （AAAI）, 2020.

Contact
Kengo Nakamura Email: cs-openhouse-ml@hco.ntt.co.jp
Linguistic Intelligence Research Group, Innovative Communication Laboratory
オープンハウス 2020

08

Handle a huge quantum world through a tiny window
Investigation of the ability of indirect quantum controls

Abstract
It is difficult to manipulate an entire large quantum system directly. When we try to do so, huge noise will be injected
into the system. However, if we can indirectly control the system via a restricted part of it, we will be able to suppress
the injected noise. In this research, we investigated the effect of the restriction mathematically, and succeeded incompletely categorizing the set of operations in the case of indirect control. This result indicates that, if the degree ofthe freedom of the controllable part is more than two, we can universally control the whole quantum system in effecteven when the degree of freedom of the uncontrollable part is very large. This knowledge provides a new strategy
for constructing a noise-less quantum computer or any other noise-less device for quantum information processing.
If we can construct such a device, we can realize quantum information processing, e.g. factorization of huge
numbers with a quantum computer.

Current Quantum Computer

Noise Injection
Noise is injected from controllable parts.

Intrusion path of noise•••

Company I and Company G construct 53-qubit
quantum computers with the noise.
Noise increases as the number of qubits
increases.
To factorize a thousand-bit integer, millions ofqubits are needed even when the noise is small.
Theoretical approach for noise reduction

A quantum computer can be made as acomposite system consisting of a directly
controllable component and a directly
uncontrollable component with a spontaneousinteraction between them.

Directly controllable
Directly uncontrollable

New strategy: Prevent noise injection at
the sacrifice of controllability－Question－
Can the whole composite system becontrolled even if the directly
controllable component is small?

Noise reduction?!
Ex.: NV center in diamond
Ex.: Superconducting qubit
Discovered evidence: When the degree of freedom of the controllable component is more thantwo, we can completely control the whole system in effect for any
spontaneous interaction, though we can not when it is two.

With this strategy, the noise could be suppressed even if the total number of
qubits is drastically increased!

Mathematical basis
In the case of indirect control
of a quantum system, we
prove the fact that a set of all
the executable operations(matrices) for a fixedinteraction can be written as

𝐿≈ℒ（𝑠𝑢 𝑑𝑆 ⊗ 𝐽 ∪ 𝑖 𝐼 ⊗ 𝐽, 𝐽 ）where the set 𝐽 must satisfy acertain condition.
The condition depends onthe degree of freedom of thecontrollable component.

References[1] G. Kato, M. Owari, K. Maruyama, “Algebra and Hilbert space structures induced by quantum probes,” Annals of Physics, 412 （2020）
168046[2] G. Kato, M. Owari, K. Maruyama, “Hilbert space structure induced by quantum probes,” Proceedings of the 11th Italian Quantum
Information Science conference （IQIS2018）, Catania, Italy, 12 （2019） 4

Contact
Go Kato Email: cs-openhouse-ml@hco.ntt.co.jp
Computing Theory Research Group, Media Information Laboratory
オープンハウス 2020

09

Tuning machine translation with small tuning data
Domain adaptation with JParaCrawl, a large parallel corpus

Abstract
Recent machine translation algorithms mainly rely on parallel corpora. However, since the availability of parallel
corpora remains limited, only some resource-rich language pairs can benefit from them.
We constructed a parallel corpus for English-Japanese, for which the amount of publicly available parallel corpora
is still limited. We constructed the parallel corpus by broadly crawling the web and automatically aligning parallel
sentences.
Our collected corpus, called JParaCrawl, amassed over 10 million sentence pairs. JParaCrawl is now freely
available online for research purposes.
We show how a neural machine translation model trained with it works as a good pre-trained model for fine-tuning
specific domains and achieves good performance even if the target domain data is limited.
Training Machine Translation Model
Please clean the room.
May I have a receipt?
I have a cold.

Train

部屋を掃除してください。
領収書をいただけますか？
風邪をひいた。

・ Machine Translation (MT) model
learns automatically from the parallel
sentences (parallel corpus).
→ The amount of parallel sentences is
the key to its accuracy.

Parallel corpus（Training data）

・ We need a large parallel corpus foreach domain to make a practical MT
system, but such domains are limited.

Translationmodel

・ Our purpose is to accurately translate
low-resource domains, which only havea small number of parallel sentences.

Train

How to create a large En-Ja parallel corpus

Domain adaptation with small tuning data

Large En-Ja parallel corpus
JParaCrawl （10M sentences）
Pre-train

Crawl the web
Find parallel sentences（150k websites, 14TB）・ We created a large-scale English-Japanese parallel
corpus “JParaCrawl” that contains more than 10M
sentences by largely crawling the web and automaticallyaligning the parallel sentences.

JParaCrawl general purpose MT model
A small parallel corpus of
the target domain(e.g. medical domain)・ Until now, freely available parallel corpora are up to
3.0M, so this is more than three times larger than theprevious largest one.
・This corpus covers broad domains since it
is based on the web.
Our corpus is freely available online
for research purposes.
http://www.kecl.ntt.co.jp/icl/lirg/jparacrawl/

Fine-tune

Domain-adapted MT model
・ Even if the amount of the target domain parallel corpus issmall, we can achieve good performance with the
combination of JParaCrawl.
・ Fine-tuning only needs small computational cost.

References[1] M. Morishita, J. Suzuki, M. Nagata, “JParaCrawl: A large scale web-based Japanese-English parallel corpus,” Proc. 12th International
Conference on Language Resources and Evaluation （LREC）, May 2020.

Contact
Makoto Morishita Email: cs-openhouse-ml@hco.ntt.co.jp
Linguistic Intelligence Research Group, Innovative Communication Laboratory
オープンハウス 2020

Assessing children’s emotional development

10

Investigating developmental changes via multiple cues

Abstract
Understanding one’s own and others’ emotions is an essetial skill in interpersonal communication. To investigate
the development of emotion understanding in children, we computerized the Affect Knowledge Test (Japaneseversion) and developed an emotion voice test. These assessment tools allow us to examine how children
percieve and process multiple emotion cues and to objectively quantify their development. Compared to the
traditional method in which only trained experimenters can administer assessments to children, the computerized
tests made it possible to assess children’s emotion understanding in any setting without experts. Using the tests,
researchers as well as teachers would be able to detect children with developmental delays and those who
struggle with interpersonal communication, and to find which emotional cue is difficult for them to percieve.
Obtaining this information would allow us to develop an appropriate intervention to facilitate their development.

Developing assessment tools for children’s emotion understanding
Emotion Voice Test

Emotion cues
How doeshe feel?

For age 3-12

Voice

Word
Facialexpression

Randomly presents
45 vocal stimuli

Context

Ask the child to selectan appropriate emotion(15 expressions*3 emotions）*Originally developed in US

Affect Knowledge Test* (AKT: Japanese version） For preschoolers (age 3-5)
Show mehappy face

Emotion labeling

Emotion matching

Emotion development in 3-5 years old

Emotion labelingis more difficult
then other tasks

Situation knowledge

Participants:
116 Japanese 3 to 5-year-olds(65 boys)

Automatic scoring

Typically developing children
Developmentally delayedchildrenEmotion matching

Score

Score

Emotion labeling

How does he
feel?

Watch a short scenario
acted out with puppets

Can detect children with
developmental delays

Situation knowledge

Score

Practice phase

How does hefeel?

Age (month)Age (month)Age (month)M = 1.27, SD = .33

M = 1.76, SD = .33

M = 1.76, SD = .26

References[1] N. Watanabe, S. A. Denham, N. M. Jones, T. Kobayashi, H. H. Bassett, D. E. Ferrier, “Working toward cross-cultural adaptation: Preliminary
psychometric evaluation of the Affect Knowledge Test in Japanese preschoolers,” SAGE Open, 2019.
[2] N. Watanabe, T. Kobayashi, “Computerization of an emotion knowledge assessment for preschoolers: Supporting their school readiness,”
Proc. International School Psychology Association 41st Annual Conference, 2019.

Contact
Naomi Watanabe Email: cs-openhouse-ml@hco.ntt.co.jp
Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
オープンハウス 2020

11

Creating a personalized picture book
Support for parent-child picture book interaction

Abstract
Previous research on developmental psychology have extensively shown that picture book reading promotes childlanguage development. To support a parent-child interaction for language development, we propose a method tocreate a personalized picture book that suits child’s interest and vocabulary level. On the application site, parents
answers the words that their child can produce and what they are interested in. Then, we try to estimate the words
that the child is likely to produce based on statistical models in child vocabulary development database, to arrange
each object picture for the estimated words, and to customize a story to match the child’s interest. We are currently
conducting a field trial in Onna village, Okinawa, to distribute a personalized picture book at a public library forparent and child who participate in health check-up. Through the field trial, we are investigating whether the
personalized picture book contribute to the increase of parent-child interaction and book reading activity.

■Personalized Picture Book

① Appearing a main character to match child’s name and gender

Akio Kashiwara, NTT Print Corp.
Aki-chan（girl）
Hiro-kun（boy）② Customizing a story to match child’s interests
Elephant
story

Target Age: 1-2 years of age

Dog
story

Enter personal information from site
https://ehon.nttprint.com③ Arranging pictures for words that the child is about to produce

・child’s name、boy/girl・favorite objects
・vocabulary checklist（productive words）
Child Vocabulary
Development
Databaseelephant■Field Trial in Onna Village

with the corporation of NTT Print Corp
Feb 2020 – March 2021

ant

ladybird

butterfly

・Lead parent and child to a public library・Give opportunity to encounter picture books

Inform parents at health check-up

Apply for personalized picture book

（Onna Health & Welfare Center）（Onna Culture & Information Center）

Creating and sending
the book for 10 dayscatmonkeyFamiliarize a child with booksby personalized picture book

Support book reading
activity at home

References[1] NTT, NTT Print Corp., Onnna Village, “Joint field trial about parent-child activity using personalized picture books in Onna Village, Okinawa,”
Topics in NTT Group Web Site (https://www.ntt.co.jp/topics/oyakoehon/index.html), 2020.
[2] NTT Print Corp. & Ehon Navi, “Parent-child interaction using personalized picture books created from psychological research in childlanguage development,” Ehon Navi (https://style.ehonnavi.net/ehon/2020/03/17_358.html), 2020.

Contact
Tessei Kobayashi Email: cs-openhouse-ml@hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
オープンハウス 2020

12

How many words do you know?
– Vocabulary size test, Reiwa edition –Abstract
More than 20 years ago, NTT conducted psychological experiments to investigate word familiarity and thus
construct a Japanese lexicon of about 77,000 words. This lexicon is still used in many fields.
Now, we reinvestigated and reconstructed an unparalleled-scale Lexicon of 163,000 words throughcrowdsourcing. By applying careful screening, we succeeded in obtaining highly reliable results. This makes it
possible to make comparisons with results of 20 years ago. Based on this, we created the Reiwa edition of a
vocabulary-size estimation test. Furthermore, this lexicon allows us to estimate the vocabulary size appropriate
for the present day. We are working on vocabulary-size investigations for a wide range of ages, including
elementary school to high school students. In the future, we will investigate and analyze the relationship between
vocabulary size, reading comprehension, and academic ability, aiming to achieve effective educational support.

Word Familiarity Database, Reiwa edition

Word Familiarity Examination via Cloud
Ask good workers
to evaluate other sets
Pre-screening

Cloud workers

Comparison of Histograms with previous data

Post-screening

Rating word familiarity

Holder of official Kanji
test, level 2 or above, or
top performers in original
Kanji reading test

Evaluate 1000–2000
words/set at a time
(5% of words are
duplicated)

Point ① Earnest andeager evaluators who
have knowledge of Kanji

Vocabulary-size estimation

Cloud workers
Correlation
coefficient of
duplicated evaluation
is more than 0.5

Point ② About 163,000 targetwords, including new words,
e.g., smartphone, internet,
heterodimer, laba…
Estimated vocabulary size,
Sample user Ａ

MorefamiliarLessfamiliar

Estimated results for each grade

Please select words thatyou know.
□ bank□ charge…
□ intimacy□ recast
Point ③ Checklist of
only about 50 words
Vocabulary-size test:
http://www.kecl.ntt.co.jp/icl/lirg/resources/goitokusei/

Point ④ About 4,600 persons,
including elementary/juniorhigh/high school students

References[1] S. Fujita, T. Kobayashi, “Reexamination of word familiarity and comparison with past examination,” The 26th Annual Meeting of the
Association for Natural Language Processing, pp. 1037-1040, 2020.
[2] S. Fujita, T. Kobayashi, T. Yamada, S. Sugawara, T. Arai, N. Arai, “Vocabulary size of elementary, junior high and high school students
and analysis of relationship with word familiarity,” The 26th Annual Meeting of the Association for Natural Language Processing, pp. 355358, 2020.

Contact
Sanae Fujita, email: cs-openhouse-ml@hco.ntt.co.jp
Linguistic Intelligence Research Group, Innovative Communication Laboratory
オープンハウス 2020

13

Kyomachi Seika will guide you!
Training the role-play AI with community cooperation

Abstract
We propose a “role-play AI” that can respond to inquiries, receptions, and guides at city offices.
Conventional AI training requires a lot of accurate training data, and data collection has been an extremely
costly and difficult task. In this study, we solve this problem by utilizing community cooperation. By making
the data collection work a community cooperation activity, we collect accurate training data at a low cost. By
connecting “people who live in the area” and “people who interested in the area,” we have collectedvery high-quality training data, and we have made possible the training of role-play AI that is closely
linked to the community. By using this technology, we provide “role-play AI” to learn according to local
demand. In the future, we aim to realize AI technology that can be used in a wide range of situations, notonly by local governments.

The efficiency of business operations is being improved through AI. In particular, the realization of AI,
which provides information to users through dialogues such as receptionists and guide, is attracting
attention as a means of reducing the burden on human and promoting collaboration with people and AI.
In this research, we realize the useful AI in the real world through joint experiments with Seika-town.
Task-oriented dialogue with the role-play AI"Role-play AI" is a dialogue system that imitates the character. Having an official corporatecharacter or a celebrity serve as a receptionist or guide is expected to improve customer
satisfaction, provide entertainment, and reduce the burden on the receptionist.
Where can I get a certificate of residence?

Training data
You can get it on the reception desk.
A reception desk is located on the second floor.

Data collection as community cooperation

Official
character(Traditionally) handcrafted→Problem：high costs

We solved the problem of the data collection by matching "people with interest" and "people withknowledge“. →The data collection by residents of Seika-town is carried out as community cooperation.
Question from people who want to live or visit Seika-town.
“What's famous in Seika-town in spring?”

Answer by people who lives in Seika-town.
“In spring, they are famous for picking strawberry!”■POINTS:
Training data collection
・Question from people
who interested＝Demand information
・Answer by people
who lives in Seika-town.
＝Accurate information
Data collection with
community cooperation＝Low cost and high quality

References[1] R. Higashinaka, et al. "Role play-based question-answering by real users for building chatbots with consistent personalities." Proceedings ofthe 19th Annual SIGdial Meeting on Discourse and Dialogue. 2018.

Contact
Masahiro Mizukami Email: cs-openhouse-ml@hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
オープンハウス 2020

14

What does he/she think in this situation?
Sentiment text generation based on personality

Abstract
In order for a dialogue system to have a good relationship with humans, it is essential to understand andexpress human emotions. In previous studies, the expression of emotions has been done in typologies suchas joy and anger. In contrast, this research will realize a dialogue system that can better understand andexpress human emotions by predicting the feelings associated with people like "what does he/she think thissituation." This model take "who" and "what they did" as input and estimate the output of "how they feel"according to a particular "person". This allows the dialogue system to use a more flexible way of expressing
impressions, depending on the person. In the future, we will realize a method that learns "characteristics ofthe expression of impressions" from users in real time during the dialogue, and estimates the kind of personfrom the user's characteristics of the expression of impressions.

Sentiment text generation

is a technique for estimating how he/she feel when an event occurs.

Previous works: Estimating whether the majority believes it positive or negative
Issue: Different people have different sentiment texts of the same event= Necessary to change the output according to the person

I’m tiredWe went on a family hike.

model
It’s a goodmemorySentiment text generation model using the conditional attention mechanism
We focus on the attention mechanism → Introduce a filter that reflects personality

Sentiment text

It is a good memory .

I am tired .

family, hike→positive

hike→negative

Father’s filter

Son’s filter

We went on a family hike

We went on a family hike

Decoder
Attentionmechanism

Encoder

Event■POINT・Learn “filter that reflects attentional features” for each person
・Controlling “which words to focus on” and “how to understand thewords” on person-by-person basis

References[1] M.Mizukami, H. Sugiyama, H. Narimatsu, “Event data collection for Recent Personal Questions,” in Proc. LACATODA, 2018.
[2] M.Mizukami, H. Sugiyama, H. Narimatsu, T. Arimoto, R. Higashinaka, “話者情報を考慮する注意機構を用いた応答生成手法の検討,” 人工知能学会, 2020.

Contact
Masahiro Mizukami Email: cs-openhouse-ml@hco.ntt.co.jp
Interaction Research Group, Innovative Communication Laboratory
オープンハウス 2020

15

Can you guess the age from this voice?
Deep speaker attribute estimation with speaker clustering

Abstract
Estimating speaker-attributes such as age and gender is an important task with a wide range of applications.
While the recent proposed deep neural network models have been achieving high performance, the estimatedresults tend to be less reliable because of the overfitting problem. In order to solve this problem, we propose ageneral framework for correcting the unreliable results of the arbitrary speaker-attribute estimation models. The
proposed algorithm first applies speaker clustering to the target utterances to detect similar speakers of target
utterances. Then, the speaker-attribute class of each cluster is determined by voting on the utterances assigned
to the cluster. Finally, we can correct the result of unreliable utterances by replacing their result with the clusters’
speaker-attribute class. Our approach is evaluated on age-gender classification and gender regression tasks,
yielding significant improvements in classification accuracy and mean absolute error.

Deep speaker attribute estimation

Error correction based on speaker clustering

Problem definition

Correct estimation results of deep learning modelby majority voting of results of similar speakers

Estimate speakers’ age and gender from their voiceswith deep learning modelSet of target utterances

Applications

Deep learning
model

Call center response decision, marketing support,
realization of a voice dialogue system that changes
behavior according to user attributes, etc.
1-1. Apply clustering
based on speakersimilarity
Age: ??
male / female

Deep speakerattribute estimation
model

Difficulties of task
• There is large deviation inamount of training data foreach age-group (Fig.1)• Models tend to be
overfitting to specificspeaker / age (Fig. 2)

Fig.1 Age histogram of
NIST-SRE08● Male_Child● Male_Adult● Male_Elderly
× Female_Child× Female_Adult× Female_Elderly

1-2. Estimate age classof each utterance bydeep learning model

Adult → Elderly(Replaced with cluster’s class)
2. Decide age class ofElderlyeach cluster by voting
C1 = Elderly
Child
Elderly
3. Replace result of eachSpeaker
C2 = Childutterance with age classclusterof belonging cluster
Child
Adult → Child(Replaced with cluster’s class): Target speech and estimated class
C*: Class decided by voting for each cluster
Evaluation criteria

Conventional

Proposed

Classification accuracy(Child, Adult, Elderly)

59 %

72 %

Mean absolute error± 10.9 years± 8.7 years

Fig.2 Visualization of output of age-estimation model(cited from [1]) (left: training data, right: evaluation data)

References[1] N. Tawara, H. Kamiyama, S. Kobashikawa, A. Ogawa, “Improving speaker-attribute estimation by voting based on speaker cluster
information,” Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing （ICASSP）, pp. 6594-598, May 2020.
[2] N. Tawara, H. Kamiyama, S. Kobashikawa, A. Ogawa, “Frame-level phoneme-invariant speaker feature extraction for text-independent
speaker recognition on extremely short utterances,” Reports of the autumn meeting the Acoustical Society of Japan., pp. 815-816, Sept.
2019.

Contact
Naohiro Tawara Email: cs-openhouse-ml@hco.ntt.co.jp
Signal Processing Research Group, Media Information Laboratory
オープンハウス 2020

16

More wireless microphones are available in a room
BRAVE: Bit-error-robust low-delay audio and voice encoding

Abstract
We have developed a bit-error-robust speech-and-audio codec working in low-delay conditions. Inter-device
audio transmission, such as in microphones, requires strict real-time processing. It is a challenge to enhance thecompression efficiency in such conditions, which enables us to use more microphones at once in a room.
Sometimes, this kind of inter-device transmission encounter errors occurring in the encoded data, and codecs
have to deal with them to avoid severe decoding errors. Especially in low-delay conditions, it is hard to protect
codes with additional information keeping the bitrates. Therefore, we proposed a bit rearrangement technique,which makes lower the impact of the errors compressing data efficiently. Using this technique, the developed
codec BRAVE can compress speech and audio data in a very short time and is robust for bit errors. It is thus
expected to be useful also for other use cases such as the Internet of things (IoT).

Speech and audio codec BRAVE

Technical difficulty

We developed a speech–and-audio coding scheme for real-time Errors may happen in the codes during transmission- Many codecs protect codes by using packets
inter-device audio transmission like wireless microphones- Packets need frame-wise headers- Headers weigh too much in low-delay conditions
Input: 48-kHz 16-bit monaural We want to deal with errors without using packets
Bitrate: 96 kbps (About halfHelloof the conventional one)
…Hello Hello
Algorithmic delay: 1.5-3.0 ms

Lower bitrates allow us to use more microphones at once

Proposal: Bitplane rearrangement

Suppressing error effects

Avoiding serious impacts on the sound quality

Problem in the conventional bit assignment
Encoder

Short-time processing

Bit assignment in the proposed method

Decoder(with errors)

Encoder

Decoder(without errors)

Decoder(with errors)

Input

Binaryof input(bitplane)Bitsassignedfor input

Error

Error

Error

Error

Code
Variable separation

Constant separation

Errors may cause serious noise
wherever in a time frame they occur

Most significant bits where bit assignments are true
are guaranteed even when errors occur in the frame

References[1] R. Sugiura, Y. Kamamoto, T. Moriya, “Spectral-envelope-based least significant bit management for low-delay bit-error-robust speech coding,”
Proc. IEEE International Conference on Acoustics, Speech and Signal Processing （ICASSP）, 2018.

Contact
Ryosuke Sugiura Email: cs-openhouse-ml@hco.ntt.co.jp
Moriya Research Laboratory
オープンハウス 2020

17

Pay attention to the speaker you want to listen to (II)Neural selective hearing with audio-visual speaker clues

Abstract
Human beings have the ability to concentrate on listening to a desired speaker (= selective hearing) even when
multiple people are speaking at the same time. The purpose of this research is to realize the selective listening
mechanism of human beings on a computer. In this research, we propose multimodal selective hearing technology that uses video information as the target speaker’s clues in addition to audio information. By utilizing multiple
information sources like humans, the technology become advanced that can operate stably even in situations,where audio clues are useless, such as conversations between speakers with similar voice characteristics. This
technology will become fundamentals of various devices that take human voice as input. For example, it will contributes to the realization of robots and smart speakers that recognize people and change their response.

Selective Hearing with Audio Speaker Clue Selective Hearing
・ Ability to focus on listening to desired speakerfrom mixture signals
・ In daily conversations, multiple speakers often
speak at same time
⇨ Humans easily perform such selective hearing,but it is difficult for conventional computers⇨ First proposal of neural selective hearing withaudio speaker clue （OPEN HOUSE 2018）

 Problem
With audio clues, extraction performance degrades for
mixture signals with similar voice characteristics
voice
characteristics
of desired speaker

Selective
Hearing

Utilization of Audio and Visual Speaker Clues
 SpeakerBeam （= Selective Hearing based on Deep Learning）  Solution: Proposal of Multimodal SpeakerBeam
In addition to voice characteristics (audio info.),
Deep learning-based model, which extracts
use mouth motion (visual info.) as speaker clues
desired speaker’s voice from mixture signal
given by target speaker’s clue
extracted signal
utilize multi-modal
information astarget speaker’s clue
clue selection
according tosituation⇨ utilize multi-modal information like humans
 Expected effect・ Performance improvement by utilizing multiple modality
・ 複数モダリティの活用による性能向上
・ e.g.,
Robustness
improvement against lack of speaker clues
同性の混合音声であっても上手く抽出出来る
Audio clue is useless（similar voice characteristics）
・ e.g.,
モダリティの劣化や欠損に対する頑健性向上
e.g.,clue is missing （face not detected）
e.g., Visual
登録時と使用時で声の特徴のミスマッチがe.g.,
Possible
to extract even in above situations
e.g., あっても抽出出来る＊ About target speaker’s clue

Pre-recorded audio dataof target speakeraudio clue visual clue
mixture signal

target speaker

Video data (around mouth) recordedsame time as mixture signal

References[1] T. Ochiai, M. Delcroix, K. Kinoshita, A. Ogawa, T. Nakatani, “Multimodal SpeakerBeam: Single channel target speech extraction with audiovisual speaker clues,” Proc. Interspeech, 2019.
[2] K. Zmolikova, M. Delcroix, K. Kinoshita, T. Ochiai, T. Nakatani, L. Burget, J. Cernocky, “SpeakerBeam: Speaker aware neural network fortarget speaker extraction in speech mixtures,” IEEE Journal of Selected Topics in Signal Processing, 2019.

Contact
Tsubasa Ochiai Email: cs-openhouse-ml@hco.ntt.co.jp
Signal Processing Research Group, Media Information Laboratory
オープンハウス 2020

18

Controlling voice expression using face expression
Crossmodal voice expression control

Abstract
There are many kinds of physical or mental barriers that prevent individuals from smooth verbal communication.
One key technique to overcome some of these barriers is voice conversion (VC), a technique to convert
para/non-linguistic information contained in a given utterance without changing the linguistic information. Here, we
propose a crossmodal voice control system, which offers a way to control the vocal expression of emotion in
speech through the facial expression in a face image. The proposed system consists of performing facial
expression recognition (FER) followed by VC. For VC, we have developed a method based on sequence-tosequence (S2S) learning, which is designed to convert the prosodic features as well as the voice characteristics
in speech conditioned on the output of the FER system. We believe that this work can provide some insight onwhat it is like to be able to control our voice through different modalities.

Communication augmentation system

Voice expression control through face

Using voice conversion (VC) technique to help overcome
barriers that prevent us from smooth communication"Hello..."SenderReal-time conversion
to different styles"Hello!!"Receiver
Crossmodal voicecontrol consisting offacial expression
recognition and VC

to make communication smoothersadangry happy

Conditional sequence-to-sequence VC

Facial expression recognition (FER)
 FER using attentional convolutional network• After prediction, output is passed to VC system

Face
image

Decoderlearn to make prediction

 Sequence-to-sequence (S2S) learning
Attention• Offers a general framework for transforming
one sequence into another variable length
sequence• Encoder/decoder structure and attention
Sourcemechanism make it possible to learn
encoderconversion rules that reflect long-term
dependencies in input/output sequences• Usually requires large-scale parallel corpora
Target VC based on S2S learning (S2S-VC) [1,2]encoder
Source speech• Voice expressions are characterized byprosodic features (e.g., intonation and rhythm) feature sequence• S2S-VC is able to convert prosodic features
Target speech𝑘′
as well as voice characteristics in input
feature sequence
speech with limited amount of training data𝑘′
Target emotion
class conditioning

Emotion class
FER

𝑘

"sad"

References[1] H. Kameoka, K. Tanaka, T. Kaneko, N. Hojo, "ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion," IEEE/ACM
Transactions on Audio, Speech, and Language Processing, in Review.
[2] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo, "AttS2S-VC: Sequence-to-sequence voice conversion with attention and context preservation
mechanisms," Proc. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2019), pp. 6805-6809, May
2019.

Contact
Hirokazu Kameoka Email: cs-openhouse-ml@hco.ntt.co.jp
Media Recognition Research Group, Media Information Laboratory
オープンハウス 2020

19

Learning to search like human
Adaptive spotting for efficient object search

Abstract
We propose Adaptive Spotting, a deep reinforcement learning approach to object search from a scene represented
by a 3D point cloud map. A straightforward approach using exhaustive search is often not promising due to poor
computational efficiency. To solve this problem, our approach simultaneously learns the features of a given object
and its efficient search path. Our network is designed to have a pose estimation module to estimate promisinglocations to be explored. The network is trained in an end-to-end manner to learn efficient search paths by using areinforcement learning strategy that gives a higher reward when it finds the target in fewer search steps. Evaluation
results demonstrate that our approach outperforms several state-of-the-art methods in both search accuracy andthe number of search steps required. It is expected to be used in areas such as logistics, manufacturing, andtransportation, which require the ability to search for objects in 3D space fast and accurately.

Point Cloud Search
Search for object with certain shape in point cloud map ofspace captured by 3D sensor (LiDAR, etc.)
Space often huge and non-uniform⇒ Exhaustive search is often undesirable.

Adaptive Spotting

Point cloud map
Deep reinforcement learning for joint learning of features and efficient search paths
Point 1.
Point 2.

Pose estimation module, which estimates promising locations to be searched,
allows the search algorithm itself to determine the next location to search.
Deep reinforcement learning that gives higher rewards for finding targets in
fewer steps makes it possible to learn an efficient search path.

Examples of Search Paths (Searching chair in office room)
Possible to find target within a few steps

Target4

Performance improved significantly

Target4
Source of point cloud data：Stanford 2D-3D-Semantics Dataset http://buildingparser.stanford.edu/dataset.html

References[1] O. Krishna, G. Irie, X. Wu, T. Kawanishi, K. Kashino, “Learning search path for region-level image matching,” Proc. IEEE International
Conference on Acoustics, Speech and Signal Processing （ICASSP）, 2019.
[2] O. Krishna, G. Irie, X. Wu, T. Kawanishi, K. Kashino, “Deep reinforcement template matching,” Meeting on Image Recognition andUnderstanding （MIRU）, 2019.

Contact
Onkar Krishna Email: cs-openhouse-ml@hco.ntt.co.jp
Recognition Research Group, Media Information Laboratory
オープンハウス 2020

20

Deep learning without data aggregation from nodes
Asynchronous consensus algorithm on distributed networks

Abstract
While use of massive data benefits on training DNN models, aggregating all data into one physical location (e.g. a clouddata center) may not be possible due to data privacy concerns from consumers. For example, according to EU GDPR, it ispreferable to minimize data transmission between processing nodes.
Our goal is to construct training algorithms to obtain a global DNN model that can be adapted to all data, even when
individual nodes only have access to different subsets of the data. We assume that this algorithm is allowed tocommunicate autonomously between nodes, exchanging information such as model variables or their update difference,but data are prohibited from being moved from node they reside on.
Now, several platformers provide advanced services by aggregating/monopolizing data. However, we aim to create asociety where data ownership belongs to individual and can be used for a variety of services while protecting data privacy.

Background, goal

Asynchronous consensus algorithm

Background: We are entering an era of distributed
aggregation of due to data volume, privacy protection andlegal regulations (e.g. GDPR).

Proposed algorithm: A training algorithm is constructed toobtain a global model by asynchronously exchanging primalmodel variables and Lagrangian dual variables among nodes.
Goal: To obtain a global DNN model without data aggregation (this enables to work on arbitrary network structure).
(where asynchronous communication among nodes, such asmodel variable exchange, is allowed).
Edge computing(composed of small data centers)

Healthcare

Mobility

Smartphone

FactoryHome

OfficeUpdate direction modified tomove towards a global model

Gradient

Global model

Consensus force
Node #1
Node #2
Node #3
Node #V

Proposed algorithm

Node #i(Primal-dual)
variables are updated

ProblemU

Problem: When the data at each node is statisticallyheterogeneous, a global DNN model cannot be obtained byjust minimizing each nod cost function.

X
Variables areexchanged

Approach: We solve problem by minimizing sum of costfunctions under a consensus constraint that all nodemodels are identical with each other.

X
Node #j

Update time

Data sets are placed across V nodes (x1,…,xV). Model variables are
updated (i) such that minimizes sum of cost functions (ΣFi) (ii) under aconsensus constraint that models are identical among V nodes (s.t....)

U

Node #1

U U U U X U U U U X U U U X U U U U U U U U X

Node #2U U U X U U U U U U U X U U U U U U U UNode #3U U U U U U X U U U U U U U U U U U X U U U UNode #VU U U U XU

X U U U U U U X U U X

References[1] T. Sherson, R. Heusdens, and B. Kleijn, Derivation and analysis of the primal-dual method of multipliers based on monotone operator theory,
IEEE transactions on signal and information processing over networks 5, 2 （2018）, 334–347.
[2] K. Niwa, N. Harada, G. Zhang, B. Kleijn, Edge-consensus learning: deep learning on P2P networks with nonhomogeneous data, submitted to
KDD 2020.

Contact
Kenta Niwa Email: cs-openhouse-ml@hco.ntt.co.jp
Communication Science Laboratory, Media Intelligence Laboratory
オープンハウス 2020

21

Cardiac model that makes it heart
Gaussian process with physical laws for 3D cardiac modeling

Abstract
Cardiovascular disease is one of the leading causes of both morbidity and mortality all over the world. Early
diagnosis and treatment planning are demanded for the wide variety of etiologies and pathophysiologies. In thelast decades, intensive research in the field of computational biology has demonstrated the potential ability of
three-dimensional (3D) cardiac computational models to give us a clue to perform early diagnosis or to have highaffinity with machine learning for treatment planning. We introduces some physical laws into a Gaussian process
for a statistical 3D cardiac computational model. The heart shape must be ruled by some physical laws, whichshould be an important clue for the statistical shape estimation. For demonstration, we apply our model into thepipeline that estimates the heart shape from cardiovascular magnetic resonance (CMR) imaging, by combining itwith the deep neural networks-based anatomical segmentation of CMR imaging.

Heart shape estimation

Regression with physical laws

 Cardiac modeling from magnetic resonanceimaging (MRI) The Frank-Starling law is introduced into a
Gaussian process-based statistical shape model.

Anatomicalclassification

MRI

Deep learning forclassification

State-of-the-art
deep nets [2]Shapeestimation

Statistical shape modelingfor regression

The latent volumepressure curve isexpressed as ahidden Markov model.

Our method can obtain the similar
shape to that handles by experts.

Our contribution:
Gaussian process (GP)with physical laws forthe heart shape [1]From left to right. First: MRI. Second: Zoom-in. Third: The endocardium
manually handled by experts. Fourth: Predictive endocardium.

Difficulty

Goal: shape prediction comparable to experts

 It is not easy to obtain training datasets for theheart shape, which is typically handled by experts.
We introduce physical law into
unsupervised learning.

The lower, the better

 It is difficult to estimate details of the heart shape,since MRI provides only statistical information
locally averaging over time and space.
We extend the static model into stime-varying statistical shape model.

Mean squared error comparison between baseline
methods and our model.
For 29 out of 33 subjects,
the mean squared error is
improved by around 9%.

■ Raw: Regression without GP fitting.
■ GP-base: GP without physical laws.
■ GP-FS: GP with the Frank-Starling law.

References[1] M. Nakano, R. Shibue, K. Kashino, S. Tsukada, H. Tomoike, “Gaussian process with physical laws for 3D cardiac modeling,” under review.
[2] T. Ngo, Z. Lu, G. Carneiro, “Combining deep learning and level set for the automated segmentation of the left ventricle of the heart fromcardiac cine magnetic resonance,” Medical Image Analysis, Vol. 35, pp. 159–171, 2019.

Contact
Masahiro Nakano Email: cs-openhouse-ml@hco.ntt.co.jp
Recognition Research Group, Media Information Laboratory
オープンハウス 2020

22

Listening carefully to your heart beatCardiohemodynamical analysis based on stethoscopic sounds

Abstract
A variety of sounds are constantly emitted from the human body as a result of life activities. By listening to andanalyzing those sounds, we can obtain useful information about the function and condition of the body, which iscalled auscultation. In this research, we are focusing on heart sounds to estimate the function and condition of theheart and blood vessels based on the observation of acoustic signals. In our system, multiple microphones areattached to several places, such as the chest, to detect heart activity. Based on the captured sound, it estimates
the degree of normality as a score and generates an explanatory statement as a sentence. We have confirmedthat the normality estimation and description generation with a specified degree of detail work effectively for test
data. We aim to realize an "AI stethoscope" that contributes to the prevention and early detection of diseases inmany people, as skilled doctors can accurately understand and explain the condition through auscultation.

Working Example of Prototype

Concept of AI Stethoscope Multiple small microphones are attached to your body tocollect useful information and visualize it in various ways.

Numberof data By our machine learning techniques, advanced media
conversion such as text generation from audio [1] ispossible, in addition to abnormality detection or patternclassification.

400
300
200
100
0

 The system will be extended to a visualizing andanalyzing tool for heart activity and hemodynamics, which
is a part of the “digital twin computing” concept that weare pursuing.

Depending on the thresholdsetting, useful normal/abnormalclassification is possible← Abnormal

NormalAbnormal

Normal →Threshold Normality score

Figure 3: Distribution of Normality Scores for Test Data [3]Table 1: Generated Description Examples
Depending on the specified level of detail, useful description is generated.

C index

Example of generated text

20

Your heart sounds are abnormal.

40

There is a systolic murmur in the heart sound.

60

There is a systolic murmur in the heart sound, and it may be
a sign of cardiomyopathy.

Figure 1: Prototype of the Heart Sound Collector

Generation of Explanatory Text and Score
Audio signal

AudiofeatureDetail index to specify detail,concreteness and length of
the generated explanatory
statement

Decoder

Explanatory
statement“There is a
murmur.”LatentfeatureEncoder
Decoder

Normality score
Class labels

Information conversion from audio to descriptions and scoresare performed by neural networks called encoders and decoders.
Figure 2: Sequence Conversion Model in This Study [1]

[Hz]
Ch. １
Cuff
upstream

Water hammer soundKorotkoff sound

2k

0
Ch. 2
Cuff
2k
downstream
0
Ch. １
Cuff
2k
upstream0

The intensity, tone and timing of the sound reflect thephysical properties of blood vessels and blood flow

Figure 4: Measurement Example of Heart Sound and Blood FlowSound during Blood Pressure Measurement in the Left Upper Arm(Each Ch. Top: Sound signal waveform, Bottom: Sound spectrogram)
Part of this research is commissioned to NTT Research, Inc. (Medical and Health
Informatics Laboratories; MEI Lab.) by NTT.

References[1] S. Ikawa, K. Kashino, “Neural audio captioning based on conditional sequence-to-sequence model,” In Proc. DCASE 2019 Workshop, 2019.
[2] M. Nakano, R. Shibue, K. Kashino, S. Tsukada, H. Tomoike, “Gaussian process with physical laws for 3D cardiac modeling,” under review.
[3] The PhysioNet Computing in Cardiology Challenge, http://physionet.org/content/challenge-2016/1.0.0/, 2016.

Contact
Kunio Kashino Email: cs-openhouse-ml@hco.ntt.co.jp
Media Information Laboratory and Biomedical Informatics Research Center
オープンハウス 2020

23

Make natural-looking illusions by perceptual model
Adaptive motion retargeting for illusion-based projection AR

Abstract
Hen-Gen-Tou, invented by CS Labs, is an illusion-based projection mapping that adds motion impressions to real
static objects. It produces illusory motion impressions in the projection target by projecting luminance motion
signals that selectively drive the motion detectors in the human visual system. However, in order to successfully"fool" human vision, the amount of movements must be properly adjusted because there is a limit in shift size thatcan create the illusion. Here, to automate this laborious adjustment task, we propose an optimization frameworkthat adaptively retargets the motion information in real time based on a perceptual model. The perceptual model
predicts the perceived deviation of a projected pattern from an original surface pattern using a computationalmodel of human visual information processing. This technique will broaden the range of applications of Hen-GenTou, including interactive applications.

Mechanism of “Hen-Gen-Tou”
Motion vectors

Moving image
sequence

Animated!

The human visual system separately analyzes color, pattern, andmotion information and later integrates them. Taking advantage ofthis characteristic, HenGenTou adds illusory motion impressions
on static objects by projecting only luminance motion signals.
Problem: shift limit
Trying to add too large motion leads to subjective inconsistency
between original appearance and projected patterns.

Large motion

Projection pattern

Perceptual model to predict
subjective inconsistency level
Predict subjective inconsistency level ofthe projection result as a deviation fromthe ideal appearance in representation
space of the human visual system.

Ideal appearance Projection result
Human visual system model
Computedeviation
Subjective inconsistency level

Inconsistent
result

Impossible for interactive applications

Manual adjustments?

Optimizing motion vectors based on the perceptual model
Maximize motion impressions while keeping the subjective inconsistency level
predicted by the perceptual model within the tolerable range.

Perceptual model
Motion vectors（Input）

Shift size

Projection
targetOptimized
Motion vectors

ScalingSpace

Enhancing
local motion

Optimizing motion vectors

Automatically generates results in real time,comparable to conventional methods that require manual fine-tuning

References[1] T. Fukiage, T. Kawabe, S. Nishida, “Perceptually based adaptive motion retargeting to animate real objects by light projection,” IEEE
Transaction on Visualization and Computer Graphics, Vol. 25, No. 5, pp. 2061-2071, 2019.
[2] T. Kawabe, T. Fukiage, M. Sawayama, S. Nishida, “Deformation Lamps: A projection technique to make static objects perceptually
dynamic,” ACM Transactions on Applied Perception, Vol, 13, No. 2, pp. 1-17, 2016.

Contact
Taiki Fukiage Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory Representation Group, Human Information Science Laboratory
オープンハウス 2020

24

Tiny eye movements reflect cognitive states
Relation of eye-movement dynamics with cognition and pupil

Abstract
Recent studies have shown that the characteristics of the eye movements reflect the cognitive state varying
moment by moment. In this study, we investigated the relationship between cognitive states and the detailed
dynamics of eye movements which have been regarded as mere mechanical oscillation. We found that the
dynamic properties of eye oscillation after saccade reflect the task difficulty in gaze shift task. In addition, we
showed that the oscillation dynamics was greater for pupil-centric motion than motion of entire eyeball. The
correlation between the pupil-centric oscillation and pupil size indicates that it reflects the instantaneous states of
eye tissue inside iris (e.g., stiffness of muscles controlling pupil size). There is a potential that the measurement of
the tiny eye movements can be applied to a tool for monitoring the time-varying cognitive state (for example,
monitoring the worker who engages in task requiring attention or cognitive load).

Relation between dynamics of saccadic eye movement and cognitive task•

The gaze position after a saccade does not stop exactly at the target position but oscillate around it (overshoot)•We tested the relationship between overshoot, which has been regarded as mere mechanical vibration, and cognition
Gaze shift taskovershootGaze position

Target positionTime
Anti-saccade

Comparison of dynamics properties of oscillation
Anti-saccade ＞ Pro-saccade

We found that the dynamic properties of eye
oscillation after saccade reflects the task difficulty

Pro-saccade

Look away from stimulus（High-load task)

Look to stimulus（Low-load task）

Time

Eye-movement dynamics reflects
the physical properties of eye tissue

Correlation between pupil size anddynamics of pupil-centric motion••

Extracts the pupil-centric and iris-centric motions

Oscillation of pupil center

Calculates the correlation coefficient between eyemovement dynamics and pupil size or task difficulty
Cognitive factor

Pupil size

Oscillation of iris center(eyeball motion)

Eye-movement dynamics
CorrelationTask
difficulty

Pupil-centric motion
Entire eyeball motion

No significant correlation

Viscoelasticity of the pupil-centric oscillation isgreater than iris-centric one (entire eyeball)

Dynamics correlate with pupil size
as well as task difficulty
Reflects state of muscles controlling pupil size

References[1] S. Yamagishi, M. Yoneya, S. Furukawa, “Relationship of postsaccadic oscillation with the state of the pupil inside the iris and with cognitive
processing,” J Neurophysiol, 2020.

Contact
Shimpei Yamagishi Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory Resonance Research Group, Human Information Science Laboratory
オープンハウス 2020

25

Haptic metameric textures
Direct control of perceived texture of 3D printed stimuli

Abstract
Humans sense spatial patterns through their eyes and hands. Past studies have revealed differences(as well as similarities) between vison and touch in texture processing (e.g., eye is good at detecting
texture boundaries, while hand can discriminate subtle texture differences), but the underlying
computational differences remain poorly understood. Here we transcribed various textures as surface
relief patterns by 3D-printing, and analyzed the tactile discrimination performance regarding thesensitivity to image statistics. We found that visually very different patterns cannot be distinguished bytouch if they differ only in higher-order statistics. Human tactile texture processing differs from visual
one not only in spatio-temporal resolution but also in (in)sensitivity to higher-order image statistics.

Manipulation of surface texture by image processing
Treat a height map as a grayscale image- Converting the depth pattern of surface carving intoa luminance value of a monochrome image- Conduct image processing on the image then 3D print- Investigated that tactile texture perception is sensitive to the Fourier amplitude
spectrum while not to the Fourier phase spectrum

Similar but
distinct

Natural scene images with intensity normalization
Different amplitude, Different phase
These surfaces areclearly different

Dark rep. concave
Bright represent convex

Texture synthesis

Same amplitude, Different phase

Cannotdiscriminate

Still,
visibly different

Touch is more sensitive to small differences in the amplitude spectrum
Vision is more sensitive to higher-order statistics, including the phase spectrum
Realize the design of different looks with identical tactile feeling
References[1] S. Kuroki, S. Sawayama, S. Nishida, “Haptic metameric textures,” bioRxiv, 2019. doi: https://doi.org/10.1101/653550

Contact
Scinob Kuroki Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory representation research Group, Human and Information Science Laboratory
オープンハウス 2020

26

What causes emotional change?
Monitoring emotion in experimental settings and daily life

Abstract
Our emotions are influenced by changes in both our internal states and external environment includinginteractions with others. In this study, we aimed to investigate how emotional responses change through social
interaction in the experimental setting, and to develop a new framework for monitoring the internal change ofemotional states in daily life. The findings of two experiments measuring autonomic responses during interaction
suggested that negative emotions and positive emotions are transmitted differently through interaction. Theinterpersonal dynamics of emotional change found here will help us to understand larger group phenomena suchas crowd joy or panic. Furthermore, for the purpose of logging internal states which dynamically change throughdaily life, we developed a new self-tracking method using exclamations or onomatopoeias (e.g., “NIKONIKO”,“SHOBON”). This kind of framework will contribute to creating the system that support our wellbeing.

Emotional change through social interactionin the experimental setting

A new self-tracking system for monitoringemotion in daily life

Experiment 1 Sharing pain by simultaneously
Negative emotion stimulation of thermal stimulus

Commonlyused method

Participants with weaker reactions elevated their physiological reactivity in response to their partner’s reactions
during face-to-face interaction, but not vice versa.
Face-to-face interaction

Blocking interaction

Person A

Person A

synchrony
Person B ＜Person B

Negative emotion

Physiologicalresponse（Blood Volume Pulse）

stronger

weakerStimulation

Stimulation

Experiment 2
Positive emotion

Sharing excitement during
collaborative block game

PlayerPartnerPartner※ブロックを抜く前の25秒間の心拍数

・ High cognitive load (difficult to answer intuitively)
・ Unsuited to capture physical emotional experience・ Hesitate to answer honestly (esp. negative emotions)
The method
we propose

Reporting emotional states using
exclamations and onomatopoeias

We propose a new self-tracking method using emotional
expression words (exclamations, onomatopoeias) which
are suitable to express physical emotional experience.
positive

Who areyou with?

Time
How doyou feel?

negativestronger

Positive emotion

Physiologicalresponse（Z-score of Heart Rate）

Play alternately
in collaboration
with each other

PlayerHappy 0 ：not at all 〜 10：very much
0 ：not at all 〜 10：very much
Sad

What areyou doing?

Participants changed their physiological reactivity inresponse to their partner’s reactions, whether theirreaction is stronger or weaker than their partner.
Player alternation

Numerical rating for each adjective
expressing emotion

weakerWords were mapped to emotion type and strength
using a large-scale survey (N=14,000) to enable such
word-based emotion assessments.
ex.) Happiness Sadness Anticipation SurpriseAngerstrong うひょー

がっくり

ぴりぴり

えー

いらいら びくびく

げっ

うっとり

weak にこにこ

しゅん

そわそわ

どきっ

かちん

うわぁ

うん

Fear
ぞくっ

Disgust Accept

References[1] A. Murata, H. Nishida, K. Watanabe, and T. Kameda. “Convergence of physiological responses to pain during face-to-face interaction,”
Scientific Reports, 10（1）, 1-10. 2020.
[2] 村⽥ 藍⼦・熊野 史朗・渡邊 淳司. “協⼒場⾯における対⼈インタラクションの当事者評価と客観評価,” 電⼦情報通信学会技術研究報告, vol. 118,
No. 487, pp.111-114, 2019.

Contact
Aiko Murata Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory Resonance Research Group, Human Information Science Laboratory
オープンハウス 2020

27

Special cognitive abilities of esports experts
Performance, physiological state, and brain activity

Abstract
Since esports is not easily influenced by physical factors, it is said that their actual ability, condition, and visual
capacity are difficult to analyze. To objectively evaluate these characteristics of esports players, we haveinvestigated the relationship between game performance and the physiological/brain states of esports players bytaking a neuroscience approach. The results of our experiments show that synchrony of heart rate between
players reflects their actual abilities and that parietal brain activities are associated with the outcome of the next
round. Further investigations, including a vision-science-based test of the capacities of visual information
processing of esports players, will reveal the physiological/brain states and cognitive capacities related toperformance and will enable us to establish a neuroscience methodology for esports players to improve theirperformance in competitive environments.

Esports and player characteristics

Player characteristics

Esports• Actual ability• Condition• Visual capacity
Approach

Heart rate (HR)Brain activity

Visual test

Synchrony of HR reflects
actual abilities of players [1]Brain activities areassociated with theoutcome of next roundExamination on superior
visual capacities of players

Heart rate (bpm)Heart rate (bpm)High-level battle：synchronized

Parietal brain activity
increases

120

80

Win
0

5

10

Middle-level battle:
unsynchronized

120

Parietal brain activity
decreases

Focusing on the various types ofvisual capacity as well as visualacuityLetteridentification
Peripheral
viewing

Multiple object
tracking
Contrast
sensitivity

Measuring capacities using ourunique vision test battery [2]

Lose
80

0

5

10
Time(min)

References[1] K. Watanabe, N. Saijo, M. Kashino （2019） “The across-player correlation of the physiological change reflecting the fight-or-flight responsein esports.” Neuroscience Meeting Planner. Chicago, IL: Society for Neuroscience, 2019. Online. Program No. 769.20.
[2] K. Hosokawa, K. Maruya, S. Nishida, M. Takahashi and S. Nakadomari （2019） "Gamified vision test system for daily self-check," 2019 IEEE
Games, Entertainment, Media Conference （GEM）, New Haven, CT, USA, 2019, pp. 1-8.

Contact
Sorato Minami Email: cs-openhouse-ml@hco.ntt.co.jp
Kashino Diverse Brain Research Laboratory
オープンハウス 2020

28

Realizing a harmony in rugby scrum
Easy assessment of player coordination with wearable sensors

Abstract
It is crucial for team plays in sports that the players synchronize their actions, but objectively assessing player
coordination is not easy. We propose a convenient measurement method to immediately evaluate and feed backsome aspects of player coordination by attaching compact inertial measurement units (IMUs) to each player; we
use the example of scrumming in rugby. In a scrum, a pack of eight forwards (players) are required to coordinate
their forward drives, timing and direction, to maximize forward pressure. The IMU data allows us to calculate theacceleration vectors and its peak time structures for the group of players involved. Constant storage of thesevalues can yield a useful database for understanding each player’s characteristics and developing suitablecombinations of players to improve scrumming performance. This measurement system also makes it easy tovisualize, as well as sonify, player coordination during various joint activities other than rugby.

Convenient wearable sensor

Feedback of player coordination
Analysis, Database

IMU

Daily training

Acceleration in the y axis

Example of acceleration profiles

Example of feedback screen
Acceleration
vectors

Large amplitude inforward direction?

Movie

Peak accelerationtime• Rapid visualization of player coordination• Developing suitable combinations of players• Potential of application to various activities

Sonifying the peak time sequence of players
Example of spectrogram

1.5

Frequency（Hz）• Easy acquisition via wearable sensors

Sequential orderfrom front to back?

Pitch rises forgood sequences

1.0
0.5
0

0

0.2

0.4

0.6
Time （s）

0.8

1.0＊This research is in collaboration with NTT Communications Shinning Arks.

References[1] T. Kimura, N. Saito, H. Okamoto, K. Ohta, “Evaluation of cooperation between players in the rugby scrum using IMU,” Proc. Sports
Engineering and Human Dynamics 2019, 2019.
.

Contact
Toshitaka Kimura Email: cs-openhouse-ml@hco.ntt.co.jp
Kashino Diverse Brain Research Laboratory
オープンハウス 2020

29

What is a "straight" ball?
Physical and perceptual attributes of a pitched ball

Abstract
Athletes' perceptions are extraordinarily sensitive, but they do not necessarily capture the physical world as it is.
We accurately quantified the physical characteristics of pitched baseballs, and then investigated how battersperceived them. In the physical measurement, we devised a technique to easily measure the 3D rotation
characteristics of the ball using a single camera. In the perceptual measurement, we found that the batters’
perception of the ball’s horizontal movement in the trajectory was systematically biased, even though they
discriminated small differences in the movement. Moreover, we found that the direction of their bias was reversed,
depending on the pitcher’s handedness. In sports science, physical measurement tends to be emphasized, and thegap between the physical characteristics and players’ perception is at issue. Combining physical and perceptual
measurement to identify the cause of such a gap could lead to a dramatic revolution in training and coaching.

Physics
Camera image

Directions of ball rotation axis for an exemplaryright-handed pitcher. 2D ball diagram from thebatter’s view. The rotation axes (dots) and ball’s
horizontal movements were analyzed in detail.

Pitcher

SliderFour-seamer*

3D measurementof ball rotation

Moves to1st- base side
*Four-seamer is“straight” in Japanese.

Moves to3rd-base side

Physical trajectoryof four-seamer

Most four-seamers for right-handed pitcher physically move to 3rd-base sideThis measurement technique is being tested for practical use with professional
and amateur baseball teams, and the Japan Women's National Softball Team.

Perception

An example ofbatter’s perception

Straight

R-handed pitcher
L-handed pitcher

Response ratio of“moves to 3rd-base side”

Perceptual trajectoryof four-seamer

A batter illusorily
perceive a physicallymoving ball’s trajectory
as “straight”.

BatterMoving

The direction of the illusion
is reversed, depending on
pitcher handedness.

Horizontal movement in ball trajectory (cm)Perception is biased in the direction thefour-seamer is perceived as "straight".

References[1] D. Nasu, T. Kimura, M. Kashino, “Do baseball batters perceive straight ball trajectory as straight?” Proc. 2020 Conference on North
American Society for Psychology of Sport and Physical Activity, 2020.

Contact
Daiki Nasu Email: cs-openhouse-ml@hco.ntt.co.jp
Kashino Diverse Brain Research Laboratory
オープンハウス 2020

30

Body representation for quick and skillful action
Uncertainty of hand-state estimate regulates stretch reflex

Abstract
Reflexive motor control mechanisms are embedded in the brain to unconsciously correct ongoing body
movements. They do this by detecting changes in the external world and ones own posture, via sensory signals
from eyes and limbs. We investigated the information processing underlying the functional and context-dependent
regulation of reflexive responses. We found an interesting attenuation of muscle response to resisting suddenchanges in limb movement, which occurred if visual feedback of the limb movement was not given or was
distorted. The result suggests that the brain regulates reflexive responses depending on body states estimated by
combining multimodal information such as vision and bodily sensations, rather than single modality information aspreviously thought. We will further explore the computational mechanisms of reflexive sensorimotor control, which
may be beneficial to analyzing the performance of athletes or to developing effective sports training methods.

Movement correction without consciousness* Reflexive muscle
contraction occurringin response to sudden
extension of a muscle
to recover the posture

Motor correction
through consciousness(≥ 200 ms)

Stretch reflex:

environment

Visual information

Motor command

Somatosensory input

Stretch reflex
modulation

Proposed hypothesis: multimodal processing

Somatosensory-motor
response (~50 ms)●Interaction with

Conventional hypothesis: unimodal processing

Somatosensory
information

Consciousness●Postural change

Informational processing forreflex regulation

Multisensory
integration

Reflex responses enable quick
compensation against disturbances
to movement and posture

Stretch reflex
modulation

Somatosensory information

Visual information contributes to regulation of stretch reflex●Stretch reflex was evoked by

Setup

Results

applying force during a wrist
movement●Testing if removing visual cue ofhand position (cursor) affects thereflexive response●The stretch reflex was smallerwith longer cursor removal●Suggesting uncertainty of handstates estimated by multisensory
integration underlies the reflexive
motor control

References[1] S. Ito, H. Gomi, “Visually-updated hand state estimates modulate the proprioceptive reflex independently of motor task requirements,” eLife,9:e52380, 2020.
[2] S. Ito, H. Gomi, “Online modulation of proprioceptive reflex gain depending on uncertainty in multisensory state estimation,” Proc. The
Society for Neuroscience 49th Annual Meeting, 2019.

Contact
Sho Ito Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory and Motor Research Group, Human Information Science Laboratory
オープンハウス 2020

31

Unconscious is smarter than conscious
Environmental dependency in visuomotor responses

Abstract
We may tend to think that human understand current states of self-body and external environments, and thenconsciously control our limbs according to those states. However, dominant part of actual skilled movements arecontrolled unconsciously. We are trying to reveal those implicit sensorimotor control mechanisms to understandthe brain processing for skillful motor control. By inflicting different postural stability and/or noisy visual motion
conditions, we investigated the adaptability of voluntary and reflexive responses to visual motion stimuli, andfound that only reflex responses can be adjusted suitably to the different situations. This suggests that
unconscious processing would be smarter than conscious processing for a particular condition. By understandthe mechanisms of brain processing for sensorimotor control, we will be able to designe more sophisticatedcommunication and man-machine interface, and novel training method for athletes.

Body and arm movement control

Implicit and explicit sensorimotor processes

 In daily life, human has various physicalinteractions with external environments while
his/her body is moving, which cannot beeasily realized by current industrial robots.

It is generally considered that

 How can we realize these dexterous
interactions by using the brain information
processing whose transmission speed ismuch slower than that in current computers?

processing.

implicit process for sensorimotor
control is faster but is less flexible
than the explicit and voluntary
Conscious processing
Unconscious processing

The current study revealed that an implicit process canregulate sensorimotor response according to a particularenvironmental situation whereas an explicit process cannot [1].

Experiment: Context dependency of implicit and explicit processes for visual motion
ExperimentSupply environmental context of postural fluctuation
and/or random visual motion.

環境の文脈

Context-dependent
responses

Measure reflexive and voluntary visuomotor response.
1.Reflexive response （Manual Following Response：MFR）
2.Voluntary response （Motion Direction Discrimination）

Results
1.Reflexive：Context dependent and rational modulation
2.Voluntary：Context independent modulation

These results suggest that reflex mechanisms
properly connect the relationship between postural
stability context and visual motion while voluntary
motor system cannot.

Schematic diagram ofcontext dependent
implicit motor response

Unstable posture context ⇒ large MFR
Movable environment context ⇒ small MFR

References[1] N. Abekawa, H. Gomi, “Modulation difference in visuomotor responses in implicit and explicit motor tasks depending on postural stability,” Proc.
The Society for Neuroscience 47th Annual Meeting, 2017.
[2] H. Gomi, K. Kadota, N. Abekawa, “Dynamic reaching adjustment during continuous body perturbation is markedly improved by visual motion,”
Proc. The Society for Neuroscience 40th Annual Meeting, 2010.

Contact
Hiroaki Gomi Email: cs-openhouse-ml@hco.ntt.co.jp
Sensory and Motor Research Group, Human and Information Science Laboratory
オープンハウス 2020