Access from: https://www.kecl.ntt.co.jp/openhouse/2021/index̲en.html

Science of Machine Learning
01. Learning how to learn from various datasets Meta-learning from tasks with heterogeneous attribute spaces
02. Ask me how to make a fair decision for everyone Learning individually fair classiﬁer based on causality
03. Ask me anything about network structure Indexing graph structure with decision diagrams
04. No labels? Count on me! Self-supervised Adaptation for Unknown Domains/Classes

05. Finding groups through connection relationships Relational data model with inﬁnite ﬂexibility

06. Finding features in data fast and accurately Acceleration of feature selection with group regularization
07. Privacy-aware machine learning Distributed learning algorithm and medical application

08. Estimating Risk of Infection in a City People Flow Reconstruction Based on Anonymous Sensor Data

Science of Communication and Computation
09. Being greedy makes quantum computers work well Economic rationality makes cloud quantum computing reliable
10. Revealing hidden structures behind sentences Neural rhetorical structure parsing with pseudo-labeled data
11. Two experts, one result Fusing two experts for enhancing their specialties

12. Can a chatbot mediate trust between humans? Bridging doctor-patient rapport through a chatbot

13. Recipes for enjoy-talking conversational systems Development of Transformer-based conversational systems

Science of Media Information
14. Detecting faint sound by light Non-contact sound measurement by precision interferometry

15. Extracting voices out of noise & reverberation Joint Signal Separation, Dereverberation and Noise Reduction

16. AI that acquires knowledge just by watching TV Crossmodal learning for concept acquisition of human movements
17. Real-time speech emotion contololler using face Emotional voice conversion via facial expression recognition

18. Visualizing touched places for Corona prevention Touched places detection using heat traces by thermography
19. Telestethoscope: Looking into Body by Listening Biomedical sound analysis utilizing physical characteristics

Science of Human
20. Is falling birthrate related to population density? Prediction and demonstration via life history theory

21. Moves of magnetic ﬁelds, Moves by magnetic ﬁelds Magnetact: A Magnetic force-based Tactile technology
22. Make hard objects soft, make rough objects smooth Simple method for modulating tactile texture

23. Sense of touch connects our hearts beyond distance Empathetic telecommnunication by vibrotactile transmission
24. Why do mothers approach to infants' "crying"? Oxytocin as a neural regulator of maternal implicit approach
25. Auditory attention that appears in the eye Relation between microsaccades and auditory spatial attention
26. Blink pattern of elite car race drivers Professional drivers always blink at the same time over laps
27. Essence of "keeping your eye on the ball" Eye-hand coordination in motor learning

28. How do you deﬁne your dominant hand? Quantifying motor-skill performance using a smartphone

29. Brain functions to recognize and hit a fastball Brain mechanisms for quick judgment and motor control

Meta-learning from tasks with heterogeneous attribute spaces

01 Learning how to learn from various datasets
Abstract

Many training data are required for deep learning. We propose a meta-learning method that can improve the performancewith a limited number of training data in the target task by using data in different tasks. For example, the proposed
method can be used for learning from data in different factories. Existing meta-learning methods assumes that all tasks
have the same attribute space. The proposed method is the first method that can meta-learn from tasks with different
attribute spaces. With the proposed method, we design permutation invariant neural networks that can handle data withdifferent attributes and labels. Currently, we cannot apply machine learning methods when sufficient number of training
data are unavailable. We want to expand field where machine learning methods are applicable by extending the proposed
method.

References[1] T. Iwata, A. Kumagai, “Meta-learning from Tasks with Heterogeneous Attribute Spaces,” in Proc. Neural Information Processing
Systems (NeurIPS), 2020.

Contact
Tomoharu Iwata / Ueda Research Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Learning individually fair classifier based on causality

02 Ask me how to make a fair decision for everyone
Abstract
Machine learning predictions are increasingly used to make critical decisions that severely affect people’s lives, including
loan approvals, hiring, and recidivism prediction. For this purpose, we developed a novel machine learning technology thatmakes predictions that are accurate and fair with respect to sensitive features such as gender, race, religion, and sexual
orientation. To achieve high prediction accuracy, we utilize prior information about societal demands for each decisionmaking scenario, e.g., “rejecting applicants based on physical strength is fair if the job requires physical strength.”
Although existing methods cannot ensure fairness when the data are not generated by a restricted class of functions, ourproposed method can use various data to guarantee fairness. Thus, admitting that “what is fair” depends on a particularsense of societal values, we create innovative machine learning technologies that can more flexibly respond to societal
demands by bridging the gap between technical limitations and societal needs. In this way, we hope to mold a society thatcan make automatic decisions while ensuring that nobody will suffer detrimental treatment.

References[1] Y. Chikahara, S. Sakaue, A. Fujino, H. Kashima, “Learning Individually Fair Classifier with Path-Specific Causal-Effect Constraint,”
in Proc. the 24-th International Conference on Artificial Intelligence and Statistics (AISTATS), 2021.

Contact
Yoichi Chikahara / Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Indexing graph structure with decision diagrams

03 Ask me anything about network structure
Abstract
We developed a new substructure index called the zero-suppressed sentential decision diagram (ZSDD) and efficientalgorithms with which to construct it. A substructure index is a compact representation of network substructures. With it,
we can count and find exponentially many substructures in time linear to the index size. Since ZSDD is generally smaller
than similar data structures, we can solve network-related problems much faster. For example, we can count more than 100
million network routes within a second using ZSDD. Our algorithm can solve a wide range of problems on real-world
networks like communication networks, traffic networks, and power grids. Therefore, it will make society more efficientby finding better solutions to network-related problems.

References[1] M. Nishino, N. Yasuda, S. Minato, M. Nagata, “Zero-suppressed Sentential Decision Diagrams,” in Proc. 30th AAAI Conference on
Artificial Intelligence (AAAI 16), 2016.
[2] M. Nishino, N. Yasuda, S. Minato, M. Nagata, “Compiling Graph Substructures into Sentential Decision Diagrams,” in Proc. 31st AAAI
Conference on Artificial Intelligence (AAAI 17), 2017.
[3] Y. Nakahata, M. Nishino, J. Kawahara, S. Minato, “Enumerating All Subgraphs Under Given Constraints using Zero-suppressed
Sentential Decision Diagrams,” in Proc. 18th Symposium on Experimental Algorithms (SEA 2020), 2020.

Contact
Masaaki Nishino / Linguistic Intelligence Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Self-supervised adaptation for unknown domains/classes

04 No labels? Count on me!
Abstract

We propose a method that enables us to use miscellaneous data collected under various conditions directly for deeplearning. When we train a model using miscellaneous data, the model's performance is often degraded by "Information to be
Ignored," which disturbs correct recognition. Our proposed method dramatically improves the model's performance by
estimating the "Information to be Ignored" and training the model not to be affected by it. Our technique will make itpossible to easily utilize miscellaneous data for learning and will contribute to expanding AI services into fields where
deploying deep learning has been challenging.

References[1] Y. Mitsuzumi, G. Irie, D. Ikami, T. Shibata, “Generalized Domain Adaptation,” in Proc. IEEE/CVF Conference on Computer Vision and
Pattern Recognition (CVPR), 2021.
[2] R. Tobias, R. Stiefelhagen, “Adaptiope: A Modern Benchmark for Unsupervised Domain Adaptation,” in Proc. IEEE/CVF WinterConference on Applications of Computer Vision (WACV), 2021.

Contact
Yu Mitsuzumi / Recognition Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Relational data model with infinite flexibility

05 Finding groups through connection relationships
Abstract
Relational data, including network graphs, such as the connection relationships of users in social networking services andthe purchase histories of users for products, appear all around us. In this research, we aim to find the hidden groups in therelational data. In general, when finding groups in relational data, it is difficult to set the number and size of groups
appropriately by hand. Therefore, we propose a relational data analysis method that has the ability to automatically adjust
the number and size of groups in a data-driven manner according to the size and nature of the input data. By finding
appropriate groups in larger relational data, we will contribute to the development of technology for more efficient
information storage, search, and retrieval.

References[1] M. Nakano, A. Kimura, T. Yamada, N. Ueda, “Baxter Permutation Process,” in Proc. Advances in Neural Information Processing
Systems 33 (NeurIPS), 2020.

Contact
Masahiro Nakano / Recognition Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Acceleration of feature selection with group regularization

06 Finding features in data fast and accurately
Abstract
Selecting important feature groups from data is one of the most fundamental tasks in data analysis. However, thecomputation cost is high when the size of the data is large. In this study, we propose a fast method for feature group
selection without degrading the accuracy. It safely skips unnecessary computations and intensively optimizes importantgroups. As a result, our method is up to 35 times faster than the original method without any loss of accuracy. In addition,
our method has no additional hyperparameters and no additional tuning costs. It will be possible to create value fromcomplex and large scale data by speeding up the analysis of data with complex structures such as groups. We create value
from a wide variety of data and contribute to society.

References[1] Y. Ida, Y. Fujiwara, H. Kashima, “Fast Sparse Group Lasso,” in Proc. Neural Information Processing Systems (NeurIPS), 1700-1708,
2019.

Contact
Yasutoshi Ida / Software Innovation Centor
Email: cs-openhouse-ml@hco.ntt.co.jp

Distributed learning algorithm and medical application

07 Privacy-aware machine learning
Abstract

While use of massive data benefits on training deep learning models, aggregating all data into one physical location (e.g. acentral cloud) may not be possible due to data privacy concerns. For example, according to the EU GDPR, data transmission
should be minimized among processing nodes. Our goal is to construct training algorithms to obtain global deep learningmodels that can be adapted to all data, even when individual nodes only have access to different subsets of the data. We
assume that this algorithm is allowed to communicate between nodes in an asynchronous/sparse manner, exchanging suchinformation as model variables or their update differences. However, data are prohibited from being moved from the nodeon which they reside. We aim to indirectly exploit the overall data across countries and provide high performance services
for such industries as the medical/health-care field while protecting privacy.

References[1] J. Chen, A. H. Sayed, “Diffusion adaptation strategies for distributed optimization and learning over networks,” IEEE Transactions
on Signal Processing, Vol. 60, No. 8, pp. 4289–4305, 2012.
[2] B. McMahan, E. Moore, D. Ramage, S. Hampson, B. A. y Arcas, “Communication–efficient learning of deep networks fromdecentralized data,” in Proc. Artificial Intelligence and Statistics (AISTATS 2017), pp. 1273–1282, 2017.
[3] K. Niwa, N. Harada, G. Zhang, W. B. W Kleijn, “Edge-consensus learning: deep learning on P2P networks with nonhomogeneous data,”
in Proc. the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020), pp. 668–678, 2020.
[4] National Institutes of Health (NIH) clinical center, ChestXray14 data set.

Contact
Kenta Niwa / Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

People flow reconstruction based on anonymous sensor data

08 Estimating risk of infection in a city
Abstract

We can measure the risk of infection in a city by knowing the movement routes of infected people, but collecting
information of people's movements infringes privacy. We thus propose a new method for estimating multiple routes on thebasis of anonymized passage information to estimate the risk of infection while preserving privacy. For a better
estimation, we need to choose more appropriate path patterns of people that correctly explain the anonymized passage
information. Therefore, we consider a movement model, estimate the transit probability between passage information, andfind the most likely set of routes efficiently on the basis of the model. It can improve the accuracy of the estimation.
Infectious disease control will be one of functions of smart cities to be realized in the future.By using our work, the risk ofinfection can be estimated without collecting personal movement information.

References[1] K. Matsuda, H. Ikeuchi, Y. Takahashi, T. Toyono, “People Flow Reconstruction Based on Anonymous Sensor Data toward Smart City
Infrastructure for Estimating Infection Route,” IEICE Technical Committee on Network Systems, 2020.

Contact
Kotaro Matsuda / NTT Network Technology laboratories
Email: cs-openhouse-ml@hco.ntt.co.jp

Economic rationality makes cloud quantum computing reliable

09 Being greedy makes quantum computers work well
Abstract
We consider cloud quantum computing and have proposed a method guaranteeing the correctness of the answer received
from a cloud-quantum-computing server. Since quantum computers are strongly affected by noises, it is indispensable torealize such a method for practical cloud quantum computing. Previous methods work only under the assumption that averified quantum computer runs in a sufficiently short time or a user can perform quantum communication with the server.
By introducing the economic rationality, we have succeeded to remove these assumptions. Furthermore, our method can beapplied to a broader class of quantum-computing architectures than that of previous methods. We aim to incorporate
large-scale quantum computers into the existing worldwide network by developing our method, which makes it possible todeliver the high computational capability of quantum computers to everyone all over the world.

References[1] Y. Takeuchi, T. Morimae, S. Tani, “Sumcheck-Based Delegation of Quantum Computing to Rational Server,” in Proc. the 16th
International Conference on Theory and Applications of Models of Computation (TAMC), 2020.

Contact
Yuki Takeuchi / Computing Theory Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Neural rhetorical structure parsing with pseudo-labeled data

10 Revealing hidden structures behind sentences
Abstract
This poster presents a method to identify the hidden structures of documents. Each document has a rhetorical structure,
which expresses the relations among clauses. Since building a rhetorical structure parser is based on supervised learning, it
requires large amounts of manually annotated training data for accurate parsing. However, conventional methods sufferfrom a lack of training data, resulting in poor performance because manual annotation is quite labor intensive. To tackle
this problem, we propose a method that uses silver data: automatically annotated pseudo-labeled data. We pre-trained theparser with silver data and fine-tuned it with gold data: manually annotated data. Our experimental results demonstrated
that our method achieved the best performance. The new parser will contribute to various natural language processing
applications, such as machine translation and automatic summarization.

References[1] N. Koabayashi, T. Hirao, H. Kamigaito, M. Okumura, M. Nagata, “Improving Neural RST Parsing Model with Silver Agreement
Subtrees,” in Proc. 2021 Annual Conference of the Noth American Chapter of the Association for Computational Linguistics, 2021.

Contact
Tsutomu Hirao / Linguistic Intelligence Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Fusing two experts for enhancing their specialties

11 Two experts, one result
Abstract

Although dialogue services sometimes require operators with expertise in multiple fields, such people are rare (e.g., travel
guides for foreigners need both travel guide skills and foreign language skills). We focus on a collaboration style in which
operators with different fields of expertise work together to solve advanced problems and grow through practice. We
focus on a style in which two operators interact with a user as if they were a single operator. Our proposed style enables
operators to provide advanced services across multiple fields, to collaborate smoothly, and to learn from each other. Bybuilding a dialogue service that allows collaboration among diverse human resources, we will achieve more advanced
dialogue services and create a society that supports the employment and growth of many people.

References[1] T. Arimoto, R. Higashinaka, K. Tanaka, T. Kawanishi, H. Sugiyama, H. Sawada, H. Ishiguro, “Collection and analysis of dialogues
provided by two speakers acting as one,” in Proc. the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue(SIGDIAL), pp. 323-328, 2020.

Contact
Tsunehiro Arimoto / Interaction Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Bridging doctor-patient rapport through a chatbot

12 Can a chatbot mediate trust between humans?
Abstract
For mental health professionals (MHP) to understand patients’ mental health, it is critical that patients engage in deep
self-disclosure. However, people tend to avoid revealing their vulnerabilities for fear of being judged by others. Chatbots
show great potential in this domain because prior research has shown that people tend to disclose symptoms of depression
more truthfully when talking to a chatbot than when talking to a human interviewer. Our work extends this prior work byproposing a novel approach to facilitate people's self-disclosure to MHPs through chatbots. We designed, implemented andevaluated a chatbot that elicits deep self-disclosure and promotes trust-building between users and the MHPs. Results
show that people were more willing to share their self-disclosure content with MHPs through the chatbot, which suggests
the promise of our approach.

References[1] Y. Lee, N. Yamashita, Y. Huang, “Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental HealthProfessional,” in Proc. ACM Hum.-Comput. Interact. 4, CSCW1, Article 031, (CSCW'20), ACM, pp.1-27, 2020.
[2] Y. Lee, N. Yamashita, W. Fu, Y. Huang, ““I Hear You, I Feel You”: Encouraging Deep Self-disclosure through a Chatbot,” in Proc. the2020 CHI Conference on Human Factors in Computing Systems (CHI '20), ACM, pp.1–12, 2020.

Contact
Naomi Yamashita / Interaction Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Development of transformer-based conversational systems

13 Recipes for enjoy-talking conversational systems
Abstract
We are studying a social dialogue system that satisfies people's desire for dialogue through natural conversation.
We have developed a deep-learning-based Japanese social dialogue system, which is pre-trained with the largest-scale
Japanese dialogue data obtained from Twitter and fine-tuned with high-quality dialogue data that NTT has cultivated overmany years of research. We also conducted a quantitative analysis of the utterances of the constructed system andidentified remaining issues such as contradictions and discontinuous topics.
We believe that the desire to communicate with others is one of our fundamental desires. We aim to realize a social
dialogue system as a partner that continuously satisfies this need for dialogue.

References[1] H. Sugiyama, H. Narimatsu, M. Mizukami, T. Arimoto, Y. Chiba, T. Meguro, H. Nakajima, , “Development of conversational system
talking about hobby using Transformer-based encoder-decoder model,” in Proc. Special Interest Group on Spoken Language
Understanding and Dialogue Processing (SIG-SLUD), Vol. B5, No. 02, pp. 104-109, 2020 (in Japanese).
[2] H. Sugiyama, T. Meguro, Y. Yoshikawa, J. Yamato, "Improving Dialogue Continuity using Inter-Robot Interaction," in Proc. IEEE
International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 105-112, 2018.

Contact
Hiroaki Sugiyama / Interaction Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Non-contact sound measurement by precision interferometry

14 Detecting faint sound by light
Abstract

Optical measurement methods have been attracting attention to realize various sound measurements that are difficult toachieve with microphones. In this research, in collaboration with the Quantum Optical Physics Research Group of the NTT
Basic Research Laboratories, we have developed a sound field measurement technique with much lower noise thanconventional methods using precision optical measurement technology. Focusing on the difference in physicalcharacteristics between sound and noise components, we proposed a sound measurement technique using a differential
midfringe locked interferometer that reduces noise while maximizing sensitivity to sound. By combining this technology
with high-stability laser technology, the amplitude of measurement noise has been reduced to 1/30 compared toconventional methods. By applying the precision measurement technology cultivated in optics, we will realize the nextgeneration sound measurements such as 3D sound-field measurement, remote sensing, and ultra-high precision
measurement.

References[1] K. Ishikawa, Y. Shiraki, T. Moriya, A. Ishizawa, K. Hitachi, K. Oguri, “Reducing noise of mid-fringe locked interferometer by opticaldifferential detection,” in Proc. Acoust. Soc. Jpn., 2021. (in Japanese)[2] K. Ishikawa, Y. Shiraki, T. Moriya, A. Ishizawa, K. Hitachi, K. Oguri, “Simple and low-noise optical measurement of sound using midfringe locked interferometer,” in Proc. Acoust. Soc. Jpn., 2020. (in Japanese)

Contact
Kenji Ishikawa / Computing Theory Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Joint signal separation, dereverberation and noise reduction

15 Extracting voices out of noise & reverberation
Abstract
To enable such audio devices as smart speakers to accurately recognize human voices in real-world environments, we must
reduce the noise and reverberation from the signals observed by microphones and extract each individual’s voice. State-ofthe-art (SOTA) technology addresses this problem by sequentially applying the following three techniques: (a)dereverberation, (b) source separation, and (c) denoising. However, SOTA is ineffective in noisy reverberant conditions
because all three techniques, (a), (b), and (c), are optimized individually without considering the overall performance. In
this exhibit, we introduce a new technology that jointly optimizes (a), (b), and (c) to maximize the quality of the output
audio. Our new technology significantly improves the speech recognition performance compared to the SOTA method. It
will contribute to a more convenient world where people and computers can interact smoothly in our daily environments,
including train stations, streets, and shopping malls.

References[1] T. Nakatani, C. Böddeker, K. Kinoshita, R. Ikeshita, M. Delcroix, R. Haeb-Umbach, “Jointly optimal denoising, dereverberation, andsource separation,” in Proc. IEEE/ACM Trans. Audio, Speech, Language Process., vol. 28, pp. 2267-2282, 2020.
[2] R. Ikeshita, T. Nakatani, S. Araki, “Block coordinate descent algorithms for auxiliary-function-based independent vector
extraction,” in Proc. IEEE Trans. Signal Process., 2021, to appear.
[3] R. Ikeshita, T. Nakatani, “Independent vector extraction for fast joint blind source separation and dereverberation,” in Proc. IEEE
Signal Process. Lett., 2021, to appear.

Contact
Rintaro Ikeshita / Signal Processing Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Crossmodal learning for concept acquisition of human movements

16 AI that acquires knowledge just by watching TV
Abstract

We developed a crossmodal learning method that can acquire "concepts" corresponding to specific objects and events on
unlabeled audio and video signals. Achieving it in an unsupervised way is particularly important, since it is generally
difficult to manually label all the objects and events appearing in audio-visual data for supervised learning. Our main idea
was identifying concepts by looking at them from different modalities, just like looking at objects from different angles.
To efficiently detect and utilize temporal co-occurrences of audio and video information, we employed a guided attention
scheme. Experiments using real TV broadcasts of sumo wrestling with live commentaries show that our method can
automatically associate specific athlete techniques and its spoken descriptions without any manual annotations. We areaiming for a future in which AI can acquire knowledge autonomously by just watching and listening to everyday scenes, orwatching TV.

References[1] Y. Ohishi, A. Kimura, T. Kawanishi, K. Kashino, D. Harwath, J. Glass, “Trilingual Semantic Embeddings of Visually Grounded Speechwith Self-attention Mechanisms,” in Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2020.
[2] Y. Ohishi, A. Kimura, T. Kawanishi, K. Kashino, D. Harwath, J. Glass, “Pair Expansion for Learning Multilingual Semantic Embeddings
using Disjoint Visually-grounded Speech Audio Datasets,” in Proc. Interspeech 2020.
[3] Y. Ohishi, Y. Tanaka, K. Kashino, “Unsupervised Co-Segmentation for Athlete Movements and Live Commentaries Using Crossmodal
Temporal Proximity,” in Proc. International Conference on Pattern Recognition (ICPR) 2020.

Contact
Yasunori Ohishi / Recognition Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Emotional voice conversion via facial expression recognition

17 Real-time speech emotion contololler using face
Abstract
There are many kinds of physical or mental barriers that prevent individuals from smooth verbal communication. One key
technique to overcome some of these barriers is voice conversion (VC), a technique to convert para/non-linguistic
information contained in a given utterance without changing the linguistic information. Here, we propose a crossmodal
voice control system, which offers a way to control the vocal expression of emotion in speech through the facial
expression in a face image. The proposed system consists of performing facial expression recognition (FER) followed by VC.
For VC, we have developed a method based on sequence-to-sequence (S2S) learning, which is designed to convert the
prosodic features as well as the voice characteristics in speech conditioned on the output of the FER system. We believethat this work can provide some insight on what it is like to be able to control our voice through different modalities.

References[1] H. Kameoka, K. Tanaka, T. Kaneko, N. Hojo, “ConvS2S-VC: Fully convolutional sequence-to-sequence voice conversion,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing, pp. 1849-1863, June 2020.
[2] K. Tanaka, H. Kameoka, T. Kaneko, N. Hojo, “AttS2S-VC: Sequence-to-sequence voice conversion with attention and context
preservation mechanisms,” in Proc. 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP2019), pp.
6805-6809, May 2019.
[3] M. Shervin, M. Minaei, and A. Abdolrashidi, “Deep-emotion: Facial expression recognition using attentional convolutional network.”
Sensors 21.9:3046, 2021.

Contact
Kou Tanaka / Recognition Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Touched places detection using heat traces by thermography

18 Visualizing touched places for Corona prevention
Abstract
One effective infection prevention method is regularly disinfecting the environmental surfaces of public areas. However,no method currently identifies where and how much others have touched a particular door or shelf. This exhibition
introduces a visualization system that identifies touched places by overlay projection. When a person touches an object,her hand’s heat remains on its surface. This heat trace can be captured by thermography cameras. By combining Near
Infrared (NIR) and thermography cameras, we can detect the touched places with a light algorithm. This technology enables
us to identify the objects or places that others have touched, although the virus itself remains invisible to the naked eye.
We believe that our system will help relieve anxiety during the COVID-19 pandemic. It will also enable us to gather
statistics and data about the places touched by people and improve the efficiency of disinfection activities.

References[1] Y. Kishino, Y. Shirai, Y. Yanagisawa, K. Ohara, S. Mizutani, T. Suyama, “Identifying Human Contact Points on Environmental Surfaces
using Heat Traces to Support Disinfect Activities,” SenSys2020 COVID-19 Pandemic Response, 2020.

Contact
Yasue Kishino / Learning and Intelligent Systems Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Biomedical sound analysis utilizing physical characteristics

19 Telestethoscope: Looking into Body by Listening
Abstract
As a basic research toward providing a person with an enhanced sense of well-being, such as early detection of diseases, wedeveloped a wearable acoustic sensor array system that can collect sounds from various parts of the human body and send
the signals remotely to a receiver terminal, which is equipped with 18 acoustic sensors inside an examination vest. Whenthe system comes into practical use for medical care, a medical practitioner will be able to listen to sounds from various
locations on the patient's body without having to make direct physical contact with the patient or use of a traditional
stethoscope, which will be useful in online medical examinations. This system is also expected to play a role, potentially in
combination with other information sources, in the research and development of new medical techniques such as thevisualization of physical states and direct translation from body sounds into explanatory sentences.

References[1] S. Ikawa, K. Kashino, “Neural audio captioning based on conditional sequence-to-sequence model,” in Proc. Workshop on Detection
and Classification of Acoustic Scenes and Events (DCASE), 2019.
[2] M. Nakano, R. Shibue, K. Kashino, S. Tsukada, H. Tomoike, “Gaussian process with physical laws for 3D cardiac modeling,” in Proc.
European Signal Processing Conference (EUSIPCO), pp. 1452-1456, 2020.

Contact
Kunio Kashino / Media Information Laboratory and Biomedical Informatics Research Center
Email: cs-openhouse-ml@hco.ntt.co.jp

Prediction and demonstration via life history theory

20 Is falling birthrate related to population density?
Abstract
Low birthrates and increased longevity are critical social problems. This study approaches the issue from a biologicalperspective. By analyzing such public statistical data as census information, we found that in Japan, some areas, whichhave high population densities, also tend to have low fertility rates and extended longevity. This idea may reflect the factthat in highly competitive situations, the number of children will be reduced to allow greater investment for child toincrease the chance for success. Declining birthrates are often discussed in terms of childbirth and childcare systems,
employment, and economic conditions. Perhaps this phenomenon should be studied from multifaceted perspectives to gain
basic knowledge for planning essential solutions.

References[1] M. Matsuda, T. Kobayasi, “Do population density and economic situation affect residents' life history strategies? -- An analysis via
web survey --,” IEICE Technical Report, Vol. 119, No. 394, HCS2019-69, pp. 89-92.
[2] M. Matuda, “Low birthrate and longevity in Japan -- Comparison between prefecture level and municipal level --,” IEICE Technical
Report, Vol. 118, No. 437, HCS2018-66, pp. 107-111.

Contact
Masafumi Matsuda / Interaction Research Group, Innovative Communication Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Magnetact: A Magnetic force-based Tactile technology

21 Moves of magnetic fields, Moves by magnetic fields
Abstract
Magnetact is a magnetic tactile technology that does not require any power supply or wiring to present a tactile sensation.
The specific patterned magnetic rubber sheets present a unique tactile texture when they are rubbed together. However,it takes time to magnetize the kind of detailed lattice pattern of S/N poles that can provide a selective tactile
presentation. Therefore, we invented a new method for generating complex geometric magnetic patterns by layering
multiple magnetic rubber sheets with simple magnetic patterns. Moreover, the superposed magnetic lattice patterns can be
dynamically changed by rotating the layered magnetic sheets. This method resolves the tradeoff between the complexityof the magnetized pattern and the time required for magnetization.
The Magnetact technology is expected to be applied to a variety of tactile experiences, including in online tactile
workshops, portable tactile picture books, and low-cost tactile VR experiences.

References[1] K. Yasu, “Magnetic plotter: a macrotexture design method using magnetic rubber sheets,” in Proc. the 2017 CHI Conference on
Human Factors in Computing Systems (CHI '17), 2017.
[2] K. Yasu, “MagneLayer: force field fabrication by layered magnetic sheets,” in Proc. the 2019 CHI Conference on Human Factors in
Computing Systems (CHI '20), 2020.
[3] K. Yasu, M. Ishikawa, “Magnetact Animals: a simple kinetic toy kit for a creative online workshop for children,” in Proc. CHI EA '21,
Article. No. 198, pp. 1–4, 2021.

Contact
Kentaro Yasu / Sensory representation research Group, Human and Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Simple method for modulating tactile texture

22 Make hard objects soft, make rough objects smooth
Abstract
To artificially present rich tactile textures, a tactile display that can modulate tactile textures such as hardness androughness of various daily objects is necessary. In this study, we focused on a tactile illusion, named the ‘velvet hand
illusion’ (VHI), which could be used to modulate tactile textures in a simple setup. We evaluated tactile textures during the
VHI qualitatively and quantitatively and found that the VHI is a phenomenon wherein the tactile texture of metallic wires
being touched was perceived to be softer than it actually is, with the tactile texture of cloth. Based on the findings, wedeveloped the rotating-frame method wherein tactile textures of arbitrary objects can be modulated to seem softer andsmoother than they actually are. Since the proposed method can be applied to the tactile textures of daily objects, it isexpected that the method could be used in various application scenes such as product design and in-store demonstrations.

References[1] T. Yokosaka, S. Kuroki, S. Nishida, “Describing the sensation of the 'velvet hand illusion' in terms of common materials,” IEEE
Transactions on Haptics, DOI:10. 1109/TOH, 2020. 3046376. Online ahead of print.
[2] T. Yokosaka, Y. Suzuishi, S. Kuroki, “Feel illusory texture through a hole: Rotating stimulus modulates tactile sensation for touchedobject’s surface,” in Proc. EuroHaptics2020 (WIP137), 2020.

Contact
Takumi Yokosaka / Sensory representation research Group, Human and Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Empathetic telecommnunication by vibrotactile transmission

23 Sense of touch connects our hearts beyond distance
Abstract
Reconstructing empathic communication that allows people to share feelings even in the remote age, by designing 3
frameworks as a solution. We have created a system called “remote high-five” through which families in remote areas can
exchange “high fives” during sports events. As a framework for realizing empathic communication in society, we arranged
three elements—namely, design of the story (including the people who experience it and the flow of the experience),technology that supports tactile transmission, and context that inclusively involves all parties—and were able to show theeffectiveness of the “remote high-five” system with an example. We aim to create a society in which anyone cancommunicate with each other and feel that they are together even if the people are far apart.

References[1] H. Hayakawa, R. Owaki, T. Ishikawa, K. Minamizawa, Y. Tanaka, K. Komazaki, Y. Kamamoto, J. Watanabe, “Proposal of Interactive
Visuo-Audio-Tactile Media “Public Booth for Vibrotactile Communication” for Telecommunication with High Presence The virtual
reality society of japan, Vol. 25, No. 4, 2020.
[2] J. Watanabe, “The Institute of Electrical Engineers of Japan,” Vol. 141, No. 2, 2021.

Contact
Kakagu Komazaki, Junji Watanabe / Sensory Resonance Research Group, Human Information Science Laboratory
Yutaka Kamamoto / Computing Theory Research Group, Media Information Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Oxytocin as a neural regulator of maternal implicit approach

24 Why do mothers approach to infants' “crying”?
Abstract
It is well known that infant vocalizations, crying in particular, enhance maternal motivation to provide care and inducesmaternal approach. However, the mechanism has not been clarified. In this study, we examined which infant vocalizationinduces maternal approach and what neural factor regulates it. We measured the position of the center of pressure ofmothers to evaluate their natural and implicit movement when they listened to infant voice stimuli. We took saliva samples
from them to measure levels of oxytocin, one of the neural hormones that represent mental states. We found that mothers
approached infants' “crying” because it raised a strong sense of “urgency.” The approach distance for crying was negatively
correlated with their oxytocin levels. We think our study will help establish home-scale well-being, and finally, social wellbeing.

References[1] D. Hiraoka, Y. Ooishi, R. Mugitani, M. Nomura. “Differential effects of infant vocalizations on approach-avoidance postural
movements in mothers.” Front. Psychol., Vol. 10, 1378, 2019.
[2] Y. Ooishi, D. Hiraoka, R. Mugitani, M. Nomura, “Relationship between oxytocin and maternal approach behaviors to infants’
vocalizations,” Compr. Psychoneuroendocrinol., Vol. 4, 100010, 2020.

Contact
Yuuki Ooishi / Sensory Resonance Research Group, Human Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Relation between microsaccades and auditory spatial attention

25 Auditory attention that appears in the eye
Abstract

The eyes reflect various states of mind. It is known that gaze directed to an object of interest, but also thatmicrosaccades, involuntary tiny eye movements, reflect the visual covert attention. Here, we investigated the relationshipbetween microsaccades and spatial auditory attention. We found that microsaccades reflect the direction of auditory
attention during a dichotic selective attention task and are also associated with task performance. Although many previous
studies have already shown a link between microsaccades and visual attention, this study showed that they are also linked
to auditory attention processes. We believe that this finding will lead to the development of technology for estimating
attention states that vary spontaneously and instantaneously (e.g., estimating information such as the voice to which aperson is paying attention at the party). We also hope that this result encourages future studies to elucidate themechanisms of how our auditory system coordinates spatial attention.

References[1] S. Yamagishi, S. Furukawa, “Simultaneous measures of auditory brainstem frequency following response, pupillary response, andmicrosaccade during auditory selective attention task,” in Proc. 42nd Association for Research in Otolaryngology (ARO) MidwinterMeeting, 2020.
[2] S. Yamagishi, S. Furukawa, “Relationship between auditory selective attention and microsaccades,” in Proc. The Auditory Research
Meeting, 2020.

Contact
Shimpei Yamagishi / Sensory representation research Group, Human and Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Professional drivers always blink at the same time over laps

26 Blink pattern of elite car race drivers
Abstract

How do formula car drivers process the stream of information from the external world during ultra-high-speed driving? In
this study, we focused on spontaneous blinking and captured the dynamic changes in the psychological and cognitive statesthat top Japanese drivers exhibit when driving around a circuit. As the lap speed increased, the drivers blinked at particularlocations on the track and these locations were common between the drivers. We revealed for the first time that racers
adjust their psychological and cognitive states to control their cars in distinct areas on the track with increasing lap
speed. In future, we aim to decode these states that even professionals themselves do not understand through theunconscious physiological phenomena. By clarifying the implicit information processing occurring in the brain that leads tosuperior performance, we pursue a society where people can easily improve their desired skills.

References[1] R. Nishizono, N. Saijo, M. Kashino, “Synchronization of Spontaneous Eyeblink during Formula Car Driving”, in Proc. The 13th ACM
Symposium on Eye Tracking Research and Applications (ETRA), 2021.

Contact
Ryota Nishizono / Kashino Diverse Brain Research Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Eye-hand coordination in motor learning

27 Essence of “keeping your eye on the ball”
Abstract
Motor learning is indispensable to realize skillful behaviors. Under a dynamically changing environment like in tennis, weneed to acquire and execute new skilled reaching movements while our gaze is directed toward the ball (foveal vision) andpossibly the opponent’s location too (peripheral vision). While previous studies have taken gaze information into
consideration for motor learning, conventional theories were restricted to emphasizing the superiority of the reaching to a
foveated target over the reaching to a peripheral target. Our current study elucidated that the eye-hand spatial
coordination for both foveal and peripheral reaching movements during learning is inherently linked with the internal modelof learned reaching skills. Our results highlighted a novel interaction of gaze information with motor learning. By
understanding the brain mechanism of this interaction, we will be able to design a novel training method that utilizes
different gaze states to enhance sports training and rehabilitation.

References[1] N. Abekawa, S. Ito, H. Gomi, “Different learning and generalization for reaching movements in foveal and peripheral vision,” inProc. Adv. Mot. Learn. Mot. Control, 2019.
[2] N. Abekawa, S. Ito, H. Gomi, “Foveal and peripheral vision separate motor memories for reaching movement,” JNNS2020, 2020.

Contact
Naotoshi Abekawa / Sensory and Motor Research Group, Human and Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Quantifying motor-skill performance using a smartphone

28 How do you define your dominant hand?
Abstract

The skilled hand in sports is usually determined by a handedness questionnaire [1], but it is blind to forced right-handed
correction and does not measure motor performance. We invented a new method to easily evaluate each hand’s motor skill
by quantifying the variability of fast repetitive motion. Unlike other measures of motor performance that require
customized equipment, our method is practical as it uses a smartphone’s accelerometer to determine motor skill, making iteasier to use in the field and at home. Our method could detect individuals who were forcibly corrected as their left hand
was more skilled than the left hand of natural right-handers. In the near future, we plan to use our methodology toquantify sports training aimed at specific movements, and to motivate physical rehabilitation via daily feedback. It may
also serve as a tool in understanding how and why the brain’s control of the left and right arms is different.

References[1] A. Takagi, S. Maxwell, A. Melendez-Calderon, E. Burdet, “The dominant limb preferentially stabilizes posture in a bimanual taskwith physical coupling,” Journal of neurophysiology, Vol. 123, No. 6, pp. 2154-2160, 2020.
[2] R. C. Oldfield, “The assessment and analysis of handedness: the Edinburgh inventory,” Neuropsychologia, 1971.

Contact
Atsushi Takagi / Sensory and Motor Research Group, Human and Information Science Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp

Brain mechanisms for quick judgment and motor control

29 Brain functions to recognize and hit a fastball
Abstract
When hitting a 95-mph blazing fastball in baseball, the batter must judge the ball’s path and control the bat in about 0.4
seconds. However, it takes longer to achieve accurate judgment and movement. Though many studies have examinedjudgment and movement processes in the brain separately, they are closely related. We investigated how the brain
establishes quick judgment and movement processing under strict time constraints. To this end, we conducted a baseballlike hitting experiment and clarified that the Strike/Ball judgment had less effect on hitting performance as the time
constraint became stricter, but changing the movement strategy restrained the decrease. Our goal is to provide novel
methods to evaluate and improve the brain functions of athletes to enhance cognitive-motor control in support ofconventional approaches to physical fitness testing and training. We believe that this work will elucidate now hidden
mental processes and find application in other research fields.

References[1] A. Kobayashi, T. Kimura, “Effects of cognitive strategy on hitting tasks,” IEICE Technical Report, Vol. 118, No. 470, pp. 37-42,
2019.
[2] A. Kobayashi, T. Kimura, “Motor redundancy affects decision-making behavior,” SICE Division of Life Engineering Symposium 2020,
2020.
[3] A. Kobayashi, T. Kimura, “Go/No-go decision making under severe time constraints interferes with hitting task performance,” inProc. The Society for Neuroscience 49th Annual Meeting, 2019.

Contact
Akemi Kobayashi / Kashino Diverse Brain Research Laboratory
Email: cs-openhouse-ml@hco.ntt.co.jp