22

Listening carefully to your heart beatCardiohemodynamical analysis based on stethoscopic sounds

Abstract
A variety of sounds are constantly emitted from the human body as a result of life activities. By listening to andanalyzing those sounds, we can obtain useful information about the function and condition of the body, which iscalled auscultation. In this research, we are focusing on heart sounds to estimate the function and condition of theheart and blood vessels based on the observation of acoustic signals. In our system, multiple microphones areattached to several places, such as the chest, to detect heart activity. Based on the captured sound, it estimates
the degree of normality as a score and generates an explanatory statement as a sentence. We have confirmedthat the normality estimation and description generation with a specified degree of detail work effectively for test
data. We aim to realize an "AI stethoscope" that contributes to the prevention and early detection of diseases inmany people, as skilled doctors can accurately understand and explain the condition through auscultation.

Working Example of Prototype

Concept of AI Stethoscope Multiple small microphones are attached to your body tocollect useful information and visualize it in various ways.

Numberof data

 By our machine learning techniques, advanced media
conversion such as text generation from audio [1] ispossible, in addition to abnormality detection or patternclassification.

400300200
1000 The system will be extended to a visualizing andanalyzing tool for heart activity and hemodynamics, which
is a part of the “digital twin computing” concept that weare pursuing.

Depending on the thresholdsetting, useful normal/abnormalclassification is possible← Abnormal

NormalAbnormal

Normal →Threshold Normality score

Figure 3: Distribution of Normality Scores for Test Data [3]Table 1: Generated Description Examples
Depending on the specified level of detail, useful description is generated.

C index

Example of generated text

20

Your heart sounds are abnormal.

40

There is a systolic murmur in the heart sound.

60

There is a systolic murmur in the heart sound, and it may be
a sign of cardiomyopathy.

Figure 1: Prototype of the Heart Sound Collector

Generation of Explanatory Text and ScoreAudio signal

Audiofeature

Detail index to specify detail,concreteness and length of
the generated explanatory
statement

Decoder

Explanatory
statement“There is amurmur.”

LatentfeatureEncoder
Decoder

Normality score
Class labels

Information conversion from audio to descriptions and scoresare performed by neural networks called encoders and decoders.
Figure 2: Sequence Conversion Model in This Study [1]

[Hz]
Ch. １
Cuff
upstream

Water hammer soundKorotkoff sound

2k0
Ch. 2
Cuff
2k
downstream
0
Ch. １
Cuff
2kupstream0

The intensity, tone and timing of the sound reflect thephysical properties of blood vessels and blood flow

Figure 4: Measurement Example of Heart Sound and Blood FlowSound during Blood Pressure Measurement in the Left Upper Arm(Each Ch. Top: Sound signal waveform, Bottom: Sound spectrogram)
Part of this research is commissioned to NTT Research, Inc. (Medical and Health
Informatics Laboratories; MEI Lab.) by NTT.

References[1] S. Ikawa, K. Kashino, “Neural audio captioning based on conditional sequence-to-sequence model,” In Proc. DCASE 2019 Workshop, 2019.
[2] M. Nakano, R. Shibue, K. Kashino, S. Tsukada, H. Tomoike, “Gaussian process with physical laws for 3D cardiac modeling,” under review.
[3] The PhysioNet Computing in Cardiology Challenge, http://physionet.org/content/challenge-2016/1.0.0/, 2016.

Contact
Kunio Kashino Email: cs-openhouse-ml@hco.ntt.co.jp
Media Information Laboratory and Biomedical Informatics Research Center
オープンハウス 2020