Seeing through Sounds

Visual Scene Understanding from Acoustic Signals
Go Irie
Tokyo University of Science
goirie@ieee.org
1Visual Scene Understanding• Recognize visible information in scene, e.g., object name, human action, etc.
• Object Recognition

PersonBike
Images from Pascal VoC dataset

2

Visual Scene Understanding• Recognize visible information in scene, e.g., object name, human action, etc.
• Object Recognition, Human Pose Estimation,

Images from https://pose.mpi-inf.mpg.de/3

Visual Scene Understanding• Recognize visible information in scene, e.g., object name, human action, etc.
• Object Recognition, Human Pose Estimation, Scene Depth Estimation, etc.

Images from NYU Depth V2 dataset

4Visual Scene Understanding• SOTA models publicly available online• Just use them to get excellent recognition results

5What if we cannot use camera?
••••

Dark room
Highly private/public area
Place where camera prohibited by ruleEtc.

Our idea
Use Microphone Instead!
6

Predicting Segmentation Results from Sound7

Why Audio? -- Auditory Scene Analysis••

Human can “recognize” object around her/him from just hearing sound,
without looking at source of sound
Build Auditory Scene Analysis artificially?

[Gaver, 1993]

Shutterstock8

Why Audio? -- Semantic & Geometric Cues
•Different sound shows different spectral property

https://www.jonnor.com/2021/12/audio-classification-with-machine-learning-europython-2019/9

Why Audio? -- Semantic & Geometric Cues••

Different sound shows different spectral property
Time difference of arrival gives direction of sound arrival

Estimated Direction of Sound

𝑡𝑡1

𝑡𝑡2

𝑡𝑡3
10

Our Projects
Passive Sensing

Active Sensing

StaticObject Recognition

Scene Depth Estimation

Dynamic

Human Action Recognition

Human Pose Estimation

11Our Projects
Passive Sensing

Active Sensing

StaticObject Recognition

Scene Depth Estimation

Dynamic

Human Action Recognition

Human Pose Estimation

12Predicting Semantic Segmentation Results
Goal: Recognize twopeople in conversation
Microphone Array

CameraMultichannel Audio Signals

Prohibited!
RGB Image

1) Record speech
Temporally
Synchronized2) Extract direction ofarrival and phonetic
featuresFeatureExtraction

Angular Spec.

MFCCs

3) Predict segmentation
results by CNNs
Convolutional Neural Network

Semantic
Segmentation

Segmented Imagemalefemale
background13

Prediction Results
Image Recognition
Results (Ground Truth)

Prediction
from Sound

Image Recognition
Results (Ground Truth)

Prediction
from Sound

14

Our Projects
Passive Sensing

Active Sensing

StaticObject Recognition

Scene Depth Estimation

Dynamic

Human Action Recognition

Human Pose Estimation

15

Active Acoustic Sensing

m1

m1
m2t
m2t
Dokmanić et al., 201316

Echo-based Depth Estimation
Scene

Ground Truth

Ours

Scene

Ground Truth

Ours

17

Listening Human Pose18

Framework

Ambisonics Mic
https://www.soundfield.com/

Adversarial learning to subject classification
Diversifying subject prediction probability
to make subject unpredictableprobability

Capturing Direction of Arrival

Subject-Agnostic Learning

probability

Intensity Vector

Subject ID

Subject ID

19

Results

Project page:

20

Our Projects
Passive Sensing

Active Sensing

StaticObject Recognition

Scene Depth Estimation

Dynamic

Human Action Recognition

Human Pose Estimation

21

Contact Point Estimation

22

Overview（X,Y）＝（3.2, 5.6）

Sound
Acquisition

Spectrogram

Regression
Network

Estimated
Position23

Our Projects
Passive Sensing

Active Sensing

StaticObject Recognition

Scene Depth Estimation

Dynamic

Human Action Recognition

Human Pose Estimation

24Summing Up• Sound is a promising alternative to image for solving various
visual scene understanding tasks, e.g., object recognition,
scene depth estimation, human pose estimation, and action
recognition.
• Light and sound are often “equivariant”:
Audio-visual model may have a potential to learn various
physical/geometric knowledge in the real world.

25

Seeing through Sounds

Visual Scene Understanding from Acoustic Signals
Go Irie
Tokyo University of Science
goirie@ieee.org
26