Rescue Dog Action
Recognition by Integrating
Ego-centric Video, Sound andSensor Information

2021/1/5

UEC Yanai Lab. Tokyo.
1Introduction
What is a rescue dog?
•A dog that does not perform rescue work, but searches forvictims in disaster areas.

•Handler manually records actions and verbally explainsthem to commander•Make a disaster rescue plan based on the informationfrom rescue dogs and handlers.

2021/1/5

Rescue dog

Handler
UEC Yanai Lab. Tokyo.

Fire and Disaster
Management
2Challenges in Utilizing Rescue Dogs
•Manual recording by handlers is insufficient in terms ofobjectivity and quantity of information.
•Verbal explanation is insufficient to ensure accuracy ofinformation•More and more accurate information on the rescue dog
is required
2021/1/5

UEC Yanai Lab. Tokyo.

3

Background
ImPACT

https://www.jst.go.jp/impact/program/07.html
2021/1/5

UEC Yanai Lab. Tokyo.

4Cyber suit
Development of a wearable measurement and recording
deviceEquipped with camera, microphone, inertial sensor, etc.

SensorCamera and microphone[K. Yuichi, at el. ICIR, 2015]
2021/1/5

UEC Yanai Lab. Tokyo.

5

Objective•Using Deep Learning to Recognize Rescue
Dog Behavior•Using the multimodal rescue dog data (video, audio,and sensor information) provided by Professor Ohnoof Tohoku University.
Found a Victim!!

2021/1/5

UEC Yanai Lab. Tokyo.

6

Related Work
•Two-Stream CNN[K.Simonyan, NIPS 2014]•Sound Net[Y. Aytar, NIPS 2016]

2021/1/5

UEC Yanai Lab. Tokyo.

7

Related Work

•Temporal Binding Network[K.Evangelos, ICCV 2019]2021/1/5

UEC Yanai Lab. Tokyo.

8

Previous Work
CNN
0100.
.
.
10Sound
Opticalflow

CNN

RGBCNN

multi-classlabels•Dogcentric activity recognition by integrating appearance,motion and sound. [T. Araki,EPIC 2019]2021/1/5

UEC Yanai Lab. Tokyo.

9Dataset
Rescue Dog Data Set•A group of videos showing rescue dogs in training.

rescue dog's point

the handler's point

third party's point

RGB Image

Optical flow Image

STFT

2021/1/5

UEC Yanai Lab. Tokyo.

10Dataset
Rescue Dog Data Set•Information obtained from sensors•x sens time:second(s)•𝐺𝑥 , 𝐺𝑦 , 𝐺𝑧 : angular velocity(deg/s)•𝐴𝑥 , 𝐴𝑦 , 𝐴𝑧 : acceleration(m/s^2)•Roll,Pitch,Yaw: posture(degree)

Data used•𝑀𝑥 , 𝑀𝑦 , 𝑀𝑧 : geomagnetism(μT)•Pressure(hPa)•Temperature•https://www.amtechs.co.jp/product/gps/post-130.html

2021/1/5

UEC Yanai Lab. Tokyo.

11

Dataset
Number of frames appearing in each class per data
set[frame]

50000
45000
40000
35000
30000
25000
20000
15000

10000
5000
0

barkcling

Full 9299
Mini 2281

3631
1936

2021/1/5

command8035
4903

eathandler
drink
876
7066
145
3718

run
414
0

UEC Yanai Lab. Tokyo.

seevictim
414
1849

shake

sniff

stop

1349
405

16573 31533 45915
8208 13194 14154

12walkDog Activity Classes
Some of the behavior classes are described in detail• cling : The situation in which a doc is sniffing with thenose close to the smell.

• command : The situation in which the dog is being
instructed by the handler.
• look at handler : The situation in which the dog islooking at the handler. Hereinafter, this action is called
just “handler”.
2021/1/5

UEC Yanai Lab. Tokyo.

13

Outline of the proposed method
CNN
0
100.
.
.
10Sound
Opticalflow

CNN

RGBCNN

Sensorsignal2021/1/5

multi-classlabels

LSTM
UEC Yanai Lab. Tokyo.

14

Multi-class labels
MultiLabel SoftMarginLoss

x : output of the networky : target,
C :numer of the classes

2021/1/5

UEC Yanai Lab. Tokyo.

15

MethodconcatSound stream(1,224,224)
RGB stream

VGG-16(3,224,224)
Optical flow stream

VGG-16(3,224,224)(11,)

Sensor stream

Bi-LSTM(310,9)2021/1/5

FC

Resnet-101(1024,)(1x1x2048x4,)

UEC Yanai Lab. Tokyo.

16

Method•Sensor information(310,9)•Sensor acquires information every 0.005 seconds
0.005 × 320 = 1.6(s)Smoothing with 10 pieces of data 320
310

9 :𝐺𝑥 , 𝐺𝑦 , 𝐺𝑧 : angular velocity(deg/s)
𝐴𝑥 , 𝐴𝑦 , 𝐴𝑧 : acceleration (m/s^2)
Roll,Pitch,Yaw: attitude (degree)
2021/1/5

UEC Yanai Lab. Tokyo.

17

Experiments•Data set used
Rescue dog training data recorded on July 10, 2016.
(about 12 minutes)
Rescue dog training data recorded on November 11, 2016.
(about 4 minutes and 50 seconds)•Training data set
First 80% of the data set used (22979 frames)•Data set for evaluation
Second half 20% of the dataset used (6643 frames)•Indicators for accuracy comparison
Jaccard coefficient

2021/1/5

𝑇𝑃(𝑇𝑃 + 𝐹𝑃 + 𝐹𝑁)
UEC Yanai Lab. Tokyo.

18

Differences between the previous
study and the proposed method
RGB

Araki etal.

VGGResNet

Proposedmethod 1

Proposedmethod 2

VGG-16

VGG-16

VGG-16

VGG-16

Optical VGG-16 VGG-16
VGG-16
VGG-16
Flow
Sound 2D Conv ResNet-101 ResNet-101 ResNet-101(MFCC)(STFT)(STFT)(STFT)
Sensor None
None
Bi-GRU
Bi-LSTM
2021/1/5

UEC Yanai Lab. Tokyo.

19Comparison with each method [%]5047.514848.0546.01

464441.32

4240
3836

Total
Araki

VGG-ResNet

Proposed method 1

Proposed method 2

UEC Yanai Lab. Tokyo.
2021/1/520

Comparison with each method [%]
90
80
7060
50

4030
20100Bark

Cling

Araki

Command Eat-drink

Handler See-victim

VGG-ResNet

Proposed method 1

Shake

Sniff

Stop

Proposed method 2

UEC Yanai Lab. Tokyo.
2021/1/5

21WalkDiscussion
Comparison between Araki's method and the proposed method[%]
90

73.68

80

7080.06

64.7

62.64

57.9

55.68

60

45.29

5039.324030
2048.05
46.01
41.32

20.3616.6

14.53
12.33

100

barkshake

Araki

sniff

VGG-ResNet

stoptotal

method 2• Use of sensor information is essential.
2021/1/5

UEC Yanai Lab. Tokyo.

22

Conclusions•Proposed an image/sound/sensor-based four-stream CNN•The effectiveness of using sensor information iseffective.
2021/1/5

UEC Yanai Lab. Tokyo.

23

Future challenges•Use of time series information
Video information :2D conv

Video information :3D conv•Expansion of the data set
Increase in the number of datasets that include video andsensor information

2021/1/5

UEC Yanai Lab. Tokyo.

24

2021/1/5UEC Yanai Lab. Tokyo.

25