This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier xx.xxxx/ACCESS.20xx.DOI

A Framework for Knowing Who is Doing
What in Aerial Surveillance Videos
FAN YANG1,2 , SAKRIANI SAKTI1,2 (Member, IEEE), YANG WU1 (Member, IEEE), SATOSHINAKAMURA1,2 (Fellow, IEEE),
1
2

Nara Institute of Science and Technology, Nara, JapanRIKEN, Center for Advanced Intelligence Project AIP, Japan

Corresponding author: Yang Wu (e-mail: yangwu@rsc.naist.jp, wuyang0321@gmail.com).

ABSTRACT Ultra-high-resolution aerial videos are used to relieve the shortage of surveillance systemin sparsely populated regions. For realistic application purpose, it is important to automatically analyze“who is doing what?” in such videos. Although atomic visual action (AVA) detection has been successfully
used to recognize “who is doing what?” in movie data, it is challenging to adapt it to ultra-high-resolution
aerial videos, where the target persons are relatively tiny and sparsely located. Besides, due to the lack ofevaluation metrics, AVA detection has been evaluated by the single-label action, however, using multi-label
actions in evaluation is more reasonable since several actions can be simultaneously performed by a person(e.g., making a phone call and walking). To tackle these issues, we propose a novel framework for multilabel AVA detection in ultra-high-resolution aerial videos, and, introduce novel metrics for multi-label AVA
detection evaluation. Experimental results demonstrate that our framework outperforms other methods forinterpreting “who is doing what?” in our target task.
INDEX TERMS Aerial Surveillance Videos, Multi-label Atomic Visual Action Detection.

I. INTRODUCTION

Surveillance cameras are commonly installed in city regions
to increase public safety. However, it is inapplicable todensely set up surveillance cameras in sparsely populated
regions (e.g., suburb), while the safety concern is needed
therein. Considering the fact that some of the sparsely populated regions are not covered by tall trees or buildings, it ispossible to periodically take surveillance videos by drones.
Due to drones’ mobility, a wide range of sparsely populated
regions can be monitored at a low cost.
To facilitate the efficiency of surveillance analysis, it isdesirable to automatically analyze “who is doing what?” insurveillance videos. In movie data, Atomic Visual Action(AVA) detection was proposed to detect the spatio-temporal
location and action for each person [1], which means, “whois doing what?” can be determined at each frame in videos.
Nonetheless, aerial surveillance videos have some special
properties and existing AVA detection methods may notwork properly on them. These special properties include:(1) to capture visual details from the sky, each frame ofaerial surveillance videos is preferred to be an ultra-highresolution image (e.g., 2160 × 3840); (2) relative to theentire aerial image, each person appears to be a tiny object

but could still contain a large amount of pixels, which
are sufficient for obtaining his/her actions; (3) persons aresparsely located; (4) the drone could move fast, resulting
in significant relative position shift of the targets even in
adjacent frames.
To approach AVA detection in aerial surveillance videos,
we specifically designed a new framework by constructing
new modules to seamlessly integrate object detection, multiobject tracking, and action recognition (see FIGURE 1).
Object detection plays a fundamental role in AVA detection, which locates each person in a spatial domain by bounding boxes. An ultra-high-resolution aerial image, however, is
too large to be the input of normal object detectors [2]–[5],while down-scaling it could impair detection performance.
As an alternative approach, an ultra-high-resolution aerial
image could be cropped into smaller patches before performing object detection. Some existing methods divide theentire aerial image into patches by a sliding window [6]–[8]. Although such methods have considerably improved
object detection performance, they are inefficient when target
objects are sparsely located. We propose a Clustering Region
Proposal Network (C-RPN) to alleviate this issue. C-RPN
works by only selecting patches that may include target
1

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

detection evaluation. To the best of our knowledge, existing
metrics cannot be applied to multi-label AVA detection, andwe are the first to introduce such metrics.
II. RELATED WORKSIn this section, we briefly discuss related works of object detection on aerial videos, AVA detection, and related datasets.

Ultra-high-resolution Aerial Image
2160 x 3840
Downscale
Generate
PatchesA. OBJECT DETECTION ON AERIAL VIDEOS
Cropping

Detection
Tracking

Patches 608 x 608

ActionRecognition

Multi-label
Actions

FIGURE 1: Overview of proposed framework to know “who isdoing what?” in aerial videos.

objects. Subsequently, the number of selected patches could
be fewer than using a sliding window when persons aresparsely located. In spite of that AVA detection estimatesactions at each frame (i.e., “is doing what”), spatio-temporalcontext is needed to obtain the person motion information.
Generally, spatio-temporal tubes are used in AVA detection
for such a purpose. Previous works [9], [10] obtain spatiotemporal tubes by extending bounding boxes from the central
frame to nearby ones. In drone-recorded aerial videos, even
if the absolute location of a person is static, its relative
location may shift remarkably due to the drone movement.
To eliminate the effect of drones’ movement, we constructspatio-temporal tubes by a multi-object tracking method [11],and then align a spatio-temporal tube referred to its first
frame. Since non-target objects might be included in thespatio-temporal tubes, action recognition performance could
be affected. To tackle this issue, we assume the target person
can be consistently observed in his/her spatio-temporal tube
while others may not. Based on this assumption, we propose
a novel Spatio-temporal Attention Module (STAM) to obtainattention for the target person in the spatio-temporal tube.
In addition, solely proposing the AVA detection framework
is insufficient to fully analyze “who is doing what?” inaerial surveillance videos. In AVA detection, it is intuitive toconsider that each person could take several actions simultaneously, which are corresponding to multi-label actions. For
instance, a person could be making a phone call and walking
at the same time. Due to the lack of evaluation metrics, AVA
detection has been evaluated with only the single-label action
for a while [1], [12]. Therefore, we provide novel metrics for
multi-label AVA detection evaluation, which also contributes
to the general AVA detection studies.
In summary, our contributions include: (1) proposing anovel framework for multi-label AVA detection on aerial
surveillance videos, which outperforms other methods in our
experiments; (2) providing novel metrics for multi-label AVA

Detecting tiny objects is a nontrivial problem and manystudies are trying to tackle it. Basically, there could be twocases in tiny object detection. One is the entire image hasa low resolution and thereby the tiny objects only containa few numbers of pixels. To improve the detection performance, amplification [13] and resolution enhancement [14]are applied. In another case, the object itself has plenty of
pixels, but the object only constitutes a very small portionof the entire image so that it is relatively tiny. An ultrahigh-resolution aerial image belongs to the second case andperforming object detection on the original image size isdesired.
Although the idea of transforming each frame of aerial
videos into smaller patches for object detection has beenaround for some time [6]–[8], it is only recently that region
proposals and clustering have been jointly applied to reduce
the number of patches when objects are sparsely located [15].
Using the downsized aerial image, promising regions that
may contain objects can be learned by density map regression. Based on the predefined patch size, these regions canbe further clustered by their relative distances.
We assume that a good clustering strategy should satisfy
two conditions: first, reducing the number of patches; second, keeping the object appearance complete in patches.
However, to some extent, these two conditions work against
each other. Solely satisfying the first condition may lead to an
object being partially cropped, while assigning each object
to a patch can effectively satisfy the second condition but
may introduce redundant patches. In the previous study [15],grid-based clustering is used. Nevertheless, it is limited bypredefined grid size and location, and thus objects may be
incompletely cropped and further affect the bounding box
detection. To resolve this issue, peak point Non-Max Suppression (NMS) and hierarchical clustering are used in our
C-RPN, attempting to make every object complete in at least
one patch.
B. ATOMIC VIDEO ACTION DETECTION

Atomic Video Action (AVA) detection concentrates on the
study of the action unit in videos. Just like the word unit
(i.e., the unit of natural language processing) is exploredto understand an article, the phoneme unit (i.e., the unitof speech) is analyzed to understand human speaking, andthe object detection/segmentation (i.e., the unit of image) is
studied to understand high-level visual tasks in images. AVA
detection is important for understanding video scenes.

2

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

Actions

Boxes 1:L
(link by tracking)Actionsii). 2D
ConvNeti). LSTM

Temporal Stacking

Cropping
2D
Features 1:L

Boxes 1:L

(link by tracking)2D ConvNet

Detector

Images 1:L

the entire image’s global coordinate, our inputs can only bealigned at the box level. Therefore, it is challenging to jointly
detect bounding boxes and actions in our framework. Similar
to FIGURE 2b, our framework (i.e., FIGURE 2e) generates
boxes by an independent detector and then connects boxes inthe temporal domain by a multi-person tracking algorithm.
Moreover, we propose a STAM to focus on the target object
at each frame and use a 3D ConvNet for action recognition.

Images 1:L

C. RELATED DATASETS(a) [9](b) i): [16] ii): [17]Actions

Actions

2D RoI & Temporal Pooling
Spitted 2D
Features1:Lii). 3D
ConvNeti). Temporal
Pooling

3D RoI Pooling
Boxes 1:L

(link by tracking)3D
Features

3D Box3D
Features(Extended bycentral 2D box)

3D ConvNet

3D ConvNet

Images1:L

Images1:L(c) [18](d) i): [1] ii): [10]Multi-label Actions
3D ConvNet

⨁

⊙

STAM

2D
Attention1:L

Detector⨁ : Concatenation
2D
Features1:L⊙ : Multiplication

2D ConvNet
Boxes 1:L

(link by tracking)

Cropping

AerialImage1:L(e) Our proposed structure.
FIGURE 2: A visualization of model structures for AVA detection when only RGB data is used. L denotes the number of frames
used in the model. i) and ii) represent different models that share thesame structure at the beginning.

Several models that can be used for AVA detection are
illustrated in FIGURE 2. FIGURE 2a, FIGURE 2c and FIGURE 2d learn actions and boxes by networks with end-to-end
training, while FIGURE 2b and FIGURE 2e use independent
detectors to generate boxes, and then connect boxes by trackers. This means generating spatio-temporal tubes and predicting actions are separate steps. Actions and boxes are jointly
generated in FIGURE 2a, and boxes are linked to form tubes
by an offline tracking. To better model the spatio-temporal
information, FIGURE 2c and FIGURE 2d learn features by
a 3D ConvNet. The main difference is that FIGURE 2cgenerates boxes and performs 2D Region of Interest (RoI)
pooling for each frame, while FIGURE 2d extends the central
frame boxes to adjacent frames. Additionally, FIGURE 2cand FIGURE 2d.i) apply temporal pooling to fuse features,
while FIGURE 2d.ii) uses a 3D ConvNet to process featuresand obtain a better action recognition performance.
Owing to the divergence of the patch’s local coordinate andOther than AVA detection, the primary focus of aerial video
study has been object detection and tracking [19]–[21]. In
this paper, since we concentrate on multi-label AVA detection in aerial videos, we utilize Okutama-action dataset [12]for our experiments. The dataset comprises 43 minute-longdrone-recorded aerial videos, with fully annotated boundingboxes in each frame and corresponding multi-label action
classes. In all, there are 12 categories of human actions:Handshaking, Hugging, Reading, Drinking, Pushing/Pulling,
Carrying, Calling, Running, Walking, Lying, Sitting andStanding. In the multi-label action annotation, one action
class could associate with another one. For instance, “Reading” and “Sitting” could be assigned to the same person at
the same time.
III. METHODOLOGY

Our proposed framework coherently generates patches,bounding boxes, spatio-temporal tubes, 2D CNN features, attention maps, and multi-label action classes (see FIGURE 3).
Using a video frame of size 2160 × 3840, our C-RPN first
generates patches of size 608 × 608. Based on selected
patches, normal detectors (e.g., YOLOv3-tiny [5]) can generate fine-grained bounding boxes for each person. After that,
fine-grained bounding boxes are connected to form spatiotemporal tubes by a multi-person tracking algorithm (e.g.,
Deep SORT [22]). Next, we sample L frames from spatiotemporal tubes and obtain their corresponding 2D CNN
features. STAM then takes 2D CNN features to generateattention maps that focus on target persons. In the end, theconcatenation of 2D CNN features and their multiplication
with attention maps, are used to estimate multi-label action
classes by a 3D ConvNet. For the overall processing, it isa special multi-label AVA detection that serves for aerial
surveillance videos.
A. CLUSTERING REGION PROPOSAL NETWORK(C-RPN)The Clustering Region Proposal Network (C-RPN) takes
downsized aerial images (544 × 960) as its input. Since each
person is relatively tiny compared with the aerial image, thecoarse position of person could be modeled by a 2D Gaussian
density map. The mean of 2D Gaussian is the centroid of aperson and the covariance represents the uncertainty of thisposition, which is set to be roughly half of the bounding box
size. Thus, coarse person locations can be learned by density
map regression. Based on the predefined patch size, coarse
3

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

Fan Yang et al.: Multi-label AVA detection on Aerial videos

⊙

⨁

⊙

⨁

⊙

⨁

Actions

3D ConvNet

Spatio-temporal Attention Module (STAM)

Frame
k+1
Patches

Attention
Masks

2D CNN
Features

Sampling L frames for action recognition

Frame
k+1
Image

Frame
k-1
PatchesFramek
Patches

Attention Action Recognition Network(AARN)

Tubes

Detectors

Frame
k
Image

Clustering Region Proposal Network (C-RPN)

Frame
k-1
Image

Cropping
Resizing

2D ConvNet

Boxes

Patches

Actions

Actions⨁ : Concatenation⊙ : Multiplication

Multi-person Tracking

FIGURE 3: Architecture of the proposed framework. Given each ultra-high-resolution aerial image of size 2160 × 3840, C-RPN is
utilized to select patches (608 × 608) that might contain persons. Based on selected patches, normal detectors are used to generate finegrained bounding boxes for each person. After that, fine-grained bounding boxes are further connected to be spatio-temporal tubes by amulti-person tracking algorithm. Next, we sample L frames from spatio-temporal tubes and obtain their corresponding 2D CNN features.
STAM then takes 2D CNN features to generate attention maps that focus on target persons. In the end, the concatenation of 2D CNN features
and their multiplication with attention maps, are used to estimate multi-label action classes by a 3D ConvNet.

person locations can be further clustered by their relative
distances and patches that may contain persons are generated(see FIGURE 4).
At frame k, let the network output of C-RPN be Hkpred
and its ground truth be Hktrue . When both Hkpred and Hktrue
have a row number of R and a column number of C, we canrepresent them by
Hkpred =Hktrue =R [C[
r=1 c=1
R [
C
[

hpred
krc ,

(1)
htrue
krc ,

r=1 c=1

where r and c are the row index and column index of thetrueheat map, respectively; hpred
krc and hkrc denote the pixel at
pred
true
position [r, c] of Hk
and Hk , respectively.
The htrueisgenerated
by
krc
htrue
krc =
N
X

exp −

r − pki (x) ∗ s1 ∗ s2

i=1



1,
htrue
krc ,

2

+ c − pki (y) ∗ s1 ∗ s2
2
2σki

2 
;

if htruekrc > 1;else.

(2)where [pki (x), pki (y)] are the center coordinates of the ith
ground-truth bounding box. Since the overlapping boxes may
generate values larger than 1, we clip the maximum value ofhtrue
krc at 1. The downscale factor from original image to CRPN input is denoted as s1 , and the down-sampling factor
from C-RPN input to C-RPN output is denoted as s2 . In thiswork, we set s1 ≈ 1/4 (to be divisible by s2 ) and s2 = 1/8.
4

Lraw_pos,k

=−



htrue
krc =

More specifically, σki , pki (x) and pki (y) are generated by
s1 ∗ s2
max
min
max
min
σki =
(xki − xki ) + (yki − yki ) ;4
s1 ∗ s2 max(3)pki (x) =(xki + xminki );2
s1 ∗ s2 max
minpki (y) =(yki + yki);
2minmax max
where [xminki , yki , xki , yki ] are corner positions of theth
i ground-truth bounding box at frame k. Here, σki isroughly half size of the bounding box i at frame k.
We modify a penalty-reduced pixel-wise logistic regression with focal loss [23] and let it be our loss function
Lraw_pos,k as follows:







R
C
XX



α


1 − hpredlog hpred
,
krc
krc
true
if hkrc = 1;


α



pred
pred
true β

(1
−
h
)
h
log
1
−
h
,
krc

krc
krc


otherwise;(4)
where α and β are hyper parameters for focal loss and wefollow work [23] to set α and β to be 2 and 4, respectively.
Ideally, each object center is a peak point on this density
map, thus, we can apply peak point Non-Max Suppression(NMS) to obtain corresponding peak points. Nonetheless,there is no magic in the network of C-RPN, and it is still
suffering the dilemma of detectors in setting a confidence
threshold: better precision, or better recall. In C-RPN, although false-positive (FP) peak points may generate redunr=1 c=1 VOLUME 4, 2019

Fan Yang et al.: Multi-label AVA detection on Aerial videos

B. ATTENTION ACTION RECOGNITION NETWORK(AARN)

In our approach, Deep SORT [22], a multiple-object tracking
method, is employed to link bounding boxes into spatiotemporal tubes. Deep SORT takes an IoU (Intersection overUnion) descriptor, an appearance descriptor, and a Kalmanfilter to perform bipartite bounding box assignments acrossframes. The appearance descriptor, which is used to overcome occlusion and long-time tracking issues, is a CNN
network trained on a person re-identification dataset [24] bya Cosine Softmax Classifier [25].
After obtaining the spatio-temporal tube for each person,
we obtain their actions at each frame by a novel Attention
Action Recognition Network (AARN). Since AVA detection
focuses on instantaneous actions other than long-term actions, we only take a short-term temporal context and sample
L frames from each spatio-temporal tube for action recognition. Frames within 2 seconds (i.e., 60 frames in 30 FPS
videos) ahead of the target frame are excluded. For a person
whose track ID is n, we denote the earliest and latest frames
in the corresponding spatio-temporal tube as kmin and kmax ,respectively. Setting kmax as the target frame, then L frames
are sampled to form a set {xn0 , xn1 , ..., xnL } ∈ Xknmax
for action recognition. The details of our online samplingstrategy are described in Algorithm 1.
Instead of directly processing RGB data Xknmax by 3D
ConvNet, we extract their corresponding 2D CNN features{f1n , f2n , ..., fLn } ∈ Fknmax at the first step. Then, we proposed a Spatio-temporal Attention Module (STAM), which
VOLUME 4, 2019

≈1/4 Arial Image

Arial Image& Patches

544 x 960

2160 x 3840

Hierarchical
Clustering

Down-sampling
Encoder
Skip
connections

320

Decoder

Rescaling
Density Map

Peak Points

68 x 120

1x1 Conv_1

2D
ResBlock_32

3x3 (2,2)
De-Conv_64

2x2
Max-pool
2x2
Max-pool

2D
ResBlock_64

2D
ResBlock_32
2D
ResBlock_128

3x3 (2,2)
De-Conv_128

2x2Max-pool
2x2
Max-pool

3x3, 2x2, 1x1: Kernel size or Pool size
(2, 2): Strides⊕ : Element-wise adding

2D
ResBlock_256

2D
ResBlock_32
2D
ResBlock_64

68 x 120
3x3 (2,2)
Conv_32

dant patches, such a redundancy has little effect on the
final fine-grained object detection. Therefore, we set a low
confidence threshold for peak point NMS to obtain peak
points, regardless of it may end up with low precision andhigh recall.
Because peak points could be sparsely distributed, grouping neighboring peak points to guide patch generalization can
reduce the number of patches. As we have discussed in thesection of related works, grid-based clustering may not fit ourrequirements as it may incompletely crop person appearance
in all patches. To make a trade-off between reducing thenumber of patches and preserving the objects appearance, wechoose hierarchical clustering. In hierarchical clustering, by
adjusting the threshold distance to generate suitable overlapping regions dynamically, we could make person appearancecomplete in at least one patch.
We do not need to specify how many persons are included in each patch, because another object detector (e.g.,
YOLOv3) will take patches as inputs to generate boundingbox for each person. Since overlapping patches could be
generated, we not only have duplicated boxes in the same
patch, but also have duplicated boxes on the overlapping
regions between patches. In our approach, therefore, we only
perform bounding box NMS once after transferring boundingboxes from the patch coordinate to the original aerial image
coordinate.

⊕⊕

FIGURE 4: The demonstration of generating patches by C-RPN.
The downscale factor s1 ≈ 1/4, and the down-sampling factor s2 =1/8.
Algorithm 1: On-line sampling from a spatio-temporal tuben
Input : Spatio-temporal tube T[kmin :kmax ]n
1 if len(T[k) < L thenmin :kmax ]n
2
Xknmax ← {T[k+ Repeat Padding withmin :kmax ]nTkmax };
3 elsen4δ = len(T[max(k)//L;min ,kmax −60):kmax ]n5
Xkmax ← {Randomly choose L frames fromnT[max(kwith the interval δ}.
min ,kmax −60):kmax ]Output: Xknmax

is a 3D encoder-decoder with skip connections, to generateattentions maps {an1 , an2 , ..., anL } ∈ Ankmax by encoding anddecoding the global spatio-temporal representation of Fknmax .
After that, we perform element-wise multiplication between
Fknmax and Ankmax , and concatenate with Fknmax to obtain arepresentation that can selectively focus on the target person
across all frames. Finally, aforementioned 2D CNN featuresare stacked to be 3D CNN features, which are then fed toa 3D ConvNet to estimate multi-label action classes (seeFIGURE 6).
Although it is common to utilize optical flow for action
recognition, we do not use it in our framework. In drone5

Fan Yang et al.: Multi-label AVA detection on Aerial videos

Adjacent frames

Optical flow map

FIGURE 5: Visualizations of optical flow maps generated by PWC-Net [26], using Okutama-action Dataset. Due to the tiny size of person
and the drone camera movement, it is challenging to obtain person motion information from the optical flow.

3D
ResBlock_96

3x3x3 (2,2,2)
De-Conv_96

3D
ResBlock_96

3x3x3 (2,2,2)
De-Conv_128

3D
ResBlock_256

3x3x3, 2x2x2, 1x1x1: Kernel size or Pool size
(2,2,2): Stride⨁ : Concatenation operation

2x2x2
Max-pool

3D
ResBlock_128

2x2x2
Max-pool

3D
ResBlock_96

Spatio-temporal
Attention
Module

⊕
⊕

𝒙𝒏𝑳#𝟏

𝒙𝒏𝑳

2D ConvNet

𝒙𝒏𝑳#𝟐

3D Temporal Stacking

Person tracking ID: n

𝒇𝒏𝑳#𝟐

Spatio-temporal
Attention
Module

𝒂𝒏𝑳#𝟐
𝒂𝒏𝑳#𝟏

𝒇𝒏𝑳#𝟏

𝒂𝒏𝑳

𝒇𝒏𝑳
Multiplication⊙⨁ Concatenation
Stacking to

3D CNN features

3D ConvNet
Actions

FIGURE 6: An illustration of proposed Attention Action Recognition Network (AARN), with its Spatio-temporal Attention
Module (STAM). Three frames are used in this illustration, wherenn{xn
L−2 , xL−1 , xL } are RGB features sampled by Algorithm 1,
and they are fed to 2D ConvNet to generate 2D CNN features
n
n
{fL−2
, fL−1, fLn }. STAM takes stacking 2D CNN features to obnntain corresponding attention maps {anL−2 , aL−1 , aL }. The multiplication results of 2D CNN features and attention maps areconcatenated with 2D CNN features again, and then be used toestimate multi-label actions by a 3D ConvNet.

recorded aerial videos, even if the absolute location of an
instance is static, its relative location may have a huge change
across nearby frames, which is caused by the drone camera
movement and tiny object size. In Okutama-action data, weuse a state-of-the-art optical flow generator [26] to produce
optical flows between nearby frames, and show them in
FIGURE 5. We can see, it is hard to identify the movementof each person in the optical flow map.
6

IV. EVALUATION METRICS FOR MULTI-LABEL AVA
DETECTION

The evaluation metrics for object detection and multi-label
classification have been well studied separately [27], [28], but
the problem remains on how to associate them together for
multi-label AVA detection evaluation.
A simple approach could be evaluating the “person” object
detection performance for all detected samples and thenevaluating the multi-label action recognition performance forpositively detected samples. For instance, assuming that a
predicted sample is positive when IoU ≥ 0.5 for the predicted
and ground-truth bounding boxes, we can apply h.l.@0.5,
which corresponds to Hamming Loss associated with IoU≥ 0.5, to measure its multi-label classification performance.
Below, we show how the h.l.@0.5 is extended from theoriginal Hamming Loss.
h.l.@0.5 =1

1

Npersons@0.5 Nlabels

Nperson@0.5 Nlabels

X

X

i=1

l=1

i,l
i,l
Ytrue
XOR Ypred,(5)

where XOR is an exclusive-or operation and Nlabels stands
for the number of action categories. Ytrue and Ypred areboolean arrays that denote the ground truth and predictedlabels, respectively. The number of positively detected samples are represented by Npersons@0.5 . To help understand theabove metrics, we illustrate how h.l.@0.5 is calculated by atoy example in FIGURE 7.
Due to the complexity of multi-label classification evaluation, usually more than one criterion is included to inspect theperformance from different perspectives [28]. By applying
the same modification as h.l.@0.5, we propose another threecriteria as follows.
•

co.@0.5: this is extended from Coverage, and it evaluates how far on average it is necessary to go throughthe ranked scores to cover all true labels for positive
samples (IoU≥ 0.5).
VOLUME 4, 2019

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

IoU = 0.52

IoU = 0.15

IoU = 0

1010

1010

1010

1001

1001

CountedNot counted

IoU = 0Bounding Boxes:
Ground Truth Ytrue [Green]
Prediction Ypred [Red]Multi-label Actions:
Ground Truth Ytrue [Green]
Prediction Ypred [Red]

Npersons@0.5 = 1, Nlabels = 4,1001
Not counted

Not counted

Not counted

h.l.@0.5 = 0.5

FIGURE 7: An example of calculating h.l.@0.5. Only the first casewith IoU=0.52 is considered as a positively detected sample, andtherefore the overall h.l.@0.5=0.5.

r.l.@0.5: this is extended from Ranking Loss, and it
evaluates the average proportion of label pairs that areincorrectly ordered for a positive sample (IoU≥ 0.5).
• o.e.@0.5: this is extended from One Error, and it
evaluates the average number of top-ranked predictedlabels that are not true labels for the positive sample(IoU≥ 0.5).
•

V. EXPERIMENTS
A. TRAINING AND TESTING SETUP

By following the previous work [12], Okutama-action dataset
is split into a training set with 33 aerial videos and a testing
set with 10 aerial videos.
For C-RPN, the Adam [29] optimizer with a learning rate
0.001 is applied for the first 50 epochs and then the learning
rate is changed to be 0.0001 for another 150 epochs. The
batch size is set up to be 8. Images and their corresponding
density maps are jointly augmented by Albumentations [30].
We perform a peak point detection on a validation set (i.e.,
20% of the training set) and find that a density map canreach the confidence of 0.5 ∼ 1.0 and 0.0 ∼ 0.1 at thetarget and the non-target positions, respectively. To reach ahigh recall on the testing set, we set the peak point NMS
confidence threshold as 0.3. We search the maximum personbounding boxes size in Okutama-action dataset to decide thedistance threshold in peak point NMS. More specifically, themaximum person bounding box size is about 200 on theoriginal size image. Considering the total downscale from theoriginal size image (2160 × 3840) to the output density map(68 × 120) is about 32, the maximum person size on output
density map is about 6. Since distance threshold should be an
odd number, we take value 5 here. Using Python code, peak
point NMS can easily be implemented by1
2
3
4

from s c i p y . ndimage i m p o r t m a x i m u m _ f i l t e r
Peaks_map = ( H_pred > 0 . 3 ) *( H_pred == m a x i m u m _ f i l t e r ( H_pred ,f o o t p r i n t = np . o n e s ( ( 5 , 5 ) ) ) )Listing 1: Peak piont NMS

For other detectors used for comparison, as R-FCNResNet50 [3], Retinanet-ResNet50 [4], SSD-ResNet50 [2]and YOLOv3-tiny [5], we take their pre-trained weights on
COCO dataset [31] and fine-tune them on our experimental
datasets by their default training strategy.

To train AARN, we equally sample 64 ground-truth spatiotemporal tubes from each action class, and then sample
Xknmax from each spatio-temporal tube (see Algorithm 1). As
only part of the training samples are included in one epoch
training, it takes more iterations to get converged. We also
apply the Adam optimizer for it, with learning rate 0.001,
0.0001, and 0.00001 for each 500 epochs. The batch size
is set up to be 16. We perform the same data augmentation,i.e., flipping, rotation, resizing, and cropping to all samples in
Xknmax . During the inference process, Algorithm 1 is applied
again to obtain inputs for the inference process.
Even though we are working on AVA detection with largesize aerial videos, our framework decomposes the wholeproblem into multiple simple tasks. Thus, all our experiments
can be implemented on a single NVIDIA TITAN X GPU.
B. PERFORMANCE EVALUATION

Our proposed metrics evaluate the multi-label AVA detection performance by two steps. Firstly, we evaluate person
detection performance, by using the mAP@0.5 metrics [31].
Secondly, we evaluate multi-label action recognition performance for positively detected samples (i.e., a sample withmAP > 0.5). We jointly inspect the performance of two steps
to obtain the overall multi-label AVA detection performance.
1) Detection Evaluation

For the person detection evaluation, our main purpose isto verify three assumptions: (1) compared with detectorsthat work on the downsized aerial image (608 × 608 withpadding), although using our proposed C-RPN may takemore running time, it should improve the person detection
performance; (2) compared with partitioning the entire aerial
image (2160 × 3840) into patches with a sliding window [8],our C-RPN should be faster when persons are sparsely located; (3) in contrast to grid-based clustering [15], using
hierarchical clustering with a proper distance threshold cankeep the complete appearance in at least one patch so that
our method can achieve better person detection performance.
For detectors that take the entire aerial image as input, westandardize their input size to be 608 × 608 by padding, since
it is difficult to train and test a detector with larger input size.
When the sliding window passes the aerial image margin, we
pad zeros to the inputs. To reach a fast speed, we choose
YOLOv3-tiny [5] as the base detector in our framework.
Although our default setting is hierarchical-clustering CRPN, for a fair comparison with previous work [15], we forma grid-clustering C-RPN by solely replacing the clustering
method.
The qualitative results of our patch generation and bounding box estimation are shown in FIGURE 8 and FIGURE 9,
respectively. The quantitative results of the Okutama-action
testing set are shown in TABLE 1. Taking the originalsize aerial images (2160 × 3840), our C-RPN + YOLOv3tiny achieves 85.2 mAP@0.5 in terms of “person” object
detection, which remarkably outperforms detectors that utilize downsized aerial images. Besides, by using C-RPN,
7

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

1.2.3-f300

1.2.3-f400

1.2.3-f500

2.2.1-f250

2.2.1-f300

2.2.1-f350

FIGURE 8: Patch proposals in Okutama-action testing sets, which are generated by C-RPN. Generated peak points are marked by red,and patches are enclosed by colorful rectangles. The first row shows three sequential frames (i.e., 300, 400 and 500) in video 1.2.3. The
second row shows three sequential frames (i.e., 250, 300 and 350) in video 2.2.1. To efficiently cover target persons, clusters automaticallymerge and split, based on the relative distance within peak points.
MethodUsing entire downscale image of size 608 × 608
R-FCN-ResNet50 [3]Retinanet-ResNet50 [4]SSD-ResNet50 [2]YOLOv3-tiny [5]Using Patches of size 608 × 608 (without downsizing)
Sliding (stride=388,404) [8]+YOLOv3-tiny
Sliding (stride=580,580) [8]+YOLOv3-tiny
C-RPN (Grid:gridsize =216 × 384) [15]+YOLOv3-tiny
C-RPN (Grid:gridsize =270 × 480) [15]+YOLOv3-tiny
C-RPN (Hierarchical:dthreshold = 128)+YOLOv3-tiny
C-RPN (Hierarchical:dthreshold = 320)+YOLOv3-tiny
C-RPN (Hierarchical:dthreshold = 512)+YOLOv3-tiny

mAP@0.5↑

Speed for entire image (FPS)↑

Average patches↓

53.5
56.3
52.3
52.4

6
10
18
120

-

82.0
79.4
77.5
78.3
85.0
85.2
82.9

3
5
25
28
26
30
38

45.0
28.0
3.9
3.7
3.8
3.1
2.2

TABLE 1: “Person” object detection performance on Okutama-action dataset. The symbol ↑(↓) indicates that the larger(smaller) thevalue, the better the performance.

the final object detection performance is even better thanusing a sliding window, since some ambiguous backgroundmight be excluded by C-RPN in advance. Last but not least,
because we try to make the person appearance complete inat least one patch, the performance of hierarchical-clustering
C-RPN outperforms grid-clustering C-RPN [15]. Moreover,
we quantitatively calculate the average number of patches
generated by each method in Okutama-action testing set.
When hierarchical-clustering C-RPN reach the best detection performance, it only generates 3.1 patches averagely
on Okutama-action testing set, which is more efficient thansliding window approach and similar to the grid-clustering
C-RPN. Therefore, our approach can achieve a comparable
speed of 30 FPS on the full resolution data.
2) Multi-label AVA Detection Evaluation

Better-AVA model [10] is one of the state-of-the-art models
for AVA detection on movie data. It performs AVA detection
for the central frame and need an odd number of frames

as its inputs. We modify it to jointly estimate multi-label
actions and bounding boxes. Its inputs are L frames ofdownscale aerial images (608 × 608 with padding), which
are sampled near the target frame. Due to the limitation of
our computational resource, we choose L = 5 for it.
To inspect whether our AARN can improve the action
recognition performance by introducing spatio-temporal attention, we construct an ablation study by replacing AARN
with I3D [32] and Lite ECO [33] in our framework. The
results of applying our proposed metrics are shown in TABLE 2.
Compared with Better-AVA, our framework achieves better performance in both person detection and multi-label
action recognition. Besides, our framework is faster thanBetter-AVA on our target task. Considering our framework
decomposes the whole pipeline into several independent
steps, less memory cost is needed in our framework.
Through introducing spatio-temporal attentions, our
AARN performs better than I3D and Lite ECO, in terms of

8

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

Pred: Carrying, Standing
True: Carrying, Standing

Pred: Carrying, Standing
True: Carrying, Standing

Pred: Pushing/Pulling, Walking
True: Pushing/Pulling, Walking

Pred: Reading, Sitting Pred: Reading, Standing
True: Reading, Sitting True: Reading, Standing

Pred: Walking
True: Calling, Walking

Pred: Standing
True: Carrying, Standing

Pred: Pushing/Pulling, Walking
True: Pushing/Pulling, Walking

Pred: Carrying, Walking
True: Carrying, Walking

Pred: Reading, Sitting
True: Reading, Sitting

Pred: Walking
True: Carrying, Walking

Pred: Sitting
True: Sitting

Pred: Lying
True: Lying

Pred: Sitting
True: Reading, Sitting

FIGURE 9: Examples of multi-label AVA detection results in our framework.
MethodOff-line multi-label AVA detection
Input: L × 608 × 608 (L frames of downsized
aerial images with padding)
Better-AVA [10] (L=5)
On-line multi-label AVA detection
Input: L × 96 × 96 (L frames of cropped
images from spatio-temporal tube)
Replacing AARN by I3D [32] in our framework (L=8)
Replacing AARN by Lite ECO [33] in our framework (L=8)
Our framework (L=8)

h.l.@0.5↓

co.@0.5↓

r.l.@0.5↓

o.e.@0.5↓

mAP@0.5↑

Speed (FPS)↑

0.20

3.73

0.19

0.35

54.1

8

0.14
0.15
0.13

3.45
3.46
3.38

0.14
0.13
0.11

0.28
0.27
0.25

85.2
85.2
85.2

14
15
14

TABLE 2: Multi-label AVA detection results. The symbol ↑(↓) indicates that the larger(smaller) the value, the better the performance. Only
RGB data is used in this test. Note, we choose L = 5 for Better-AVA due to computation memory limitation and it has to be an odd number.
While other models utilize L = 8 since instantaneous actions are defined in AVA detection. Except for Better-AVA, other action detection
models use bounding boxes that are generated by C-RPN + YOLOv3-tiny, which achieves mAP@0.5=85.2.

action recognition in our target task. Examples of attention
maps generated by STAM can be visualized in FIGURE10,
which shows that STAM can learn to focus on the target
person in an unsupervised manner.

propose a novel multi-label AVA detection framework andcorresponding evaluation metrics. Our framework gives theflexibility to replace its detector and tracker based on theneed, which makes it possible to train and infer all modules
on a single GPU. Thus, our framework can be more suitable
than existing solutions for multi-label AVA detection in aerial
videos. On a final note, our proposed evaluation metrics
are not limited to aerial videos, and other AVA detection
tasks can also leverage such metrics to perform a reasonableevaluation.

FIGURE 10: Visualization of attentions for the target person. We
assume that the target person consistently appears in his/her spatiotemporal tube while others may not. The attention mask is learnedin an unsupervised manner.

REFERENCES

VI. CONCLUSION

The aerial surveillance videos make it possible to increase
public safety in sparsely populated regions. To automaticallyanalyze “who is doing what?” in such videos, we specifically[1] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar et al., “Ava: A
video dataset of spatio-temporally localized atomic visual actions,” CoRR,
abs/1705.08421, vol. 4, 2017.
[2] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C.
Berg, “Ssd: Single shot multibox detector,” in European conference on
computer vision. Springer, 2016, pp. 21–37.
[3] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via regionbased fully convolutional networks,” in Advances in neural information
processing systems, 2016, pp. 379–387.
[4] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for
9

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

dense object detection,” IEEE transactions on pattern analysis and machine
intelligence, 2018.
[5] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv
preprint arXiv:1804.02767, 2018.
[6] R. Porter, A. M. Fraser, and D. Hush, “Wide-area motion imagery,” IEEE
Signal Processing Magazine, vol. 27, no. 5, pp. 56–65, 2010.
[7] P. C. Hytla, K. S. Jackovitz, E. J. Balster, J. R. Vasquez, and M. L. Talbert,“Detection and tracking performance with compressed wide area motion
imagery,” in Aerospace and Electronics Conference (NAECON), 2012
IEEE National. IEEE, 2012, pp. 163–170.
[8] A. Van Etten, “You only look twice: Rapid multi-scale object detection insatellite imagery,” arXiv preprint arXiv:1805.09512, 2018.
[9] V. Kalogeiton, P. Weinzaepfel, V. Ferrari, and C. Schmid, “Action tubelet
detector for spatio-temporal action localization,” ICCV, Oct, vol. 2, 2017.
[10] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “A better baseline
for ava,” arXiv preprint arXiv:1807.10066, 2018.
[11] A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler, “Mot16: A
benchmark for multi-object tracking,” arXiv preprint arXiv:1603.00831,
2016.
[12] M. Barekatain, M. Martí, H.-F. Shih, S. Murray, K. Nakayama, Y. Matsuo,
and H. Prendinger, “Okutama-action: An aerial view video dataset for
concurrent human action detection,” in 1st Joint BMTT-PETS Workshop
on Tracking and Surveillance, CVPR, 2017, pp. 1–8.
[13] P. Hu and D. Ramanan, “Finding tiny faces,” in Proceedings of the IEEE
conference on computer vision and pattern recognition, 2017, pp. 951–
959.
[14] Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Finding tiny faces in the wildwith generative adversarial network,” 2018.
[15] R. LaLonde, D. Zhang, and M. Shah, “Clusternet: Detecting small objects
in large scenes by exploiting spatio-temporal information,” in Computer
Vision and Pattern Recognition, 2018.
[16] J. He, Z. Deng, M. S. Ibrahim, and G. Mori, “Generic tubelet proposals
for action localization,” in IEEE Winter Conference on Applications ofComputer Vision. IEEE, 2018, pp. 343–351.
[17] Z. Li, W. Wang, N. Li, and J. Wang, “Tube convnets: Better exploiting
motion for action recognition,” in IEEE International Conference on Image
Processing. IEEE, 2016, pp. 3056–3060.
[18] R. Hou, C. Chen, and M. Shah, “Tube convolutional neural network (tcnn) for action detection in videos,” in IEEE international conference on
computer vision, 2017.
[19] A. Robicquet, A. Sadeghian, A. Alahi, and S. Savarese, “Learning social
etiquette: Human trajectory understanding in crowded scenes,” in European conference on computer vision. Springer, 2016, pp. 549–565.
[20] M. Mueller, N. Smith, and B. Ghanem, “A benchmark and simulator for
uav tracking,” in European conference on computer vision. Springer,
2016, pp. 445–461.
[21] P. Zhu, L. Wen, X. Bian, H. Ling, and Q. Hu, “Vision meets drones: A
challenge,” arXiv preprint arXiv:1804.07437, 2018.
[22] N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking
with a deep association metric,” in IEEE International Conference on
Image Processing. IEEE, 2017, pp. 3645–3649.
[23] H. Law and J. Deng, “Cornernet: Detecting objects as paired keypoints,” inProceedings of the European Conference on Computer Vision, 2018, pp.
734–750.
[24] L. Zheng, Z. Bie, Y. Sun, J. Wang, C. Su, S. Wang, and Q. Tian, “Mars:
A video benchmark for large-scale person re-identification,” in European
Conference on Computer Vision. Springer, 2016, pp. 868–884.
[25] N. Wojke and A. Bewley, “Deep cosine metric learning for person reidentification,” in 2018 IEEE Winter Conference on Applications of Computer Vision. IEEE, 2018, pp. 748–756.
[26] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “Pwc-net: Cnns for optical flowusing pyramid, warping, and cost volume,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2018, pp. 8934–
8943.
[27] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman,“The pascal visual object classes (voc) challenge,” International journal ofcomputer vision, vol. 88, no. 2, pp. 303–338, 2010.
[28] E. Gibaja and S. Ventura, “A tutorial on multilabel learning,” ACM
Computing Surveys, vol. 47, no. 3, p. 52, 2015.
[29] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014.
[30] E. K. V. I. I. A. Buslaev, A. Parinov and A. A. Kalinin, “Albumentations:fast and flexible image augmentations,” ArXiv e-prints, 2018.

[31] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,”in European conference on computer vision. Springer, 2014, pp. 740–755.
[32] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new modeland the kinetics dataset,” in IEEE Conference on Computer Vision and
Pattern Recognition. IEEE, 2017, pp. 4724–4733.
[33] M. Zolfaghari, K. Singh, and T. Brox, “Eco: Efficient convolutional
network for online video understanding,” in Proceedings of the European
Conference on Computer Vision, 2018, pp. 695–712.

FAN YANG received a B.S. degree and a M.S
degree from Nanjing University and Nara Instituteof Science and Technology in 2012 and 2018,
respectively. He is currently a Ph.D. candidate
at Nara Institute of Science and Technology. His
research focus on video processing.

SAKRIANI SAKTI received her B.E. degree inInformatics (cum laude) from Bandung Instituteof Technology, Indonesia, in 1999. In 2000, she
received DAAD-Siemens Program Asia 21st Century Award to study in Communication Technology, University of Ulm, Germany, and received her
MSc degree in 2002. During her thesis work, she
worked with Speech Understanding Department,
DaimlerChrysler Research Center, Ulm, Germany.
Between 2003-2009, she worked as a researcher
at ATR SLC Labs, Japan, and during 2006-2011, she worked as an expert
researcher at NICT SLC Groups, Japan. While working with ATRNICT,
Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She
actively involved in collaboration activities such as Asian Pacific Telecommunity Project (2003-2007), A-STAR and U-STAR (2006-2011). In 20092011, she served as a visiting professor of Computer Science Department, University of Indonesia (UI), Indonesia. In 2011-2017, she was an
assistant professor at the Augmented Human Communication Laboratory,
NAIST, Japan. She served also as a visiting scientific researcher of INRIA
Paris-Rocquencourt, France, in 2015-2016, under JSPS Strategic Young
Researcher Overseas Visits Program for Accelerating Brain Circulation.
Currently, she is a research associate professor at NAIST, as well as a
research scientist at RIKEN, the Center of for Advanced Intelligent Project
AIP, Japan. She is a member of JNS, SFN, ASJ, ISCA, IEICE and IEEE. She
is also the officer of ELRA/ISCA Special Interest Group on Under-resourced
Languages (SIGUL) and a board Member of Spoken Language Technologies
for Under-Resourced Languages (SLTU). Her research interests include
statistical pattern recognition, graphical modeling framework, deep learning,
multilingual speech recognition synthesis, spoken language translation,affective dialog system, and cognitive communication.

10

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI
10.1109/ACCESS.2019.2924188, IEEE Access
Fan Yang et al.: Multi-label AVA detection on Aerial videos

YANG WU received a BS degree and a Ph.D
degree from Xi’an Jiaotong University in 2004
and 2010, respectively. From Sep. 2007 to Dec.
2008, he was a visiting student in the GRASP
lab at University of Pennsylvania. From 2011 to2014, he was a program specific researcher at
the Academic Center for Computing and Media
Studies, Kyoto University. Within this period, he
was an invited academic visitor at the Big Data
Institute of University College London from Jul.
2014 to Aug. 2014. He is currently an assistant professor of the NAIST International Collaborative Laboratory for Robotics Vision, Institute for Research
Initiatives, Nara Institute of Science and Technology. His research is in thefields of computer vision, pattern recognition, and image/video search andretrieval, with particular interests in human-centric computer vison problems(detection, tracking, pose estimation, recognition, re-identification, action,
activities, etc.). He is also interested in pursuing general data analysis models(with various kinds of supervision) for real applications.

SATOSHI NAKAMURA is a Professor of Graduate School of Science and Technology, Nara Institute of Science and Technology, Japan, Project
Leader of Tourism Information Analytics Team of
RIKEN, Center for Advanced Intelligence Project
AIP, Honorarprofessor of Karlsruhe Institute ofTechnology, Germany, and ATR Fellow. He received his B.S. from Kyoto Institute of Technology
in 1981 and Ph.D. from Kyoto University in 1992.
He was Associate Professor of Graduate School ofInformation Science at Nara Institute of Science and Technology in 19942000. He was Director of ATR Spoken Language Communication ResearchLaboratories in 2000-2008 and Vice president of ATR in 2007- 2008. He
was Director General of Keihanna Research Laboratories and the Executive
Director of Knowledge Creating Communication Research Center, National
Institute of Information and Communications Technology, Japan in 20092010. He is currently Director of Augmented Human Communication laboratory and a full professor of Graduate School of Information Science at
Nara Institute of Science and Technology. He is interested in modeling andsystems of speech-to-speech translation and speech recognition. He is one of
the leaders of speech-to-speech translation research and has been serving forvarious speech-to-speech translation research projects in the world including
C-STAR, IWSLT and A-STAR. He received Yamashita Research Award,
Kiyasu Award from the Information Processing Society of Japan, Telecom
System Award, AAMT Nagao Award, Docomo Mobile Science Awardin 2007, ASJ Award for Distinguished Achievements in Acoustics. He
received the Commendation for Science and Technology by the Minister of
Education, Science and Technology, and the Commendation for Science andTechnology by the Minister of Internal Affair and Communications. He also
received LREC Antonio Zampolli Award 2012. He has been Elected Board
Member of International Speech Communication Association, ISCA, sinceJune 2011, IEEE Signal Processing Magazine Editorial Board Member sinceApril 2012, IEEE SPS Speech and Language Technical Committee Member
since 2013, and IEEE Fellow since 2016.

11

VOLUME 4, 2019

This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/.