男1絲氏41 (課程博士(英文))Date of submission (month day, year)

Depa11ment

MasakiAonoOf

Computer science

StudentlD Number

D179304

and Englneering
PP】icant's name

January 6 ', 2023

Sh喰eru Kuriyama
Supervisors

YuriYudhaswana JoefrieAbst"act (Doctor)
T北le ofThesis

On spatio・Temp01'aland Motion Feature Representation Learning with Deep NeuralNeNork
forAccurate video Action Recognition

Accurate action rec0今nition from digital videos is one of core and cha11en牙in今Computer vision problems that has been studied for decades. The potential
applications of action rec0牙nition include video survei11ance, human、computerinterface, visualinformation retrieval, and unmanned driving. How to P田'form
effective and efficient analysis of the video foota牙e is vital given the recent
exponentia1 曾rowth of survei11ance data on the lnternet. More and n〕.ore
individuals are becomin曾 accustomed to posting photographs and phrases on
Social n〕edia sites like lnstagram, Facebook, TW北ter, and Flickr to express theirfeehngs and ideas on almost any occasion or topic. As a result, studying the vastquantity of videos on social media has become in侃'easin牙ly important. Traditional
machine learnin晉 techniques that only extract computable characteristics have
Iimitations and need to function better with lar晉e amounts of visualinput.

However, convolutional neural networks (CNNS) and other deep learningtechniques have significantly improved in this area.
Even thoU旦h deep learnin牙 n)odels exhibit such prominent results, the proposed
Work for action rec0牙nition sti11 needs to improve, despite the nun〕erous solutions
that have been developed thus far. The model's ability to precisely detect chan曾es
in action variety, in addition to its ability to e任ectively recognize t雛'号ets andactions fron〕 the background, presents a significant cha11en牙e. The conventionalmodel wi11 be hampered as a result of this issue
Meanwhile, action recognition in videos can be divided into two tasks. Recognizing
Various actiV北ies consecutively or simultaneously in each video at an arb北rary
time is the first task. Each actiV北y may have a unique set of features. Anothertask is to recognize a sin牙le action inside a video Thus, diff田'ent approaches areneeded to deal with any issues that lnay develop.1n this dissertation, we propose
two approaches to tackle the cha11enges of action recognition and vaHdate them in
Cha11en曾in今 datasets.
＼刃e introduce deep fusion schemes for multi'1abel multi・class classification for thefirst task. This approach extends the previous work by introducin曾 a fusion of
Spatial and temporal branches to provide superior action rec0晉nition capability
toward multi'1abel multi、class classification problems. we propose three fusion
models with different fusion strate曾ies. we first build seV田'al efficient TemPの・alGaussian Mixture (TGM) 1ayers to form spatial and temporal branches to learn a
Set of features.1n addition to these branches, we introduce a new deep

Spatio'temporal branch consistin晉 of a series of TGM layers to learn the featuresthat emer今ed from the existin今 branches. Each branch produces a temporal"aware
feature that assists the modelin understandin牙 the underlyin今 action in a video
We verify the performance of our proposed models On the we11・known
MultiTHUMos benchmarking dataset＼刃e tackle severalissues re今ardin目 Spatio"temp01'al and motion feature learning
for the second approach. As these aspects are the key to action recognition, webuild a novelbuilding block for a 2D CNN・based network for efficient and e丘ective

Iearnin今 of the abovementioned features. Typical previous approaches utilize 3D
CNNs to cope with spatial and temporal features while they suffer from massive

Computations. other approaches are to utilize (1+2)D CNNs to learn spatial andtemporal features efficiently while they need to pay more attention to theimportance of motion representations. To overcome problems with previousapproaches, we propose a novel block that can capture spatial and temporalfeatures more faithfU11y and efficiently learn motion features. This proposed blockincludes Motion Excitation (ME), Multi'view EXC北ation (MVE), and Densely
Connected Temporal A牙牙re今ation (DCTA). The purpose of ME is to encodefeature・1evelframes difference; MVE is desi号ned to enrich spatiotemporalfeatures
W北h multiple view representations adaptively; DCTA is to mode110ng'ran今e

temporal dependencies. we inject the proposed building block, which we refer to as
the META block (or simply "META"), int0 2D ResNet'50. we verify thePerformance of our proposed model on three lar牙e、scale benchmarkin牙 datasets,includin牙 the somethin晉・something vl, Jester, and Moments in Time Mini
datasets

Extensive experimental studies prove that both approaches demonstrate
Competitive results on several benchmarkin号 datasets Compared to the baselineand state・of'the'art methods