Analyzing the similarities ofactions based on video clustering
Vu Gia Truong

Do Hang Nga

Keiji Yanai

The University of Electro-Communications, Tokyo1

Research area: computer vision• Our lab are working on– Video recognition• We are working on– Finding videos which have visually similar content.

2

Background– Verbs• Express actions.
• Represent by video shots

Running3

Background– Analyzing visual similarities between verbs has notbeen conducted.

Walking

Eating

Running

Writing

Which pair is most similar ? Which pair is most dissimilar ?

– Difficulties• How to represent verb by computable unit.
• How to collect data for several different actions.
4Research purpose• Analyzing visual similarities between actions.
= Finding verbs pairs whose corresponding actions
look similare.g. 「play+piano」 vs 「type+keyboard」
5

Method summary
1. Extract bag-of-spatial-temporal(BOSFT)vectors
1. Extract spatial-temporal features
2. Encode features using Bag-of-Features approach

2. Calculate attribution vector for each verb
1. Cluster all of BOSFT vectors
2. Calculate distribution rate per cluster

3. Evaluate similarities
1. Calculate distances between each verb
2. Rank distances– Low distance = high similarity61. Extract bag-of-spatialtemporal(BOSFT)vectors
1. Extract spatial-temporal features••

Spatial-temporal feature tracks motion of special points ineach frame during time.
For each video shot, several spatial-temporal feature
vectors are extracted

2. Encode features using Bag-of-Features approach••Bag-of-Features counts occurrence frequency of eachfeature vector based on a dictionary
Vector represent occurrence frequency of spatial-temporalfeatures is called Bag-of-spatial-temporal vector.

71.1 Extract spatial-temporal features
Extract
visualfeature

Decidetrackingpoint

Extract
motionfeatureExtract
vectordescriptor

vi vi vi

vi feature vi
vectorsvi vi vivi

A. Noguchi and K. Yanai. A surf-based spatio-temporal feature for feature-fusionbased action recognition. In Proc. of ECCV WS on Human Motion: Understanding,Modeling, Capture and Animation, 2010

1.2 Encode features using Bag-ofFeatures approachvi vivvi feature ivi vectorsvi

6vi vivivivi vivi vivivivivi vi
vivi vi
Dictionary

frequencyvi

543210Dictionary

2. Calculate attribution vector
1. Cluster all of BOSFT vectors•

Use Probabilistic latent semantic analysis (pLSA) to cluster–•

Set cluster count from 100 to 200

Calculate attribution vector of each video shot

2. Calculate attribution vector for each verb•

Calculate mean value of all attribution vectors of all
video shots belong to each verb10

2.1 Cluster all of BOSFT vectors• Use pLSA method to cluster all of BOSFT vectors

Cluster iVideorate

0.006

0.00420.00280.00240.00170.0011Cluster (i +1)

Videorate

0.004

0.003411

2.1 Cluster all of BOSFT vectors• Attribution vector of each video shot
• Is distribution rate per cluster• Has sum equal to 1

Video shot
j of verb

𝑥𝑗1
Cluster 2: 𝑥𝑗2
Cluster 1 :…….

Cluster i:

𝑥𝑗𝑖

…….

Cluster n:

Attribution vector

𝑥𝑗1 , 𝑥𝑗2 , . . , 𝑥𝑗𝑛

𝑥𝑗𝑛
12

2.2 Calculate distribution vector for each verb
Video shots

Attribution vector ofeach video shot

Attribution vectorof verb(𝑥11 , 𝑥12 , . . , 𝑥1𝑛 )
…….

…….

( x1 , x2 ,..., xn )m(𝑥𝑚1 , 𝑥𝑚2 , . . , 𝑥𝑚𝑛 )1xi  *  xijm j 113

3. Evaluate similarities
1. Calculate distances between each verb•

Use Jesen-Shannon divergence to calculate

2. Evaluate similarities based on distances••

Rank calculated distances
Find pairs of verbs which are visually similar14

3.1 Calculate distances between each verb
• Jesen-Shannon divergence

DJS (q || p)   DKL (q ||  q  (1   ) p)
(1   ) DKL ( p ||  q  (1   ) p)f ( x)DKL ( f ( x) || g ( x))   f ( x) logdxg ( x)15

3.2 Evaluate similarities
1. Find pairs of verbs which are visually similar• Rank calculated distances ascending•

Find pairs of verbs which are visually similar– Low rank score = high similarity

16

Experiment• Dataset: 13320 videos for 101 verbs

1. Evaluate visual similarities between verb
pairs
2. Find verb pairs which are very similar

17

Experiment data
• Dataset: UCF-101– Contain 13320 videos for 101 verbs– Divide into 5 main categories•••••

Human-Object Interaction
Body-Motion
Human-Human Interaction
Playing Musical Instruments
Sports18

Experiment result：similar verb pairs
Verb A
MixingThrowDiscs
ShavingBeard
SoccerJuggling
ShavingBeard
Haircut
FrontCrawl
PlayingTabla
PlayingTabla
MilitaryParade

Absolute
Relative
Verb B
distance
distance
CuttingInKitchen
0.01571.00
HammerThrow
0.0181
1.16
Haircut
0.0197
1.26
Nunchucks
0.0205
1.31
ApplyLipstick
0.0222
1.42
BrushingTeeth
0.0224
1.43
BreastStroke
0.02341.49
PlayingGuitar
0.0234
1.50
PlayingCello0.0239
1.53
BandMarching0.0241
1.54
19

Examples of visually similar verb pair• “ThrowDiscus” and “HammerThrow”– Body swings around20

Examples of visually similar verb pair• “PlayingGuitar” and “PlayingTablar”– Hands move in small area

21

Examples of visually un-similar verb pair• ”SoccerJuggling” and “NunChuck”– Body moves

22

Examples of visually un-similar verb pair• ”Haircut” and “BrushingTeeth”– Hand moves

23

Conclusion• Propose a novel approach to representverbs• Evaluate visual similarities between verbs• Future work
– Improve represent method to find visualsimilar verb pairs– Expand dataset• Add cooking related verbs• Add eating related verbs
24

Thank you for listening

25

Future work details
1. Improve representing method– Apply mid-level feature approach•

Extract feature descriptors from volumes of video

2. Expand dataset– Add other verbs•

Add cooking, eating related verbs– Add more sample video shots for each verb•

Expand to 200 vide shots per verb

26

Contributions of our work
• Analysis similarities between verbs• Can be applied to– Text-based video retrieval– Content-based video retrieval

27