Movement-to-Action Transformer Networks for Temporal Action Proposal Generation
Authors: Xiaodan Hu
Narendra Ahuja
Venue: Submitted to ICLR 2023 (2023)
Keywords: Temporal Action Proposal Generation
Video Action Segmentation

Abstract: The task of generating temporal action proposals is aimed at identifying temporal intervals containing human actions in untrimmed videos. For arbitrary actions, this requires learning long-range interactions. We propose an end-to-end Movement-and-Action Transformer Network (MatNet) that uses results of human movement studies to encode actions ranging from localized, atomic, body part movements, to longer-range, semantic ones, involving movements of subsets of body parts. In particular, we make direct use of the results of Laban Movement Analysis (LMA). We use LMA-based measures of movements as computational definitions of actions. We input RGB + Flow (I3D) features and 3D pose, compute LMA based low-to-high-level movement features from it, and learn the action proposals by applying two heads on the boundary Transformer and three heads on the proposal Transformer, and using five losses with different weights. We visualize and explain relations between the movement descriptors and attention map of the action proposals. We report results from extensive experiments on the Thumos14, ActivityNet and PKU-MMD datasets, showing that MatNet achieves SOTA or better performance on the temporal action proposal generation task.

Under review as a conference paper at ICLR 2023
MOVEMENT-TO-ACTION TRANSFORMER NETWORKS FOR TEMPORAL ACTION PROPOSAL GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
The task of generating temporal action proposals is aimed identifying temporal intervals containing human actions in untrimmed videos. For arbitrary actions, this requires learning long-range interactions. We propose an end-to-end Movement- to-Action Transformer Network (MatNet) that uses results of human movement studies to encode actions ranging from localized, atomic, body part movements, to longer-range, semantic movements involving subsets of body parts. In particular, we make direct use of the results of Laban Movement Analysis (LMA). . We use LMA-based measures of movements as computational definitions of actions. the input of RGB + Flow (I3D) features and 3D pose, we compute LMA based low- to-high-level movement features, and learn action proposals by applying heads on the boundary Transformer, three heads on the proposal Transformer and using five types of losses. We visualize and explain relations between the movement descriptors and attention map of the action proposals. We report results from a number of experiments on the Thumos 14, ActivityNet and PKU-MMD datasets, showing that MatNet achieves SOTA or better performance on the temporal action proposal generation task.
INTRODUCTION
Atami
Masiion
Repoer Vauape T Tower
The
Piice
M
L Susettio
mation
Anslysi
Weight
Function
Iipartite Matching
heald
10r30
leg
Proposal Head
loU Head
Clacificatioo Head
Proposal Transformer
Atomic Movements Descriptor
Start
End
LMA
Roundary Transtormer
Video Clip Low Features
Semantic Movement Descriptor
Temporal Action Proposal Generation
Figure 1: Overview of MatNet architecture. It contains main components: (1) Movement descriptors, shown in the middle column, and (2) Transformer networks for action proposal generation, shown in the right column. Given sequence of untrimmed video frames. MatNet uses Laban Movement Analysis constructs to generate body part (atomic) level and subset-of-pand ofrttrts (semantic) level descriptors of human movements from videos, and input as movement representations to action- boundary sensitive Transformer networks to generate action proposals.
With advances the understanding of trimmed, human action videos, the focus has begun to shift to longer and untrimmed videos. This has increased the need for segmentation of the videos into action clips, namely, identification of temporal intervals containing actions. This is the goal of the task of temporal action proposals generation (TAPG) for human action understanding. There many factors that make the problem challenging: (1) Location: an action can start at any time. (2) Duration:
The time taken by the action can vary greatly. (3) Background: Irrelevant content can
We will make all of the data sets, resources and programs publicly available.

---

Under review as a conference paper at ICLR 2023
Comp.
Description
Factor
Variahle
Cognition and Geometry
Movement dynamics:
Space Weight
c
g x2
Cognitve
Attention
[Direct.Indiret]
Effort
strength, control, tim- ing to reflect inner in- tention
2
R
Rxxi
Intention
(troongLLight]
Time
R Txt
Decision
SustennSSstained]
1. Non-Kinemati
Flow
Progression
Bound
Vertical
Skising.Sinking
Shaping
E R Txx
Plane
Horizontal
Spreedinng, Enclosingg
Shape
Shape change during
Sagittal
movement
Directional
Co
R gTxia
(Spoke like. Akk-AArk- like)
Shape Flow
C7
Txi
|
Body
Structural physical properties of moving body
Body
08 R
2. Kinematic
Space
Movement patterns,
trajectories tension
3.Relat-
Interaction among
jonships
people, body parts & objects
Table I: LMA classifies movement into three main categories (Col 1) Santos (2014): (1) Non- Kinematic (2) Kinematic, each represented by two Components (Col 2) Effort and Shape for (I), and Body and Space for (2). Category is about Relationships between (1) and (2) (Col 2). The components describe the categories using eight Factors, denoted by multidimensional variables { (Col 4), each having a different dimension (Col 5). Each Factor is also associated with movement's underlying (cognitive) category, and its geometric structure and space (plane), that capture the position, direction, rotation, velocity, acceleration, distance, curvature and volume associated with the movement, and have values between two extremes (Col 7). Human movement analysts point out that although each individual may combine the Factors in ways that are specific to the individual and their cultural. personal and artistic preferences, {c { remain valid for all movements Bartenieff & Lewis (1980) and human activities Santos (2014). and can be used to describe human movement at the semantic level. In this paper, we use these Factors as bases obtain temporal action proposals through MatNet. MatNet automatically determines combinations of the Factors most suited for action detection and localization.
be highly diverse. (4) Number of Actions:
Unknown and unlimited. (5) Action Set: Unknown (6)
Ordering: Unknown. Many problems can benefit from accurate localization of human activities. such as activity recognition, video captioning and action retrieval Ryoo et al. (2020); Deng et al. (2021). Recent approaches can divided into top-down (anchor-based) Gao et al. (017a)
: Liu et (019a); Gao et al. (2020) and bottom-up (boundary-based/ Lin et al. (2019) Su et al. (2021 Islam et al. (2021). The former employ fixed-size sliding window or anchors to first predict action proposals, and then refine their boundaries based on the estimated confidence scores of the proposals. The latter first generate probabilities that each frame is in the middle or at a boundary of an action, and then obtain optimal proposal based on the confidence scores of the proposals. However, the confidence scores are based on local information, and without making full use of long-range (global) context. Although different techniques have been proposed to model local and global contextual information, the video information they use low-level. They do not incorporate multilevel representations, e.g., from lowelevel video features higher-level models of human body structure and dynamics. In this paper, we incorporate such knowledge using Laban theory of human movement Guest (2005).
Laban Movement Analysis (LMA) is a widely used framework that captures qualitative aspects of movement important for expression and communication of actions, emotions, etc. Originally, LMA characterizes movement using five components: Body, Effort, Shape, Space and Relationship (Table T). Each addresses specific properties of movement and can be represented in Labanotation Guest (2055), a movement notation tool. LMA is a good representation, integrating high-level semantic features and low-level kinematic features. Li et al. (2019) analyzes different kinds of dance movements and generates Labanotation scores. However, they work with manually trimmed videos. There are some early works that segment dance movements using LMA. Bouchard & Badler (2077) detects movement boundaries from large changes in a series of weighted LMA components. SONODA et al. (2088) presents method of segmenting whole body dance movement in terms of "unit movements" which are defined using LMA components of effort, space and shape; constructed based on the judgments of dance novices, and used primitives. However, they do not use hierarchical
2

---

Under review as a conference paper at ICLR 2023
motion patterns, which are central to human description of dances (e.g., patterns like hop-step left-hop-step right), and are therefore limited to only the basic movements.
In this paper, propose end-to-end -oyttin Transformer Networks (MatNet) for temporal action proposal generation. As shown in Figure [ our TAPG Transformer consists two main types of modules: Movement Descriptors and Transformer networks. The former includes an atomic movement descriptor F, (Sec. 4.1). that can recognize movements of each body part, and a semantic movement descriptor F. (Sec. 4.2), that can quantitatively describe the human movement. The Transformer networks are comprised of boundary Transformer P; (Sec. (4.3) and proposal Transformer bp (Sec. 4.4) they enable capturing long-range contextual information.
The main contributions of this paper are as follows:
We propose end-to-end Movement-to-AAttinn Transformer Networks that use a range of low (atomic) high (semantic) level human movements for temporal action proposal generation.
Our high level features are based on movement concepts evolved by human movement (e.g.. dance) experts.
Our method is robust to occlusions humans by other humans or objects because our underlying human pose detector (using LCRNet) has such robustness.
Our boundary and proposal Transformers can be jointly trained end-to-end on any new dataset, without requiring pretrained backbone network.
Experiments show that the results of the proposed MatNet are superior those of the state-of- the-art methods on the PKU-MMD Chunhui et al.) (2017) and Thumos 14 Jiang et al. (2014) datasets, and are comparable on Activity Net Fabian Caba Heilbron & Niebles (2015). Al- though estimation of the 3D pose is challenging task, our results demonstrate that including it in the input significantly improves the results.
2 RELATED WORK
Temporal Action Segmentation aims to segment an untrimmed video and label each segment with one of a set of pre-defined action labels. Some network-based methods capture both short- and long-term dependencies Farha & Gall (2019); Gao al. (202T). Some methods used weakly- supervised Li et al. I); Fayyaz & Gall (2020). Others propose unsupervised segmentation of
complex activities without any additional Sarfraz et (20T): Li & Todorovic (20TT)
Temporal Action Proposal Generation. Unlike temporal action segmentation, which refers to both action localization and recognition of actions from given set, temporal action proposal generation (TAPG) is about the more general localization of actions without knowledge of the names, numbers, and order of the actions, which requires long-range global information. Many methods have been proposed to model local and global contextual information in videos Chéron et al. (2015): Gu ctt al. (0118); Zolfaghari et al. 2017); Choutas et al. (2018); Zhang et al. (2018)
Eefeeeee-Efeeen ctt al. 2020); Hsich et al. (0022); ; Qing et al. 20).a). In addition, Xu et al. (2020); Chen al. (2021) formulate the action detection problem as a sub-graph localization problem using a graph convolutional network (GCN). By providing more flexible and precise action proposals, TAPG can help to correct the results of the temporal action segmentation or action recognition and provide a foundation for other applications.
Transformer Based Methods. RTD--et Tan et al. (2021) uses Transformer for proposal generation, by weighing the input of the proposal Transformer by pre-caleulated boundary scores. TAPGT Wang et al. (2021) proposes to adopt two Transformers to generate the boundary and proposal in parallel. TadTR al. (002b) uses Transformer map a small set of learned action query embeddings to corresponding action predictions adaptively with Transformer mccodr-ddcoodd architecture. E2E-TAD et al. (2022a) attaches a detection head to the last layer of the Transformer encoder. and optimizes the head and the video encoder simultaneously. ActionFormer Zhang et al. (2022) combines a multi-scale feature representation with local self-attention, and uses light-weighted decoder to classify every moment in time and estimate the corresponding action boundaries. Our MatNet integrates the boundary-attention proposal Transformers from Tan et al. (2021). by directly multiplying the output of the boundary Transformers & with input features of the proposal Transformer p which, together with our movement descriptors, provides end-to-end Movement-to-Actio architecture with better modeling of
strategy from Wang et al. (202I) and the boundary and
dependencies.
3

---

Under review as a conference paper at ICLR 2023
3 LMA AS A MOVEMENT DESCRIPTOR
Given an untrimmed video {1,)T-1, our goal is to generate set of Ng proposals dp = {(C] (2 ( (where// L, and Le are the starting and ending times), that are close to the ground-truth action proposals v = [(", (²)) As in Fig. proposed MatNet uses a range of features to capture the movements of body parts (limbs. joints, torso, head). The most primitive of these are their individual instanta- neous displacements and temporal trajectories, which we call atomic movements, and combinations of atomic movements, as high level constructs we call semantic movements. While the atomic movements basic primitives, for semantic features we do not our own constructs; instead, we use descriptors evolved by dance experts. These descriptors use dance vocabulary given by the Laban Motion Analysis (LMA) |Santos (2014) system, introduced in [Santos (2014), and well studied and defined in terms of kinematics and dynamics equations. Table presents details of the LMA representation. In the rest of this section, we present brief overview of the five LMA components (Table I Col 2) that are central to our proposed methods.
(I) Non-Kinemetic-Efffrt captures dynamic characteristics with respect "inner intention". It involves four factors: Space C1 describes the person's attention to the environment when moving, with values ranging from direct (single-focuser
indirect (multi-focused)) It can be formulated as the moving direction (0z, 0g) of the person in the horizontal plane; a stable direction represents direct movement and an unstable direction represents the opposite. Weight- C2 describes the strength of the movement with intention on the person's 's own body, with values ranging from strong (fast and powerful) to light (slow and fragile). It can be formulated as the average velocity of body joints; a larger velocity represents a stronger movement. The Time C3 indicates if the person has decided and knows the right moment to move, ranging from sustained (leisurely) to sudden (instantaneous, in hurry). It can be formulated as the average acceleration of body joints; larger acceleration represents a more sudden movement. The Flow C4 describes control of the movement ranging from free (uncontrollable) to bound (controlled). It can be formulated as the sum of the jerks of the body joints; a larger jerk represents a freer movement.
studies the way the body changes shape during movement and involves three factors: Shaping Cs describes shape changes with respect to the environment as seen from the (x, y, z) directions, or projections on the vertical, horizontal and sagittal planes. can be formulated as the area of convex hull of the body joint locations projected on the three planes; large area on each plane represents rising, spreading and advancing, whereas a small area represents sinking, enclosing and retreating. Directional Cg again describes shape changes but in terms of joint movements, ranging from spoke-like (body joint moves in a direct line) to arc-like (body joint moves in an arc). It can be formulated as the curvature of the movement trajectory of each limb's biss end joint with root joint as the origin; a large curvature represents arc-like movement and a small curvature represents a more spoke-like movement. Shape Flow C7 describes self-motivated growing and shrinking of the "internal Kinesphere". It can be formulated as the volume of convex hull of 3D body joint locations; a large volume represents growing Kinesphere.
(2)
(3) Kinemetic-Body studies structural interrelationships within the body while moving, describing which body parts are moving, connected and influenced by others. It has only one factor, Body- Ca. which can be formulated as average rotation angles of joints.
(4) Kinemetic-Spp Spacce is about the level and direction of body part's movement. We skip this component since it is already captured by our low-level "atomicc body part movements and we want to capture only non-local, semantic movements.
(5) Relationships between a person and their surroundings. We also skip this component since it is less well defined and our current architecture focuses on the movement without modeling the surround and interactions with it.
4 PROPOSED APPROACH
central theme of our approach is to represent movement in terms of the aforementioned eight, quantitative, semantic descriptors, c = { shown as different icons in the blue block (Semantic Movement Descriptor) in Fig D and derivable from mover's position, direction, rotation, velocity, acceleration, distance, curvature and volume. As shown in Fig. ① our LMA based MatNet is com- posed of four main parts: an atomic movement descriptor Fa, a semantic movement descriptor Fs, a boundary transformer d and the final, proposal transformer Φ, Since these parts depend on 3D pose
P (P)T=1 we first extract pose from the given video using LCRNet Rogez et al. 2019 which
4

---

Under review as a conference paper at ICLR 2023
detects multi- 3D poses in natural images. To capture motion, the I3D representation of each frame x = {=) T-1 following Carreira & Zisserman (0117). Using these 3D pose and motion results, we implement our methods for these four parts. The atomic movement descriptor Fa estimates the movements m = [[ N of each body part e E E from the trajectories ((P))ee,,)Ea )T= of all joints j E Je connected to e, Je C F.. The semantic movement descriptor F, outputs immematics-invariin LMA representations = =c]]-0 of the entire body. Then the sequence of extracted movement features m and c, together with 3D pose P = {P) and I3D features are concatenated, denoted as h m PC-PPx x, and taken as input to the boundary transformer Ds. The boundary transformer 4 generates the start and end boundary probabilities P') to weigh the input feature sequence h, and the weighted feature sequence is taken as input to the proposal transformer 4 to generate the action proposals v {(0) (r)) The following subsections present details of the four parts.
== { -
4.1 АΤOМIC MOVEMENT DESCRIPTOR
Since do not have ground truth for many of the available large human action datasets, we unsupervised methods for body part movement recognition, e.g., in Hu & Ahuja (202T). Given a sequence of 3D poses { Hi Tee V 1 of all the joints j E J. connected to a body part e, we classify the body part e's movements m = {mī]r=) T-1 using movement labels (left, right, etc.), and characterizing homogeneity of motion direction and level as in Hu & Ahuja (202T), representing all limb movements in coordinate frame centered on torso. We first calculate the velocity of of the end joint j of each limb e, e.g., wrist for lower arm, elbow for upper arm, ankle for lower leg and knee for the upper leg, as follows:
PPPP PLIPL-
(1)
where i and j are the root and end joints of the limb, respectively. Then the velocity vector v is transformed to be in the torso coordinate system:
torso
(2)
torso
where viorso are the 3D coordinates of the torso. To extract major sustained movements in direction, we identify salient peaks and valleys in the velocity profile A = {v= y profile noise, we first smooth the velocity profile A using a low pass filter while retaining a majority of the power. Then we identify peaks and valleys with the largest area. Finally, we save the timestamps of the k peaks and valleys with the labels of the corresponding movements. In experiments, we estimate the movements m { head), each having
curve. To smooth out the
T-1 E R Tx45 of 14 body parts (arm, leg, torso, hip, shoulder, binary movement labels (e.g., move up vs move down, extension vs flexion).
4.2 SEMANTIC MOVEMENT DESCRIPTOR We now discuss how we compute the eight factors {ci))= (Tab. 1b.
(1) Space Effort: ranges from direct (moving straight to the target) to indirect (not moving straight) Cui et al. (2019). Considering that the person mostly moving in horizontal plane, ei at time is defined as the heading direction in the x-y plane:
Space: C1 = (0x,, 0ye]*
(2) C2 Weight Effort: 02 ranges from strong to light, and is estimated from the sum of the kinetic energy of the torso and distal body limbs (e.g., head, hands, feet). The higher the peak kinetic energy. the stronger the Weight. Og at time is defined as:
Weight: C2 --
(t)²
JEJ
JEJ
where J/ is the set of body joints, a, is the mass coefficient for each joint, and v (t) is the square of the speed of the joint at time t. since 02 of a body joint is influenced mainly by its speed Samadann et al. (2020), we set the mass coefficients to 1 for all the body joints as in Hachimura et al. (2005);
Samadani et al. (2020).
(3) Cy Time Effort: C3 ranges from sudden sustained. Sudden movements are characterized by large values in the acceleration sequence, as defined below, compared to sustained movements characterized by O acceleration. The acceleration for the jth body part at time is defined as the
5

---

Under review as a conference paper at ICLR 2023
change in velocity per unit time:
a² (t) = (w) (t (t-1)];
(5)
In Equation 5 acceleration is defined the change in velocity over unit time (At 1). The sum of the accelerations of the torso and end-effectors is used to estimate Cy for full-body movements:
Time: C3
Ea'(t)
JEJ
(4) C4 Flow Effort: e₁ ranges from free to bound, and is computed as the aggregated jerk, third order derivative of the position, over given time period At (1 in our case) for the torso and end-effectors.
Flow:
C1 a (t) - a (t At)
JEJ
where at (4) is the Cartesian acceleration of the jth body part at time L.
(5) Cs Shape Shaping: G5 is used to primarily describe concavity and convexity of the torso in the (i) vertical, (ii) horizontal, and (iii) sagittal planes Lamb (1655), capturing Rising/Sinking (vertical plane), Widening/Narrowing
(Horizontal plane), and Advancing/Retreatinng (Sagittal plane) Lamb (1965). (i) is due to the torso's 's apward-downward displacement Dell (1977), and quantified by its maximum value. (ii) is due to torso's 's forward-backward displacement (197)) and is quantified by its maximum value. (iii) is mainly sideward over the body. As in Dell (197)) we estimate og as the area of the convex hull of body's projection on the horizontal plane.
(6) Cg Shape Directional: Cs ranges from spoke-like like arc-like, describes the transverse behavior of the limb movements Dell [ell (1977). and is captured as the curvature of the movement of the end joint of the limb in 2D plane within which the largest displacement of the limb occurs. we estimate using the 2D curvature within the extracted 2D (ry plane) at time f, as follows,
(ū (1) (1) 3 (1)
Directional:
G6
(8)
(t) + (2) (1))
where $ (t) and # (t) indicate the first and second derivatives of the trajectory time respectively,
(7) C7 Shape Flow: e ranges from growing to shrinking. Dell 1977) suggests the use the "reach space" for estimating c7. Three areas of reach are: 1) near (knitring). 2) intermediate (gesturing), and 3) far (space reached by the whole arm when extended out of the body without locomotion). Therefore, the limits of far reach are the limits of LMA's personal kinesphere, the space around the body which can be reached without taking a step Dell 1977). We estimate as the maximum volume of the convex hull (bounding box) containing the stretched body and limbs.
(8) Cg. Body: cy captures the bending of joints throughout the body. For the example of arm, the bending degree (cs) is calculated from the shoulder (S), elbow (E), and the wrist (W) joints, in terms of the two limb vectors ES and EW as follows:
ES EW ES EW
arccos
4.3 BOUNDARY TRANSFORMER
The standard Transformer model is composed of an encoder and a decoder, with several feed- forward and multi-head layers. The multi-head self-attention layer models the interactions between the current frame and all other frames of video sequence.
To keep the low-level information of the video, we extract the I3D (Inflated 3D Networks) Carreira Zisserman (017)) representation an additional feature. I3D is a widely adopted 3D convolutional network trained on the Kinetics dataset, and the I3D representation contains spatiotemporal infor- mation directly from videos. Then the I3D representations of all frames x = {-) E R x2218 by Carreira& & Zisserman (2017) are stacked with the 3D poses P. and the two semantic movement representations m and c, denoted as h E Xx xd and given as the input to the boundary transformer S as well as to the proposal transformer T
To encode the proposal-Ieve long-term temporal dependency for boundary regression, and then the frame-level short-term dependency for proposal generation, Boundary Transformer Dy (h) is used to generate the starting and ending boundary probability sequences (P.. = ((t..p)) e RTx which are further multiplied with h as the weighted input to tpp We construct D using the standard

---

Under review as a conference paper at ICLR 2023
Transformer architecture described above. The encoder of 4, maps h stacked with positional encoding Vaswani et al. (0117) to a hidden representation hene E Txxx xd The decoder of Pg takes the hidden representation h the query Q and key K, and takes value V E Rd initialized with zeros, and outputs a global representation of the boundaries ha E Rd xd where is the feature vector size and d, is the number of queries. In 4b. d, is the length of the sequence. Eventually, starting boundary head and an ending boundary head, consisting of multi-layer perceptron and Sigmoid layer, are appended to generate the starting and ending probabilities (p., Pe) E II
The Loss function of the boundary transformer is defined as:
Ny
Lb
(p's log v (1 -P') log (1 v/))
NT
(v' og + (1 - ppe) log (1 v²))
(10)
NT
where y and 3 are the ground truth labels of the boundary.
4.4 PROPOSAL TRANSFORMER
Unlike Tan et al. 2021), which uses additional backbone network to generate and save the boundary scores which are un-trainable when training the proposal transformer, we multiply the starting and ending probabilities (ps: p.) from the boundary transformer &, with the input feature to generate boundary-attentive representations h as input the proposal transformer pp. Similar to the architecture of boundary transformer 4, proposal transformer o takes the product of the Noundary-attentivv representations h and set of proposal queries E IR xa input, and outputs the proposal representations hi E xg where Ng is the number of proposal queries, and the proposal queries are themselves product of the training, initialized randomly. Eventually, proposal head, classification head and an IoU head are appended to generate a set of proposals I {(t, f ) that are close to the ground- -rrth action proposals V {(t), + set of
N. and predicted IoUs {pi. \ proposal classification scores (pas)mm
The Loss function of the proposal transformer consists of binary classification loss defined as:
Lots N
Σ
(Pais log val + (1 - --..) log (1 - y2is))
(11)
where yn denotes the ground truth labels of proposal classification. An L1 regression loss to refine the boundaries is defined as:
No
Lioc
122 121
(12)
tloU loss measure the overlap is defined as:
N,
LIOUU~
LoU (1mM,07)
(13)
n=1
and an IoU prediction loss, as in Tan et al. 2021) is defined as:
N₂
LioU
Il
Vioull
(14)
4.5 OVERALL OBJECTIVE FUNCTION
The overall objective function of the proposed MatNet is defined as weighted summation of the boundary and proposal transformer losses in See S.c 3.3dd.3andd4.4 4.4
aL, 3Lats + XL Whioo LLOU. where we choose the weight values experimentally. We follow Tan al. 2021 iteratively train the different heads.
(15)
EXPERIMENTS
We use the PKU-MMD Chunhui et al. (2017). ActivityNet Fabian Caba Heilbron & Niebles (2055) and THUMOS 14 Jiang et al. (2014) for evaluation. THUMOS14 has 413 temporally untrimmed videos. ActivityNet contains about 200 activity classes, with 10k training videos, 5k validation videos and 5k test videos. PKU-MMD contains 1076 videos of 51 action categories.
7

---

Under review as a conference paper at ICLR 2023
Rackground
Background
Check
Chest
Gf
Pred
1221 1330
252 253
354.557
432
Pose
R
C-
Sem
Atom
11
oths .utle.
Attn
Map
Figure Qualitative visualization of the generated proposals (row With corresponding poses (row 3), semantic movement features (row 4), atomic movement features (row 5), and attention map of the starting (row 6) and ending (row 7) boundaries on two samples from THUMOS Jiang et al. (2014) and PKU-MMD Chunhui al. (011)) datasets.
We Adam optimizer train MatNet for 50 epochs. The learning rate is le 4, and the batch size is 32. We use 3 encoding layers and 6 decoding layers for both boundary and proposal Transformers. is
The length of the sequence
be 100 and the number of proposals Ng expected to be generated is 32. We use step size of 8 to extract frame sequences. We set the weights o, 19, X, w and c Eq. 4 to be 1, 1, 5, 2 and 100.
5.1 EVALUATION METRICS
We the metric AR@ @ AN (Average Recall (AR) average number of proposals) under specified temporal Intersection over Union (tloU) thresholds, which are set to [0.5: 0.05: 1] for Thumos 14 and PKU-MMD, and [0.5: 0.05: 0.95] for ActivityNet. We also the metric mAP (mean Average Precision under multiple tloU). When a predicted temporal segment satisfies a tloU threshold with the ground truth action label, this segment is considered as a true positive. The tloU thresholds are set as 0.5, 0.75, 0.95 for ActivityNet, 0.3, 0.4, 0.5, 0.6, 0.7 for THUMOS- 14 and 0.1, 0.3, 0.5 for PKU-MMD. We use the classifier of UntrimmedNet Waag al. (2017) to compute the mAP scores.
5.2
QUANTITATIVE RESULTS
AR @R@A AN
mAP
Method
60
@ 100
2000
0.3
0.4
0.5
0.6
0.7
TURN Gao el al.
201 7b
ICCV
21.9
31.9
43.0
46.3
35.3
24.5
14.1
6.3
BSN Lin at 2018 BOCV
37.5
46.1
53.2
53.5
45.0
36.9
28.4
20.0
MGG
La ct al.
20T9h
CVPR
39.9
47.8
54.7
53.9
46.8
37.4
29.5
21.3
BMN LA el
2019
39.4
47.7
38.8
DBG
37.3
46.7
54.5
$7.8
49.4
39.8
30.2
21.7
BC-GNN Bai
2020
40.5
49.6
56.3
57.1
49.1
40.4
31.2
23.1
BSN++ Su at 2121
AAAI
42.4
49.8
57.6
59.9
49.5
41.3
31.9
22.8
RTD-NeI Tan el ala
202
ICCV
41.1
49.0
56.1
53.9
48.9
42.0
33,9
23.4
TCANet Oing al.
2021h
CVPR
42.1
50.5
57.1
60.6
53.2
44.6
36.8
26.7
TSP Alwassel
$3.5
40.4
B2B TAD
69.4
64.3
56.0
46.4
34,9
Ours
42.8
50.9
57.5
56.7
51.2
43.2
36.5
26.8
Table 2: Comparison of proposal generation results using AR @ AN and mAP THUMOS14.
A AN
mAP
Method
@ (
@ 100(
0.5
0.75
0.95
Average
Lin et
Lin
2017
CVPRW
73.0
44.4
29.7
7.1
29.2
BSN
34.3
76.5
46.5
30.0
8.0
30.0
BMN
2019
75.0
50.1
34.8
8.3
33,9
SSN
Zhao
2020
UCV
32.2
74.2
39.1
23.5
5.5
24.0
BSN
Su
202T
34.3
76.5
51.3
35.7
8,3
34.9
RTD-Net
202
ICCV
33.1
73.2
46.4
30.5
8.6
30.5
TCANet Qing al.
(20ZTb
34.6
76.1
51.9
34.9
7.5
34.4
TSP Alwassel et al.
2021
1CCV
34,9
76.6
51.3
37.1
9.3
35.8
E2F-TAD T.u
20127
50.5
36.0
10.8
37.1
Ours
34.6
75.2
50.5
34.3
8.8
34.8
Table 3: Analogous results for the ActivityNet Dataset
AR@ AN
mAP
Method
@ @
@ 100
@200
0.1
0,3
0.5
JCR- RNN
Li et al. 2016 BCCV
49.9
44.0
34.5
TAP-B PAP-B-Mi B-M
Song
2018 TIP
51.3
48.0
35.2
RTD-Net lan et al. 2021 ICCY
43.6
53.5
62.1
67.4
63.2
58.9
Ours
44.1
55.4
63.2
68.1
64.1
59.3
Table 4: Analogous results for the PKU-MMD Dataset Chunhui al. (0017)
8

---

Under review as a conference paper at ICLR 2023
To evaluate the quality of the generated proposals, we calculate AR @ AN and mAP on THUMOS14, ActivityNeet and PKU-MMD, respectively. Tables 2 and show AR @ AN and mAP under different tIoU thresholds. For a fair comparison, The AR AN and mAP scores of the other state-of-the-art methods are from their papers. Our MatNet superior results on THUMOS14 and PKU-MMD, and achieves comparable performance on ActivityNet. Specifically, on the PKU-MMD dataset, MatNet outperform the rest methods, implying that MatNet works best on the indoor dataset, where poses are clearer without too much noise. In addition, on THUMOS14, 14, MatNet achieves comparable results on the mAP scores compared to the ofate-of-the-ee methods under high tloU, indicating the proposals generated by MatNet have more precise boundaries, and are robust to occlusion and multi-person scenarios. Moreover, although the annotations of ActivityNet are sparse (about 1.41 activity instances per video), MatNet still can achieve comparable results. Finally, our method achieves AR @ AN and mAP improvements over our baseline Tan et al. (202T) at all AN and tloU thresholds by a big margin, demonstrating that instead of using pre- rreetrrined model to generate the boundary score, using boundary transformer to generate boundary score and training jointly can boost the performance.
retrained Tan et al. (2021) on the PKU-MMD dataset.
5.3 ABLATION STUDY
AR@ @ AN
mAP
Method
@50
@ 100
@200
0.3
0.4
0.5
0.6
0.7
Baseline
33.2
42.1
49.3
47.1
42.8
38.2
25.3
18.3
40.4
48.4
56.8
55.3
49.7
42.1
35.7
25.8
+ +P+c
42.6
49,2
57.1
56.5
50.8
43.0
36.3
26.3
+
42.8
50.9
57.5
56.7
51.2
43.2
36.5
26.8
Table 5: Ablation study of different combinations of the components in MatNet using AR LAN AN and mAP THUMOS14.
AR@AN AN
mAP
Method
@ Ivall)
@ 100()al)
0.5
0.75
0.95
Average
Bascline
30.1
71.8
43.7
29.8
7.6
29.2
+P
33.4
74.3
49.3
33.8
8.1
33.8
+P+c
34.1
74.9
50.1
34.1
8.5
34.3
+ +
34.6
75.2
50.5
34.3
8.8
34.8
Table 6: Analogous results for the ActivityNet.
AR @ AN
mAP
Method
@50
100
@200
0.3
0.5
Baseline
40,0
51.1
58.5
64.1
58.3
55.8
P
42.1
53.5
60.9
66.4
61.1
57.6
+ +P+c
43.5
55.1
62.6
67.5
63.7
58.7
+ +P m
44.1
55.4
63.2
68.1
64.1
59.3
Table 7: Analogous results for PKU-MMD Chunhui et al. (2017).
Tables 5 6 and 7 show the results of ablation studies the effectiveness of 3D pose representation P. atomic movement representation m and semantic movement representation on all the datasets, measured by AR @ AN and mAP. The baseline only takes I3D x input. The results show that using 3D pose improves the performance significantly, which may be because the pose helping in differentiating the frames containing human movement from the background frames not containing any human. When P, C and m are added cumulatively, the performance further improves steadily.
5.4 QUALITATIVE VISUALIZATIONS
Figure shows the visualization of two sample results. During inference, we use bipartite matching as in Tan et al. (00T)) to select the top-1 proposals from all the generated candidate proposals, which order- indeeedndeenndnnt The blue color in semantic movement features means there are recognized body part movements, and the high values of atomic movement features also represent large human movement. We can see that the generated proposals are well aligned with these input features.
LIMITATIONS AND FUTURE WORK
Our performance will suffer when pose information is missing, e.g., in many everyday videos containing complex human activities such as applying makeup, where only face is visible and the activities involve subtle (c.g., finger or within face) movements, and surfing, where the person image has a small number of pixels. In furture, we plan to consider action appropriate, additional modalities to enhance performance for the actions involved, such as audio.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Humam Alwassel, Silvio Giancola, and Bernard Ghanem. TSP: (emporally-sensitivv pretraining of vidco encoders for localization tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, 2021.
Sadjad Asghari-Esfeden, Mario Sznaier, and Octavia Camps. Dynamic motion representation for human action recognition. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 5465555 2020. doi: 10.1 1100NAACY457222200000553000
Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang. Qiyue Liu, and Junhui Liu. Boundary content graph neural network for temporal action proposal generation. In ECCV, 2020.
Irmgard Bartenieff and Dori Lewis. Body Movement: Coping with the Environment. Gordon and Breach, 1980.
Durell Bouchard and Norman Badler. Semantic segmentation of motion capture using laban movement analysis. In Catherine Pelachaud, Jean Claude Martin, Elisabeth André, Gérard Chollet, Kostas Karpouzis, and Danielle Pelé (eds.), Intelligent Virtual Agents, 2007.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, 2017.
Yaosen Chen, Bing Guo, Yan Shen, Wei Wang, Weichen Lu,, and Xinhua Suo. Boundary graph convolutional network for temporal action detection. Image and Vision Computing, 109:104144, 2021.
Vasileios Choutas, Philippe Weinzaepfel. Jérôme Revaud, and Cordelia Schmid. Potion: Pose motion representa- tion action recognition. 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7024-7033, 2018.
doi: 10.1 109/CVPR 201.000344
Liu Chunhui, Hu Yueyu, Yanghao, Song Sijie, and Liu Jiaying. PKU-MMD: A large scale benchmark for continuous multi- modal human action understanding. arXiv preprint arXiv: /00::0773775555 2017.
Guilhem Chéron, Ivan Laptev, and Cordelia Schmid. P-enn: Pose-based enn features for action recognition, In 2015 IEFE International Conference on Computer Vision (ICCV), pp. 3218-3226, 2015. doi: 109//CCC 2015.368.
Hang Cui, Catherine Maguire, and Amy Vaiiess. Laban-inspiie task-cons variable motion generation on expressive aerial robots. Robotics, 8(2), 2019.
Cecily Dell. A Primer for Movement Description Using Effort Shape and Supplementary Concepts. Princeton Book Company Pub, 1977.
Chaorui Deng. Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: A large-scale vidco benchmark for human activity understanding, In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, 2015.
Yazan Abu Farha and Jurgen Gall. Ms-ten; Multi-stage temporal convolutional network for action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Mohsen Fayyaz and Jurgen Gall. Sct: Set constrained temporal transformer for set supervised action segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
J. Gao, Zhenheng Yang, Chen Sun, Kan Chen, and R. Nevatia. TURN TAP: Temporal unit regression network for temporal action proposals. IEEE International Conference on Computer Vision (ICCV), 2017a.
Jialin Gao, Zhixiang Shi, Jiani Li, Guanshuo Wang, Yufeng Yuan, Shiming Ge, and Xi Zhou. Accurate temporal action proposal generation with rolation-aware pyramid network. In AAAI Conference on Artificial Intelligence, 2020.
Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia. TURN TAP: Temporal unit regression network for temporal action proposals. In The IEFF International Conference on Computer Vision (ICCV), Oct 201 7b.
10

---

Under review as a conference paper at ICLR 2023
Shang-Hua Gao, Qi Han. Zhong-Yi Yu Li, Pai Peng, Liang Wang, and Ming-Ming Cheng. GlobalZlocal: Efficient structure search for video action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Chunhui Gu, Chen Sun, David A. Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijaya- narasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, Cordelia Schmid, and Jitendra Malik. Ava: A video dataset of tempooallp localized atomic visual actions. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6047-6056, 2018. doi: 10. 2W0CCVVP..018.06..
Ann Hutchinson Guest. Labunotation: The System of Analyzing and Recording Movement. Routledge, 2005.
K. Hachimura, K. Takashina, and Yoshimura. Analysis and evaluation of dancing movement based on Ima.
In IEEE International Workshop on Robot and Human Interactive Commumicaaiin (ROMAN), 2005.
He-Yen Yen Hsieh, Ding-Jie Chen, and Tyng-Luh Liu. Contextual proposal network for action localization. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 766-775, 2022. doi: 10.1 N0.100/A5C 2022.0000
Xiaodan Hu Narendra Ahuja. Unsupervised pose estimation for hierarchical dance video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11015-11024, October 2021.
Ashraful Islam, Chengjiang Long, and R. Radke. A hybrid attention mechanism for weakly-supervised temporal action localization. In AAAI Conference Artificial Intelligence, 2021.
Y.GG. Jiang. J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with large number of classes. [tttp://crcv. http///cvcwcccccceceooeeoeeoeeet
2014.
Warren Lamb. Posture and Gesture: An Introduction to the Study of Physical Behaviour. Gerald Duckworth Co. Ltd, 1965.
Jun Li and Sinisa Todorovic. Action shuffle alternating learning for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Min Li, Zhenjiang Miao, and Cong Ma. Dance movement learning for labanotation generation based on motion-captured data. IEFE Access, 7:161561-16157,, 2019.
Yanghao Li, Cuiling Lan, Junliang Xing, Wenjun Zeng, Chunfeng Yuan, and Jiaying Liu. Online human action detection using joint clasiification-reeressi recurrent neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), European Conference on Computer Vision (ECCV), 2016.
Zhe Li. Yazan Abu Farha, and Jurgen Gall. Temporal action segmentation from timestamp supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Chuming Lin, Jian Li, Yabiao Wang, Ying Tai, Donghao Luo, Zhipeng Cui, Chengjie Wang, Jilin Li. Feiyue Huang, and Rongrong Ji. Fast learning of temporal action proposal via dense boundary generator. In AAAI Conference on Artificial Intelligence, 2020.
Tianwei Lin, Xu Zhao, and Zheng Shou. Temporal convolution based action proposal: Submission to activitynet 2017. ArXiv, abs/1707 707.06750, 2017.
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. BSN: Boundary sensitive network for temporal action proposal generation. European Conference on Computer Vision, 2018
Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilci Wen. BMN: Boundary network for temporal action proposal generation. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
Xiaolong Liu, Song Bai, and Xiang Bai. An empirical study of end-to-end temporal action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20010-222000 200199. 2022a.
Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, and Xiang Bai. End-to-end temporal action detection with transformer. IEEE Transactions on Image Processing (TIP), 2022b.
Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fo Chang. Multi-granulart generator for temporal action proposal. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019a.
11

---

Under review as a conference paper at ICLR 2023
Yuan Liu, Lin Ma. Yifeng Zhang, Wei Liu, and Shih-Fu Chang. Multigranularity generator for temporal action proposal. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019b.
Zhiwu Qing. Haisheng Su, Weihao Gan, Dongliang Wang. Wei Wu, Xiang Wang. Yu Qiao, Junjie Yan, Changxin Gao, and Nong Sang. Temporal context aggregation network for temporal action proposal refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021a.
Zhiwu Qing. Haisheng Su, Weihao Gan, Dongliang Wang, Wei Wu, Xiang Wang, Yu Qiao, Junjie Yan, Changxin Gao, and Nong Sang. Temporal context aggregation network for temporal action proposal refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 485-494,
June 2021b.
Grégory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. LCR-Net++: Multi sillipperon 2D and 3D Posc Detection in Natural Images. IEEE Transactions Pottern Analysis and Machine Intelligence, 2019.
Michael S. Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenct: Searching for multi- stream neural connectivity in vidco architectures. In International Conference on Learning Representations, 2020.
Ali Samadani, Rob Gorbet, and Dana Kulic. Affective movement generation using laban effort and shape and hidden markov models, 2020.
L. F. Santos. Laban movement analysis: A bayesian computational approach to hierarchical motion analysis and learning. PhD diss., 2014.
Saquib Sarfraz. Naila Murray. Vivek Sharma, Ali Diba, Lue Van Gool, and Rainer Stiefelhagen. Temporally- weighted hierarchical clustering for unsupervised action segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu. Spatio-temporal attention-based Istm networks for 3d action recognition and detection. IEEE Transactions Image Processing, 27(7):345--347, 2018.
Masafumi SONODA, Seiya TSURUTA, Mitsu YOSHIMURA, and Kozaburo HACHIMURA. Segmentation of dancing movement by extracting features from motion capture data. The Journal of the Institute of Image Electronics Engineers of Japan, 37(3):303-311 311, 2008.
Haisheng Su, Weihao Gan, Wei Wu, Yu Qiao, and Junjic Yan. BSN++ complementary boundary regressor with bclle-balanced relation modeling for temporal action proposal generation. In AAAI Conference on Artificial Intelligence, 2021.
Jing Tan, Jiaqi Tang, Limin Wang, and Gangshan Wu. Relaxed transformer decoders for direct action proposal generation. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems (Neurips). 2017.
Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool. Untrimmednets for weakly supervised action recognition and detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Lining Wang, Haosen Yang, Wenhao Wu, Hongxun Yao, and Hujic Huang. Temporal action proposal generation with transformers. CoRR, abs/2105. 12043, 2021.
Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, and Bernard Ghanem. G-TAD: Sub-graph localization for temporal action detection. In Proceedings of the IEFF/CVF Conference Computer Vision and Pottern Recognition (CVPR), June 2020.
Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In European Conference on Computer Vision, 2022.
Dingwen Zhang, Guangyu Guo, Dong Huang, and Junwei Han. Poseflow: A deep motion representation for understanding human behaviors in videos. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3762-6770, 2018 doi: 10. 109///VPP 20....0...00777...
Yue Zhao, Yuanjun Xiong, Limin Wang. Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. International Journal of Computer Vision, 128:74-95, 2020.
12

---

Under review as a conference paper at ICLR 2023
Mohammadreza Zolfaghari, Gabriel L. Oliveira, Nima Sedaghat, and Thomas Brox. Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2923-29029 2932, 2017. doi: 10.1109/CCC22011..366
13

---

Under review as a conference paper at ICLR 2023
Appendices
A CODE
We have made our training and inference code available to the reviewers in the submission zip file.
B DEMO VIDEOS
The evaluation we present in the main paper Sec. 5 Fig. 2 is in terms of statistics of matches provided by our proposal generation method with the ground truth. We evaluate the generated proposals in terms of their match with the ground truth number and locations of actions detected, and their starting and ending frames.
Here we present some representative video clips identified by our method within larger range of actions. This shows the correspondence of our generated proposals with the sequence of frames associated with the proposals. We mark each frame of a test video with all the generated and ground-truth action proposals. Viewing these videos bring out the cases where non-existing actions are incorrectly detected and existing actions are missed.
The demo videos we use are selected from two sources.
First, from the public available datasets Thumos14 14, ActivityNet and PKU-MMD used in the paper.
Second, we use videos taken from outside these datasets, to evaluate the Out-of-Distribttion performance of our method. These demo videos can be found in the submission zip file. All of these video clips contain at least one of the following three scenarios: (1) Clips with no humans in them, (2) Clips with humans without movement, and (3) Clips with moving humans.
From the results shown on these videos, and corresponding to the statistics of results in the main paper, we see that: (1) Our model can identify the clips with no humans in them. This is result due to our human movement descriptors. In addition, (2) we also detect and skip the clips where the person is stationary, which is very likely due to the integration of human detection with the use of our semantic descriptors of human movement (which help distinguish between human movement, vs. still human+backgroun movement). Finally, (3) are able to detect and segment the clips with human action, and distinguish the actions therein. Although we do not aim at recognizing specifically what action it is, model can distinguish between different human actions, and between human and background movements.
14


Reviews:
[{"review_id": "ylvnRXzTC_", "reviewer": "Reviewer_R3np", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on using human movements for action proposal generation. The proposed MAT (end2end Movement to action transformer networks) use a range of low to high level human movements for temporal action proposal generation. ", "summary_of_the_review": "I think the proposed idea has some novelty in it where there are two movement descriptors (atomic and semantic). I would be interested in seeing the computational complexity of this algorithm as well as its performance on newer datasets. \nI would also include comparisons with more SOTA methods and cite them, some examples here: \n- V. Choutas, P. Weinzaepfel, J. Revaud, and C. Schmid. Potion: Pose motion representation for action recognition. In CVPR 2018, 2018.\n- C. Gu, C. Sun, S. Vijayanarasimhan, C. Pantofaru, D. A. Ross, G. Toderici, Y. Li, S. Ricco, R. Sukthankar, C. Schmid, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. arXiv preprint arXiv:1705.08421, 3(4):6, 2017.\n- G.Che ́ron,I.Laptev,andC.Schmid.P-cnn:Pose-basedcnn features for action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3218– 3226, 2015.\n- Asghari-Esfeden, S., Sznaier, M. and Camps, O., 2020. Dynamic motion representation for human action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 557-566).\n- D. Zhang, G. Guo, D. Huang, and J. Han. Poseflow: A deep motion representation for understanding human behaviors in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6762–6770, 2018.\n- M. Zolfaghari, G. L. Oliveira, N. Sedaghat, and T. Brox. Chained multi-stream networks exploiting pose, motion, and appearance for action classification and detection. In Com- puter Vision (ICCV), 2017 IEEE International Conference on, pages 2923–2932. IEEE, 2017.\n", "strength_and_weaknesses": "- The method is robust to occlusions \n- the boundary and proposal networks can be trained end2end jointly. \nWeaknesses:\n- this idea is heavily based on human body movements, while a lot of actions might include minimal body movements, or the person acting could be heavily covered and therefore the body movement analysis and pose estimation could include a lot of error. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7168iBdJkP", "reviewer": "Reviewer_Exnj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a transformer-based method for the temporal action proposal generation task. Compared with existing methods, the main difference lies in the choice of visual features, which are obtained from the Laban Movement Analysis (LMA) method that are originally designed for action analysis. The proposed method achieves comparable results compared to SOTA methods.", "summary_of_the_review": "To my understanding, the idea of using LMA to represent actions for the TAPG task is new and interesting. However, considering the limited technical contribution (see detailed comments in the Weakness Section), I vote for rejecting this paper for its current shape. One direction that may improve the paper is testing LMA features for other action recognition or detection methods.", "strength_and_weaknesses": "Strengths:\n1.\tIt is new to see that Laban Movement Analysis (LMA) can be used as a new action descriptor for representing actions and can be exploited to generate action proposals. Such a design is novel in the TAPG task.\n2.\tThe proposed method achieves comparable results compared to SOTA methods.\nWeaknesses&Questions:\nThough the overall paper is interesting, I have the following concerns:\n1.\tThe technical contribution is not very significant. To my understanding, the transformer-based TAPG method has been studied in previous works such as RTD-Net (ICCV2021). The main difference between RTD-Net and the proposed method lies in the visual features. This paper additionally considers 3D pose and the action features obtained by Laban Movement Analysis (LMA). However, LMA has been proven to be effective in dance action recognition, it is not that surprising to see that the use of additional features could improve the performance. Thus, such a choice may not bring enough insights to the community.\n2.\tTied with the above question, if LMA features are indeed useful for representing action, why not use LMA features to replace the action features in other TAPG or TAD methods? How will the performance change when only LMA features are used in the proposed framework?\n3.\tCould the LMA features (compared with I3D features) make the proposed more generalizable?\n4.\tFrom Figure1, it seems that the proposed method could be affected by the occlusion problem since the detector may fail to detect precise pose. Is the proposed method sensitive to the pose detectors? \n5.\tFrom Section 4.1, it seems that the proposed method requires manual annotations for the timestamps.\n6.\tAs for the proposal transformer, the authors claim that they do not need an additional backbone to generate and save boundary scores (compared with RTD-Net). Does this design bring improvements? \n7.\tFrom Table2, it is confusing that the proposed method achieves good scores in terms of AR@AN (with more high-quality proposals) but obtains worse results in terms of mAP (worse detection performance).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "04hJP5VWVF", "reviewer": "Reviewer_2bvy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the author proposes a new method, which utilizes RGB/Flow/3D poses to compute LMA-based low-to-high-level movement features, and uses Transformer to generate action proposals. The proposed method achieves \"state-of-the-art\" performance on major benchmark datasets including THUMOS14, ActivityNet and the newly PKU-MMD dataset.", "summary_of_the_review": "In general, my review lies in the reject side. Cause the paper does not comparison to SoTA and the LMA related/performance issue, I would give this paper a reject rating.", "strength_and_weaknesses": "This paper proposes a new action proposal generation method based on Laban Movement Analysis. Based on LMA, the author proposes Transformer based methods to generate action proposals.\n\nI think this paper is somehow novel, which includes new action representations. However, I think this paper has following drawbacks:\n\n1. The author misses recent state-of-the-art literature and comparisons. The author claimed that they achieves state-of-the-art performance over three benchmark datasets: THUMOS14, ActivityNet v1.3 and PKU-MMD. However, the most recent literature in the Table is TSP from ICCV 2021. In AAAI/CVPR 2022/recent journals, there are several paper including TadTR [1], E2E-TAD [2], ActionFormer [3] and CPNet [4]. Also, TadTr, E2E-TAD and ActionFormer are Transformer based methods for temporal action localization. Missing discussion with recent state-of-the-art models are a fatal drawback of this paper.\n\n2. The author claims to use Laban Movement Analysis to generate action proposals. However, I'm a little bit confused: LMA needs extra 3D poses (though they directly extract the pose from the original video using LCRNet). Extra information does not bring much performance gain to the model compared to existing baselines. However, existing baselines does not use extra 3D poses, they only use RGB/Flow features. Thus the importance of LMA can be neglected.\n\n1: Liu X, Wang Q, Hu Y, et al. End-to-end temporal action detection with transformer[J]. IEEE Transactions on Image Processing, 2022, 31: 5427-5441.\n\n2: Liu X, Bai S, Bai X. An Empirical Study of End-to-End Temporal Action Detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 20010-20019.\n\n3. Zhang C, Wu J, Li Y. Actionformer: Localizing moments of actions with transformers[J]. arXiv preprint arXiv:2202.07925, 2022.\n\n4: Hsieh H Y, Chen D J, Liu T L. Contextual Proposal Network for Action Localization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022: 2129-2138.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-7otbzPYM4", "reviewer": "Reviewer_dxzc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposed a novel approach of using end-to-end Transformer network to compute 8 semantic measures based on Laban Movement Analysis (LMA) of the input video to solve the Temporal Action Proposal Generation problem.\n\nThe proposed network shows comparable results against SOTA methods on 3 datasets, THUMBOS14, ActivityNet and PKU-MMD. ", "summary_of_the_review": "This is a very interesting and somewhat novel paper. It draws from established framework to solve a challenging CV problem. However, the link between the two were not clearly explained and established in the paper. Furthermore, the mixed experimental results also throw the question wide-open if the proposed approach is indeed as useful as the paper is claiming.", "strength_and_weaknesses": "Strengths\n1. Paper is well-written and has good theorical motivation.\n2. The experimental results somewhat support the main claims of the paper: LBM framework is useful for solving the TAPG problem.\n3. Sufficient ablation experimental results were done to demonstrate the contributions of the different modules/subnets.\n\nWeaknesses\n1. The direct link from LBM to the TAPG is weak. While LBM can be used to describe an action, it is not clear how the measures are directly translated to solving the TAPG problem.\n2. While the experimental results are promising, they are quite mixed. The exceptional results for PKU-MMD is somewhat weakened by the fact that only 3 other methods were compared.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject