Simple Yet Effective Action Recognition for Autonomous Driving
Weijiang Xiong, Lorenzo Bertoni, Taylor Mordan∗ and Alexandre Alahi
École Polytechnique Fédérale de Lausanne (EPFL)
Visual Intelligence for Transportation (VITA)
CH-1015 Lausanne, Switzerland
firstname.lastname@epfl.ch, ∗ Corresponding author
Extended abstract submitted for presentation at the 11th Triennial Symposium on
Transportation Analysis conference (TRISTAN XI)
June 19-25, 2022, Mauritius Island

March 31, 2022
Keywords: Action Recognition, Autonomous Driving, Deep Learning, Human Pose Estimation

1INTRODUCTION

Self-driving cars and delivery robots are set to shape the future of transportation, but they still
have to learn how to co-exist with humans in close proximity. Autonomous systems need to detectpedestrians and understand the meaning of their actions before making appropriate decisions inresponse. Action recognition is therefore an essential task for transportation applications, andyet very challenging, as there is no control over the distances of pedestrians or the real-world
variations like lighting, weather, and occlusions.
In this paper, we focus on the action recognition task in the context of transportation applications
and deal with real-world variations and challenging scenarios by representing humans throughtheir 2D poses. Human poses are an effective intermediate representation for 2D and 3D humanperception tasks. Representing human postures as sparse sets of keypoints allows focusing on
essential details while providing invariance to many factors, including background scenes, lighting, textures, and clothes. Methods leveraging keypoints have obtained state-of-the-art results
and excellent generalization properties on 3D pedestrian localization (Bertoni et al., 2019) and3D pose estimation (Martinez et al., 2017) tasks. However, keypoints’ greatest strength is also
their main weakness, as such a low-dimensional representation risks neglecting other essential
elements in a scene. We propose a simple approach using keypoints as intermediate representations and aim to shed light on which tasks keypoints are effective representations for. We firstvalidate our approach on the TCG dataset (Wiederer et al., 2020), showing that a simple method
can achieve better results than temporal baselines using LSTMs, and comparable results withcomplex attention-based graph convolutional networks. Then, we compare our approach on theaction recognition task on TITAN (Malla et al., 2020), a new dataset for autonomous driving.
We show that on atomic actions, such as walking or standing, our keypoint-based approach outperforms an image-based method, validating the effectiveness of human poses as intermediate
representations for action recognition tasks in transportation applications. We publicly share
our source code to facilitate academic communication.1

2PROPOSED APPROACH

We propose a simple method (Figure 1) to recognize human activities from images, leveraging
human poses as intermediate representations. This model is inspired by MonoLoco (Bertoni
1Github repo: https://github.com/vita-epfl/pose-action-recognition

TRISTAN XI Symposium

Original abstract submittal

Weijiang Xiong, Lorenzo Bertoni, Taylor Mordan and Alexandre Alahi

2Figure 1 – Model architecture inspired by MonoLoco. The input is a set of 2D joints extracted froma raw image and the output is the estimated action of a pedestrian. We use three different heads
for image-based and video-based action recognition, and for estimating simultaneous actions.
Table 1 – Action recognition results on TCG (Wiederer et al., 2020) test set.
Cross-subject

MethodCross-view

Accuracy (%)

Jaccard (%)

F1 (%)

Accuracy (%)

Jaccard (%)

F1 (%)

RNN
GRU
LSTM
Att-LSTM
Bi-GRU
Bi-LSTM
TCN
GCN

82.81
84.44
83.23
85.67
86.80
87.24
83.44
65.42

57.40
58.16
56.32
50.70
57.25
67.00
62.06
38.55

69.45
70.45
68.59
61.87
68.95
78.48
74.23
50.73

80.94
83.47
79.58
85.30
87.37
86.66
82.66
62.40

57.21
56.25
52.0259.87
55.55
65.95
63.97
35.05

69.98
68.59
64.62
71.20
67.68
77.14
75.95
48.51

AAGCN
Pham et al.

91.13
91.09

-

85.81
86.26

90.22
90.64

-

85.21
85.52

Ours (single-frame)
Ours (temporal)

85.03
87.31

63.72
69.15

76.91
81.15

86.29
87.74

68.76
70.11

80.81
81.89

et al., 2019) and Figure 1 presents the general network architecture, consisting of two stages.
We first extract 17 body keypoints for each person in an image with OpenPifPaf (Kreiss et al.,2021), and transform them into 1 center point and 17 relative coordinates (18 in total, shownin Figure 1). Then, the model encodes the keypoints with a feedforward network, and predicts
the corresponding action from the encoded representation. We also extend our method (i) toestimate actions from videos and (ii) to predict simultaneous groups of actions (e.g., a person canwalk while being on the phone). For (i), we add a simple LSTM to process a temporal sequenceof poses before the final linear layer, and for (ii), we use a multitask approach, where multiple
parallel heads (instead of just one) process a shared representation to yield multiple predictions.

3EXPERIMENTS

3.1

Experiments on TCG

The TCG dataset (Wiederer et al., 2020) collects accurate 3D body keypoints for recognition of
traffic control gestures, which we use to validate the design of our single-frame model directly fromerror-free poses. Following the cross-subject and cross-view evaluation protocols in TCG, Table 1
compares the performances of our single-frame and temporal models with eight simple baseline
methods as well as two more complex attention-based graph convolutional networks (Pham et al.,2021). Our temporal model outperforms the simple baseline models. Specifically, it performs
better than the LSTM baseline, which directly predicts actions from raw keypoint coordinates.
TRISTAN XI Symposium

Original abstract submittal

Weijiang Xiong, Lorenzo Bertoni, Taylor Mordan and Alexandre Alahi

3

Table 2 – Action recognition results on TITAN (Malla et al., 2020) test set.
I3D

3D ResNet

Ours (multitask)

Action group

Accuracy (%)

Accuracy (%)

Accuracy (%)

mAP (%)

atomicsimple
complexcommunicativetransportive

92.19
53.18
98.81
86.49
90.80

75.5231.73
98.80
86.48
90.81

80.01
47.97
97.80
83.69
89.80

26.80
20.27
15.50
29.55
28.30

overall

84.29

76.67

79.85

24.08

Table 3 – Action recognition results with selected actions on TITAN (Malla et al., 2020) test set.
Average Precision (AP %) ↑ [Detection Recall ↑]Method

InputsWalking[75.4%]

Standing[65.4%]

Sitting[62.1%]

Bending[71.8%]

Biking[82.4%]

Average[73.7%]

Ours (multitask)

Keypoints

90.16

40.67

43.78

41.12

57.70

48.14

ResNet50 (He et al.)Ours (single-frame)
Ours (temporal)

CropsKeypoints
Keypoints

92.85
96.87
97.83

42.07
64.55
73.02

5.18
81.22
65.78

8.44
64.59
47.31

56.00
88.30
84.98

40.91
79.11
73.78

This demonstrates the effectiveness of processing raw keypoints with a feedforward network.
Our models are still outperformed by the two attention-based graph convolutional networks, but
those are much heavier and would likely be less suitable for applications with hard run-time
constraints (e.g., memory footprint, inference time), such as autonomous driving. Additionally,
traffic control gestures are designed to be unambiguous actions that could easily be understood
without temporal context, which means temporal information should not be crucial for thesegestures. The close results of our single-frame and temporal models confirm this observation.

3.2Experiments on TITAN

The TITAN dataset (Malla et al., 2020) has 700 video clips captured by an on-board camera,
which are suitable for evaluating the complete recognition workflow starting from raw images.
All annotated actions belong to atomic, simple context, complex context, communicative or transportive action groups. Notably, all the people in all the frames are annotated with five action
labels, i.e., one from each action group (including labels for no action for any group).
In Table 2, we compare our multitask model, using five prediction heads to match the five actiongroups of TITAN, with I3D (Carreira & Zisserman, 2017) and 3D ResNet (Hara et al., 2018).
We observe it has comparable accuracy to the other two methods. However, TITAN is highly
imbalanced (toward no action for almost all groups), thus the overall accuracy is not a suitable
metric to evaluate results. For this reason, we introduce mean Average Precision (mAP), whereAverage Precisions (APs) are computed for all classes separately and then averaged.
However, since the labels no action dominate most action groups, and some actions have insufficient numbers of examples, we focus on a subset of actions where our multitask model has
reasonable mAP. We also merge actions biking and motorcycling with close meaning, resulting
in five classes: walking, standing, sitting, bending and biking. Table 3 compares the recognition
performances of four models using mAP. Our multitask model is trained on the original TITANdataset, and we only keep the predictions corresponding to the selected action subset. Since theoriginal dataset contains considerable no action samples, the training process of this model isdominated by this majority class, and the model does not have satisfactory mAP. The followTRISTAN XI Symposium

Original abstract submittal

Weijiang Xiong, Lorenzo Bertoni, Taylor Mordan and Alexandre Alahi

4Figure 2 – Action recognition examples from our single-frame model on TITAN (Malla et al.,2020) test set. Predicted actions and ground truths (GT) are shown at the bottom of the boxes.
ing three models are trained and tested on the selected action subset. The first of them is aResNet50 (He et al., 2016) classification network trained on image crops centered on detectedpedestrians. For our temporal model, we follow the procedure used in TITAN (Malla et al.,2020) and obtain temporal sequences by associating detected poses using ground-truth track
IDs. The results show that our temporal model is better at walking and standing, two visuallyclose actions where temporal context should help disambiguation, while our single-frame model
is better for sitting, bending and biking, for which temporal information is not as important.
Figure 2 presents several qualitative examples from our single-frame model.

4Conclusion

In this work, we present a simple method for action recognition based on 2D human poses.
We compare our method with an end-to-end network leveraging raw images as input, and withvarious keypoint-based approaches. Our results show that processing keypoints with a simple
feedforward network is an effective approach for action recognition in transportation applications.

References
Bertoni, Lorenzo, Kreiss, Sven, & Alahi, Alexandre. 2019 (October). MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation. In: the IEEE International Conference on ComputerVision (ICCV).
Carreira, Joao, & Zisserman, Andrew. 2017. Quo vadis, action recognition? a new model and the kinetics
dataset. Pages 6299–6308 of: proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition.
Hara, Kensho, Kataoka, Hirokatsu, & Satoh, Yutaka. 2018. Can spatiotemporal 3d cnns retrace thehistory of 2d cnns and imagenet? Pages 6546–6555 of: Proceedings of the IEEE conference onComputer Vision and Pattern Recognition.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, & Sun, Jian. 2016. Deep residual learning for imagerecognition. Pages 770–778 of: The IEEE Conference on Computer Vision and Pattern Recognition(CVPR).
Kreiss, Sven, Bertoni, Lorenzo, & Alahi, Alexandre. 2021. OpenPifPaf: Composite Fields for Semantic
Keypoint Detection and Spatio-Temporal Association. IEEE Transactions on Intelligent Transportation Systems, March, 1–14.
Malla, Srikanth, Dariush, Behzad, & Choi, Chiho. 2020. Titan: Future forecast using action priors.
Pages 11186–11196 of: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition.
Martinez, Julieta, Hossain, Rayat, Romero, Javier, & Little, James J. 2017. A simple yet effective baselinefor 3d human pose estimation. Pages 2659–2668 of: The IEEE International Conference on ComputerVision (ICCV). IEEE.
Pham, Dinh-Tan, Pham, Quang-Tien, Le, Thi-Lan, & Vu, Hai. 2021. An Efficient Feature Fusion of Graph
Convolutional Networks and Its Application for Real-Time Traffic Control Gestures Recognition. IEEEAccess, 9, 121930–121943.
Wiederer, Julian, Bouazizi, Arij, Kressel, Ulrich, & Belagiannis, Vasileios. 2020. Traffic Control Gesture Recognition for Autonomous Vehicles. Pages 10676–10683 of: 2020 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS). IEEE.

TRISTAN XI Symposium

Original abstract submittal