# Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction
## 1 Introduction  
Predicting future action information is important due to its undeniable applications in computer vision and robotics, among which action semantics and human motion predictions are two key aspects [1]. As shown in Figure 1, our prediction task is different from the traditional recognition task. Our prediction task aims to predict action-related information before the action completes its execution, including action semantics and future human motion. Due to the limited action information of partial observed sequences, the action prediction task is particularly challenging. As shown in Figure 1, the action execution of the partial observed sequence is incomplete. Its semantics could be ambiguous, such as \"Directing\" and \"Basketball Shots\", and its future movement trend may be uncertain. Aiming at these challenges, these works [2, 3] predict multiple plausible human motion sequences. Unlike these tasks, we aim at predicting future action information accurately using a limited partial observed sequence as soon as possible. Therefore, to address this challenge, the action semantics and human motion predictions can be associated with eachother. For example, the action semantic information can reduce future motion uncertainty, which helps guide the model to predict a more natural motion sequence. Conversely, future motion information can help reduce the ambiguity of the action semantics. Most of the existing works model action semantic prediction and future human motion prediction separately, which ignores the relationships between action semantics and human motion, thus leading to unsatisfied predictions. For example, these works [4; 5] focus on predicting a short certain future human motion sequence using on a short historical human motion sequence. Furthermore, these works [6; 7] focus on predicting the action semantics of an incomplete human motion sequence very early. Due to the action ambiguous semantics and uncertainty of future motion caused by limited observed information, it is challenging to separately predict action semantics and certain future motion sequences, leading to poor performance of these models. By contrast, we jointly model action semantics and future human motion predictions. In this way, these tasks can be associated with each other in a unified framework, which helps to reduce the semantic ambiguity and uncertainty of future motion. In this paper, we propose a new multi-task framework, Multi-head TrajectoryCNN, to jointly model action semantic and human motion prediction tasks. Specifically, we first introduce a trajectory block proposed in [8] as our primary building block, named the regression block. Moreover, we build an encoder using stacked regression blocks to extract general spatiotemporal features of the input sequence. Furthermore, the extracted general spatiotemporal features are shared by the classification and regression heads for computational efficiency. Finally, we build a regression head and a classification head for predictions of future human motion and its action semantics. For the regression head, we first build a decoder with the regression block to predict future poses. For the classification head, we build another decoder using a modified trajectory block and fully connected (FC) layers to predict the action semantics of human motion. Instead of using the original trajectory block, we propose a modified trajectory block named a classification block for the classification head, which shows the effectiveness of the proposed modified trajectory block. With these elegant designs, the regression head and the classification head are connected together, using each other's information to reduce the uncertainty and semantic ambiguity caused by the partial observed motion sequence. Our main contributions can be summarized as follows. 1. We improve the model performance of human motion prediction utilizing action semantics, which helps reduce the uncertainty of future movements. 2.