# Body-Part-Aware and Multitask-Aware Single-Image-Based Action Recognition
## 3 Proposed Method
### Multitask-Aware Loss Function  
The proposed method is based on multitask learning of human pose and action. The multitask loss function consists of two losses in learning as  
\[\mathcal{L}(\mathbf{I}_{i},\mathbf{p}_{i},y_{i})=\frac{1}{N}\sum_{i=1}^{N} \mathds{1}_{p}(\mathbf{p}_{i})\mathcal{L}_{p}(\mathbf{I}_{i},\mathbf{p}_{i})+ \lambda\mathds{1}_{a}(y_{i})\mathcal{L}_{a}(\mathbf{I}_{i},y_{i}), \tag{3}\]  
where \(\lambda\) is a hyperparameter to balance the losses between the pose and action tasks. \(N\) is the number of samples, \(\mathcal{L}_{p}(\cdot)\) is the loss in pose estimation, and \(\mathcal{L}_{a}(\cdot)\) is the loss in learning the human action. \(\mathbf{I}_{i}\) is the \(i\)th input image and its ground truth labels for a human pose and action are \(\mathbf{p}_{i}\) and \(y_{i}\), respectively. All elements of \(\mathbf{p}_{i}\) are set to \(-1\) if there is no available ground truth for the \(i\)th input image in the \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Layer Name** & **Image Based FS** & **Attention Image Based FS** & **Part Image Based FS** \\ \hline Conv & - & \([7\times 7,64]\) stride = 2 & \([7\times 7,64]\) stride = 2 \\ \hline Max Pooling & - & \([3\times 3]\) stride = 2 & \([3\times 3]\) stride = 2 \\ \hline Layer 1 & - & \(\begin{bmatrix}3\times 3,64\\ 3\times 3,64\\ \times 2\end{bmatrix}\) & \(\begin{bmatrix}3\times 3,64\\ 3\times 3,64\\ \times 2\end{bmatrix}\) \\ \hline Transition & - & \([3\times 3,64]\) & - \\ \hline Layer 2 & - & \(\begin{bmatrix}3\times 3,128\\ 3\times 3,128\\ \times 2\end{bmatrix}\) & \(\begin{bmatrix}3\times 3,128\\ 3\times 3,128\\ \times 2\end{bmatrix}\) \\ \hline Transition & - & \([3\times 3,128]\) & - \\ \hline Layer 3 & - & \(\begin{bmatrix}3\times 3,256\\ 3\times 3,256\\ \times 2\end{bmatrix}\) & \(\begin{bmatrix}3\times 3,256\\ 3\times 3,256\\ \times 2\end{bmatrix}\) \\ \hline Transition & - & \([3\times 3,256]\) & - \\ \hline Layer 4 & - & \(\begin{bmatrix}3\times 3,512\\ 3\times 3,512\\ \times 2\end{bmatrix}\) & \(\begin{bmatrix}3\times 3,512\\ 3\times 3,512\\ \times 2\end{bmatrix}\) \\ \hline Transition & - & \([3\times 3,512]\) & - \\ \hline Avg. Pooling & Average Pooling & Average Pooling & Average Pooling \\ \hline Fully Connected & \#classes-d & \#classes-d & \#classes-d \\ \hline  & softmax & softmax & softmax \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Configuration of the three feature streams (FS) in the proposed method for action recognition. Attention image-based FS and part limage-based FS are based on ResNet18 [23]. The layer names are the same as in Figures 2–4 (see [23] for more details). Notably, there are additional transition layers unlike ResNet18 [23]. human pose dataset. Similarly, \(y_{i}\) is set to \(-1\) if there is no available ground truth for the \(i\)th input image in the action dataset. \(\mathds{I}_{p}(\mathbf{p}_{i})\) is an indicator function that evaluates to \(1\) if the pose annotation (\(\mathbf{p}_{i}\)) is available and \(0\) otherwise. Similarly, \(\mathds{I}_{a}(y_{i})\) evaluates to \(1\) if the action annotation (\(y_{i}\)) is available and \(0\) otherwise. Therefore, for each training sample, we calculate the pose loss and/or the action loss depending on the availability of the respective annotations. For the human pose loss, \(\mathcal{L}_{p}(\cdot)\), we use a weighted mean squared error as in [14]. The action loss \(\mathcal{L}_{a}(\cdot)\) is a combination of loss in learning the action through multiple feature streams, as in Equation (3). \[\mathcal{L}_{a}(\mathbf{I}_{i},y_{i})=\mathcal{L}_{img}(\mathbf{I}_{i},y_{i})+ \mathcal{L}_{att}(\mathbf{I}_{i},y_{i})+\mathcal{L}_{part}(\mathbf{I}_{i},y_{ i})+\mathcal{L}_{concat}(\mathbf{I}_{i},y_{i}) \tag{4}\]  
Here, \(\mathcal{L}_{img}(\cdot)\), \(\mathcal{L}_{att}(\cdot)\), and \(\mathcal{L}_{part}(\cdot)\) are the loss in learning human action using the image-based feature stream (Section 3.2), attention-image-based feature stream (Section 3.3), and part-image-based feature stream (Section 3.4), respectively.