# A Deep Attention Model for Action Recognition from Skeleton Data
## 3 Proposed Method
### Triplet Attention Loss  
It is known that different joints can be of different degrees of importance to different action classes. Therefore, the attention weights for different joints are different for different action classes. For example, \"Kicking\" mostly focuses on the leg, while \"Punching\" mostly focuses on the hand. Accordingly, the attention weights assigned to the informative joints are supposed to be larger than others, and the attention weights for samples from one class are supposed to be more similar than those from different classes. However, this cannot be achieved in the current attention model framework learned completely by the final objective without any direct constraints on the attention model. To address the above problem, a new triplet loss function is proposed to guide the learning of the attention weights in addition to the final classification objective. This new loss function enforces the intra-class attention distance to be smaller than the inter-class attention distance. As mentioned before, for skeleton-based attention models, the attention weights are assigned to the joints and the mapping between attention and joints is fixed. Therefore, for different samples in the same class and different classes, the attention distance over the joints can be calculated, and thus, the triplet loss can be implemented. The details on the new triplet loss functions are defined as follows. For a skeleton sample \(v_{i}\) (_anchor_), let the attention weights on the sample at time step \(t\) be \(s^{a}_{i,t}\). A sample from the same action class represents the _positive_ sample whose attention weights is \(s^{p}_{i,t}\), and one sample from different action classes represents the _negative_ sample whose attention weights by \(s^{n}_{i,t}\). The following constraint stands. \[||s^{a}_{i,t}-s^{p}_{i,t}||^{2}_{2}+\alpha<||s^{a}_{i,t}-s^{n}_{i,t}||^{2}_{2} \tag{2}\]  
where \(\alpha\) is a margin that is enforced between  
one action. To overcome this issue, a new triplet loss is proposed based on a sample to class distance, instead of the distance between the attention weights of anchor and one positive. Figure 2 illustrates the new triplet loss. The anchor to the positive class distance is defined as the minimum distance between the anchor and multiple positive samples, \(\min\limits_{m=1,2,\ldots,M}||s^{a}_{i,t}-s^{p_{m}}_{i,t}||_{2}^{2}\), where \(M\) is the number of samples used in the positive class. Through learning, the attention of the anchor sample is pushed to be closer to the attention of the positive classes. In this way, the attention weights for the anchor only need to be closer to any sample in the positive class than the negative sample, which allows for multiple sets of attention weights for each action. This also compensates, to some extent, the differences of attention (if there is any) due to the styles of performing the actions or other factors. To reduce the computation, the positive class is represented by a few randomly selected positive samples. Accordingly, the new triplet loss (\(L_{tri,t}\)) follows  
\[L_{tri,t}=\sum\limits_{i}^{N}[\min\limits_{m=1,2,\ldots,M}||s^{a}_{i,t}-s^{p_{ m}}_{i,t}||_{2}^{2}-||s^{a}_{i,t}-s^{n}_{i,t}||_{2}^{2}+\alpha]_{+}. \tag{4}\]  
Assuming the probability of the attention set used by the current anchor sample is \(p_{a}\), the probability of at least one sample in the selected positive class samples sharing the attention set as the current anchor sample is \(1-(1-p_{a})^{M}\). Comparing with the probability (\(p_{a}\)) of drawing one random positive sample sharing the attention set, the probability is higher with \(M>1\), and it is much higher when \(M\) is large. Notice that different action samples may be not well aligned in the time domain in practice because of the varying starting points and speeds of an action being performed. Accordingly, the frame-by-frame distance between the anchor sample and the positive sample could be large even if they may share similar attention weights on some key frames.