## 4 Method
### Network Architecture
#### 4.2.3 Feature Aggregation  
Once all the selected supporting frames have been fed into the feature sampling module, we obtain a series of newly sampled feature maps \(\mathrm{U_{t,t+k}}\), where \(\mathrm{i}=1\), \(\ldots\), \(\mathrm{N}\) represents the index, and \(\mathrm{N}\) (=2 S) denotes the number of supporting frames. These groups of supporting feature maps are then accumulated onto the reference feature map. As a result, we obtain richer information regarding the same person instance, such as poses, different viewpoints, and illuminations. For the feature aggregation block, we employ adaptive weights at various spatial locations, with all feature channels of the newly sampled feature maps sharing the same weight. The output of the feature aggregation block is given by  
\[\overline{\mathrm{U}}_{\mathrm{t}}=\sum\limits_{\mathrm{k}=-\mathrm{S}}^{ \mathrm{S}}\omega_{\mathrm{t,t+k}}\mathrm{U_{t,t+k}} \tag{1}\]  
where S is the range of supporting frames for feature aggregation. Equation (1) bears similarity to the attention mechanism [36]. The adaptive weights signify the importance of each supporting frame \(\mathrm{I_{t+k}}\) to the reference frame \(\mathrm{I_{t}}\), where \(\mathrm{k}=1\), \(\ldots\), S denotes the index of the supporting frames. To compute the adaptive weights \(\mathrm{w}\) accurately, we employ an embedding sub-network \(\mathrm{E(x)}\) to learn the feature representations of video frames in a high-dimensional space. Specifically, at each location \(\mathrm{(x,y)}\), if the newly sampled feature \(\mathrm{U_{t,t+k}}\) is similar to the feature \(\mathrm{F_{t}}\), it is assigned a relatively large weight. Conversely, it is assigned a relatively smaller weight. In this case, we employ cosine similarity between the features extracted from the reference frame \(\mathrm{F_{t}}\) and each corresponding point in the newly sampled feature \(\mathrm{U_{t,t+k}}\). Our embedding sub-network consists of four layers: a \(1\times 1\times 512\) convolution, a \(3\times 3\times 512\) convolution, and a \(3\times 3\times 1024\) convolution. It is randomly initialized.  
We calculate the adaptive weights \(\omega_{\mathrm{t,t+k}}\) with  
\[\omega_{\mathrm{t,t+k}}=\exp\left(\frac{\mathrm{E(F_{t})}\cdot\mathrm{E(F_{t +k})}}{|\mathrm{E(F_{t})}||\mathrm{E(F_{t,t+k})}|}\right) \tag{2}\]  
where \(\mathrm{E(\cdot)}\) denotes the representation of the individual frame in the high-dimensional feature space. Subsequently, we employ the exponential function to measure the similarity between any two frames \(\mathrm{(I_{t},I_{t+k})}\) and normalize the adaptive weights \(\omega_{\mathrm{t,t+k}}\) across all supporting frames with a softmax layer. The softmax layer enables the conversion of adaptive weights \(\omega_{\mathrm{t,t+k}}\) into a probability distribution, yielding \(\sum_{\mathrm{k}=-\mathrm{S}}^{\mathrm{k}=\mathrm{S}}\omega_{\mathrm{t,t+k}}=1\), which guarantees that the weights accurately capture the relative importance of each supporting frame in relation to the reference frame. After applying a softmax layer, the process of calculating each weight \(\omega\) can be interpreted as measuring the cosine similarity between a reference frame and a supporting frame in the embedding feature space.