# Visual Tracking with FPN Based on Transformer and Response Map Enhancement
## 3 Proposed Method
### Transformer Feature Pyramid  
It is generally considered that feature extraction is the key step in visual tracking. During feature extraction, the high-level feature map obtained by convolution and pooling provided a better representation of semantic information. However, for similar objects, the semantic information is similar and difficult to distinguish. In the case of occlusion and small object, the object has small amounts of pixels information, so it is difficult to track well. Feature fusion can effectively combine the advantages of low-level and high-level features: low-level features contain rich spatial information such as color, texture and edge and corner points, which is conducive to locate small and similar objects, but they lack semantic information. On the other hand, high-level features pay more attention to semantic information such as categories and attributes, which is more conducive to foreground and background differentiation. The effective fusion of high-level and low-level features can improve a tracker's ability to deal with challenges such as similar objects, object deformation and scale change. Existing feature fusion methods are mainly feature pyramid, which simply concatenates high-level and low-level feature maps, or uses different level features to accomplish different tasks. However, this method only uses a simple combination of features without global interaction, which is a fundamental limitation. Based on the limitations of above methods, this paper adopts Transformer-based feature pyramid structure for feature fusion, establishes the connection of global context,and gives full play to the advantages of feature fusion. We modified the Transformer structure proposed in the literature [25]; specifically, we removed steps of position coding in the traditional Transformer and used zero-padding with its own location information for object tracking. This allowed us to obtain a lighter Transformer-based structure and better accomplish feature fusing. The module is shown in Figure 2.  
The MLP in this figure is the full connected layer, which is used to integrate the feature sequence information. Layer Norm standardizes samples one by one to make the input's distribution stable and avoid the problem of gradient disappearance. The proposed method calculates the input features through the multi-head attention mechanism as follows:  
\[ \begin{split}\text{Attention}\ (\text{Q},\ \text{K},\ \text{V})=\text{ Softmax}\ (\frac{\text{Q}\cdot\text{K}^{\text{T}}}{\sqrt{\text{d}_{\text{k}}}})\cdot\text{V}\\ \text{MultiHead}(\text{Q},\ \text{K},\ \text{V})=\text{Concat}(\text{H}_{ 1},\text{H}_{2},\dots,\text{H}_{\text{N}})\text{W}^{\text{O}}\\ \text{H}_{\text{i}}=\text{Attention}\ (\text{Q}\text{W}_{\text{i}}^{ \text{Q}},\text{K}\text{W}_{\text{i}}^{\text{K}},\text{V}\text{W}_{\text{i}} ^{\text{V}})\end{split}  \tag{1}\]  
In this formula, where Q, K, V represent query, key and value; \(\text{d}_{\text{k}}\) represents the number of channels, i.e., sequence length; N = 8 represents the number of attention heads; \(\text{W}_{\text{i}}^{\text{Q}}\), \(\text{W}_{\text{i}}^{\text{K}}\), \(\text{W}_{\text{i}}^{\text{V}}\), \(\text{W}^{\text{O}}\) represent the weight coefficient obtained by network training. This module calculates the similarity between Q and K, multiplies V by the normalized weight, and realizes the feature enhancement of V. As shown in Figure 3, multiple Transformer based feature enhancement modules are used to realize the feature fusion pyramid. Firstly, we use convolution to flatten the features of each layer into sequence information and take the middle layer Feat4 as the query of hierarchical feature fusion. Then, feature fusion is carried out with the upper and lower layers Feat3 and Feat5 to generate three groups of combinations with different feature information. At last, the results are sent to a TFE module to enhance self-attention, so as to obtain the final semantic features that integrate the underlying spatial information and deep semantic information.