# FusionTrack: Multiple Object Tracking with Enhanced Information Utilization
## 3 Methodology
### Joint Track-Detection Decoder  
The joint track-detection decoder is designed to incorporate information from both detection and tracking branches. It takes object queries and track queries as inputs and refines them with two relatively independent yet correlated branches.  
We chose the decoder of two-stage DAB Deformable DETR [12] as our initial model to minimize the difference between the detection and tracking branches. This ensures the attention calculations between features of two branches are valid. Compared to the single-stage Deformable DETR used by MOTR, our model initializes the object query based on image features rather than a fixed set of features. This design aligns with the feature-based procedure of track query generation. We did not choose the two-stage variant of Deformable DETR because its object query is obtained by linearly projecting the predicted box. Such a design prevents the appearance information from propagating to the initial query and contradicts our approach of using appearance to obtain track queries.  
The joint track-detection decoder has two branches for detection and tracking. Each of them takes a set of queries and position anchors as input. For the detection branch, a dense prediction is first performed on the image feature map \(\mathcal{F}\) output by the encoder to obtain a series of predictions \(D_{d}^{0}\). The model selects the predictions with the highest confidence and utilizes their corresponding features as the object queries \(q_{d}\). The corresponding predicted boxes are used as the detect position anchors \(p_{d}\). For the tracking branch, the appearance feature and bounding box motion of each object are used to obtain the track queries \(q_{t}\) and track position anchors \(p_{t}\). The detailed track propagation procedure is described in Section 3.6. The object and track queries and position anchors are iteratively refined through multiple layers of the joint track-detection decoder.  
As shown in Figure 3, in each layer of the decoder, the object and track queries \(q_{d^{\prime}}^{i}\), \(q_{t}^{i}\) and anchors \(p_{d^{\prime}}^{i}\), \(p_{t}^{i}\) first pass through a shared masked self-attention module for feature fusion. An attention mask is used to block the flow of information from the tracking branch to the detection branch while allowing the tracking branch to receive information from the detection branch. This design makes the detection branch completely identical to the original DAB Deformable DETR. The model can easily inherit the pre-trained knowledge of the original detector without any change. At the same time, knowledge can be effectively transferred from the detection branch to the tracking branch for better tracking. The object and track queries are then processed independently in the following FFN and cross-attention module as the original DAB-DETR to obtain queries \(q_{d}^{i+1},q_{t}^{i+1}\) for the next level. Separate prediction heads are finally used to generate offset values \(\Delta p_{d^{\prime}}^{i},\Delta p_{t}^{i}\) for iterative anchors refinement and other object properties for calculating auxiliary loss.  
Figure 3: Structure of a joint track–detection decoder layer. Both features are processed by individual feed forward networks while sharing the same attention module to allow the sharing of information.