# Multi-Agent-Supported Tracking Based on Hidden Markov Model with Weighted Entropy
## 3 Multi-Agent-Supported Tracking Based on Hidden Markov Model with Weighted Entropy
### Details of Process of Tracking  
Based on the principle of StrongSORT [1], tracks are initialized with detections and their extracted features and assigned to an agent, as shown in Figure 1. In the \(t\)-th frame  
Figure 1: An overview of our framework. The association matching establishes correspondences between new detections and existing tracks by constructing a cost matrix using weighted entropy metrics. Missed tracks and missed detections undergo IoU matching to generate additional valid associations. Matched tracks are used as observations to update the Kalman filter. Then the updated Kalman filter uses these clues to predict the subsequent trajectory of the target through HMMs in the prediction step.  
of the video sequence \(V=\{f_{1},f_{2},\cdots,f_{L}\}\), the object detector first generates detections \(D_{t}=\{d_{1},d_{2},\cdots,d_{N}\}\), where \(N\) represents the total number of detections. Each detection contains a location, width, and height parameter, denoted by \(d_{j}=[x,y,w,h]\). These detections are distributed to agents as candidates, and the agents are allowed to either accept the detections or wait until the next frame. For agents choosing to wait, a reward of 1 is assigned if the corresponding ground-truth tracks also opt for waiting. Otherwise, they receive \(-1\) as a penalty. In the case of agents accepting detections, a detection may potentially be assigned to more than one agent. For instance, there are four tracks managed by agents \(\{\tau_{2},\tau_{4},\tau_{5},\tau_{6}\}\) that attempt to claim detection \(d_{2}\) at the same time, and it is obvious that \(d_{2}\) cannot be selected repeatedly. To address this, \(\tau_{2}\) is designated as the valid assignment, and \(\{\tau_{4},\tau_{5},\tau_{6}\}\) are allocated to the collision set \(\mathcal{T}_{collision}\) temporarily. Upon completion of all detection processing, the agents in the collision set locally compute and share the rewards of each candidate, and the total rewards are optimized to produce the final distribution of detections:  
\[r_{total}=\sum_{i}r_{i} \tag{4}\]  
\[r_{i}=\omega dist^{m}(\tau_{i},d_{j})+(1-\omega)dist^{a}(\tau_{i},d_{j}),\quad \tau_{i}\in\mathcal{T}_{collision},\quad d_{j}\in D_{candidate} \tag{5}\]  
Then the agents associate these detections with the managed track according to the cost matrix, constructed through multi-cue fusion, via weighted entropy. The cost matrix represents the similarity between existing tracks and new detections, and models a bipartite graph for the matching problem. Before fusion, the agent performs normalization on appearance and motion features. The entropy of the fused feature is then computed to quantify its uncertainty. Through entropy minimization, the optimal coefficient \(\alpha\) is derived to balance feature weights dynamically. Details of the formulations are provided in Section 3.4.  
Following association matching, the agent utilizes tracks with detections as measurements for the Kalman filter update, which involves calculating the Kalman gain, updating the state estimate, and refining the error covariance. Subsequently, with the hypothesis that pedestrians do not exhibit abrupt large displacements between consecutive frames, the agent deploys the updated HMM-based Kalman filter (detailed in Section 3.3) to predict object locations in the subsequent frame, thus providing priors for the next iteration of association matching. For missed tracks and missed detections from the initial association, the agent employs IoU matching to generate additional valid track-detection pairs. Remaining unassociated detections are initialized as new tracks, while unmatched tracks are marked for deletion unless the corresponding targets reappear in subsequent frames.