## 2 Materials and Methods
### Modifying the Loss Function  
Repulsion loss [21] was designed to enhance pedestrian detection performance in a crowd setting. Repulsion-by-surrounding is highly helpful, which is the major driver behind Repulsion Loss. Attraction-by-target loss alone may not be sufficient for training an  
Figure 3: The architecture of our modified YOLOv8. optimal detector under occlusion. Repulsion loss is made up of three components, defined as follows:  
\[Repulsion\ Loss=L_{Attr}+\alpha*L_{RepGT}+\beta*L_{RepBox} \tag{1}\]  
where \(L_{Attr}\) is the attraction term, which requires a predicted box to approach its designated target; while \(L_{RepGT}\) and \(L_{RepBox}\) are the repulsion terms, which require a predicted box to keep away from other surrounding ground-truth objects and other predicted boxes with different designated targets, respectively. Coefficients \(\alpha\) and \(\beta\) act as the weights to balance auxiliary losses. Attraction Term. With the objective of narrowing the gap between predicted boxes and ground-truth boxes measured by some kind of distance metric 1, e.g., Euclidean distance [22], \(Smooth_{L1}\) distance [23], or IoU [24], attraction loss has been commonly adopted in existing bounding box regression techniques. To make a fair comparison, the \(Smooth_{L1}\) distance for the attraction term, as in [25,26], is adopted. The smooth parameter in \(Smooth_{L1}\) is set to 2. Given the proposal \(P\in P_{+}\), the ground-truth box that has the maximum IoU is assigned as its designated target: \(G_{Attr}^{P}=\operatorname*{argmax}_{G\in G}IoU(G,P)\). \(B^{P}\) is the predicted box regressed from proposal \(P\). Then, the attraction loss could be calculated as:  
\[L_{Attr}=\frac{\sum_{P\in P_{+}}Smooth_{L1}(B^{P},G_{Attr}^{P})}{|P_{+}|} \tag{2}\]  
Repulsion Term (RepGT). RepGT Loss is designed to repel a proposal from its neighboring ground-truth objects that are not its target. Given the proposal \(P\in P_{+}\), its repulsion ground-truth object is defined as the ground-truth object with which it has the largest IoU region, except its designated target:  
\[G_{Rep}^{P}=\arg\max_{G\in G\{G_{Attr}^{P}\}}IoU(G,P) \tag{3}\]  
Inspired by IoU Loss in [24], RepGT Loss is calculated to penalize the overlap between \(B^{P}\) and \(G_{Rep}^{P}\). The overlap between \(B^{P}\) and \(G_{Rep}^{P}\) is defined by Intersection over _Ground-truth_ (IoG): \(IoG(B,G)\triangleq\frac{area(B\cap C)}{area(G)}\). As \(IoG(B,G)\in[0,1]\), RepGT Loss is defined as:  
\[L_{RepGT}=\frac{\sum_{P\in P_{+}}Smooth_{In}(IoG(B^{P},G_{Rep}^{P}))}{|P_{+}|} \tag{4}\]  
where  
\[Smooth_{In}=\left\{ \begin{array}{ll}-\ln(1-x)&x\leq\sigma\\ \frac{\xi-\sigma}{1-\sigma}-\ln(1-\sigma)&x>\sigma\end{array} \right. \tag{5}\]  
is a smoothed ln function that is continuously differentiable in (0, 1); and \(\sigma\in[0,1)\) is the smooth parameter to adjust the sensitiveness of the Repulsion Loss to the outliers. From Equations (4) and (5), the more a proposal tends to overlap with a non-target ground-truth object, a larger penalty will be added to the bounding box regressor by RepGT Loss. In this way, RepGT Loss could effectively stop a predicted bounding box from shifting to its neighboring objects that are not its target. Repulsion Term (RepBox). NMS is a necessary postprocessing step in most detection frameworks to merge the primary predicted bounding boxes that are supposed to bound the same object. However, the detection results will be affected significantly by NMS, especially for the crowd cases. To make the detector less sensitive to NMS, RepBox Loss is proposed:  
\[L_{RepBox}=\frac{\sum_{i\
eq j}Smooth_{In}(IoU(B^{P_{i}},B^{P_{j}}))}{\sum_{i \
eq j}id\left[IoU\left(B^{P_{i}},B^{P_{j}}\right)>0\right]+\epsilon} \tag{6}\]  
where \(id\) is the identity function and \(\epsilon\) is a small constant in cases divided by zero. In Equation (6), to minimize RepBox Loss, the IoU region between two predicted boxes with different designated targets needs to be small.