PBADet: A One-Stage Anchor-Free Approach for Part-Body Association
Authors: Zhongpai Gao
Huayi Zhou
Abhishek Sharma
Meng Zheng
Benjamin Planche
Terrence Chen
Ziyan Wu
Venue: ICLR 2024 poster (2024)
Keywords: Human detection
part detection
association detection
anchor free
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: This paper presents PBADet, a novel one-stage, anchor-free approach for part-body association detection. We introduce a singular part-to-body center offset that effectively encapsulates the relationship between parts and the parent bodies.

Abstract: The detection of human parts (e.g., hands, face) and their correct association with individuals is an essential task, e.g., for ubiquitous human-machine interfaces and action recognition. Traditional methods often employ multi-stage processes, rely on cumbersome anchor-based systems, or do not scale well to larger part sets. This paper presents PBADet, a novel one-stage, anchor-free approach for part-body association detection. Building upon the anchor-free object representation across multi-scale feature maps, we introduce a singular part-to-body center offset that effectively encapsulates the relationship between parts and their parent bodies. Our design is inherently versatile and capable of managing multiple parts-to-body associations without compromising on detection accuracy or robustness. Comprehensive experiments on various datasets underscore the efficacy of our approach, which not only outperforms existing state-of-the-art techniques but also offers a more streamlined and efficient solution to the part-body association challenge.

Published as a conference paper at ICLR 2024
PBADET: A ONE-STAGE ANCHOR-FREE APPROACH FOR PART-BODY ASSOCIATION
Zhongpai Gao', Huayi Zhou', Abhishek Sharma', Meng Zheng', Benjamin Planche', Terrence Chen', Ziyan Wu'
'United Imaging Intelligence, Burlington, MA 01803, USA Shanghai Jiao Tong University, Shanghai 200240, China
(first. last @uuii mowii-aie com, s sjtu zhy0s jtu, edw.
ABSTRACT
The detection of human parts (e.g., hands, face) and their correct association with individuals is essential task, e-g., for ubiquitous human-maching interfaces and action recognition. Traditional methods often employ multi-stage processes, rely on cumbersome anchor-based systems, or do not scale well to larger part sets. This paper presents PBADer, a novel one-stage, anchor-fre ffreer--feeeeeee approach for part-body as- sociation detection. Building upon the anchor-fire free object representation across multi-scale feature maps, we introduce singular part-to- part-to-boyy center offset that effectively encapsulates the relationship between parts and their parent bodies. Our design is inherently versatile and capable of managing multiple parts-to-body associations without compromising on detection accuracy or robustness. Compre- hensive experiments various datasets underscore the efficacy of our approach, which not only outperforms existing state-of- tfate-of-the-ard techniques but also offers a more streamlined and efficient solution to the part-body association challenge.
Body Object
Body Object
d² (1. 8), LAN.A, &e d5 22g 22
- (6) AC..A.AAA AC.
1, -)
Body Object
Body Object (2), 27,47.00
(L. 16, AC, D.L.A.
D.I.O.O..m.
(LA2, &2 0.0.I.
0,011.0.s. m.).)....))))
left hand
letthand
Of = (Kf' . 66.
C1AA Ap.
0.001.
right hand
righthand
Object (0. 0.0.C.
Object
(-----
Representation Length: 6+K+2K (K=
Representation Length:
(k= Parts)
to- P Poayyoo-PaAsssiiaat Association (BP) Det)
Part-to- Body Association (Our proposed PBADet:
Figure 1: A comparative illustration between BPJDet (Zhou et al. 2023a and our PBADer in terms of the extended representations, Using the joint detection of three parts head, left hand, and right hand-as as an example, the original BPJDet's body-to-pan configuration demands an extended rep- resentation of length (6+K+2K, K=3). Due to unused positions in the part object representation, this approach result in inefficiencies and redundancies. In contrast, our PBADet, operating part-to-body principle, adopts a more concise representation length 10 (5+K+2, K=3), optimizing for greater utilization and efficiency.
INTRODUCTION
Human part-body association refers to the task of detecting human parts within an image and identi- fying the location of the corresponding person for each detected part, e.g., left and right hands, face, left and right feet, etc. This task vital in scenarios where multiple individuals are present, and, e.g., specific gestures from particular person must be recognized and acted upon. An illustrative example is found in medical scan rooms, where technicians use hand gestures to locate the scanning area of patient and initiate the scanning process. In such intricate scenarios, it is crucial that the

---

Published as a conference paper at ICLR 2024
system responds only to the technician's 's gestures, effectively filtering out the patient's hands avoid unintended interference. By achieving this nuanced recognition, part-body association serves as a foundational technology across various fields, including human-computer interaction. virtual reality, robotics, and medical analysis, paving the way for more intuitive and precise control systems.
Despite the evident necessity of part-body association, achieving accuracy in complex scenarios mains a significant challenge. Existing approaches often involve two-stage processes that rely on heuristic strategies or learned association networks. For example, [Zhou al.) (2018) presents an approach that detects hands and the human body pose, employing a heuristic matching strategy for hand-body association; specifically, they verify whether a left or right arm joint falls within a bound- ing box of raised hand. BodyHands (Narasimhaswany et al.] (002)) proposes to detect hands and bodies separately, utilizing a hand-body association network to predict association scores between them. However, these methods predict hand-body body relationships in two distinct stages. The two-stage nature potentially introduces complexity and possibly compromises efficiency, highlighting the need for more streamlined approach to hand-body association.
2023b) introduces a body-part joint detector that can be integrated with one- stage anchor-based detectors, such as the YOLO series (Redmon et al. 2016; Redmon & Farhadi 2018 2018|Jocher Jocher et al. 2022 Wang et al. 2023b). This method extends object detection by incorporating center offsets from body to specific body parts, such as the left and right hands, left and right feet, and head. These center offsets act as bridges, enabling body-part association with post-processing. However, the method poses some inherent challenges. Incorporating center offsets each body part means that as the number of body parts increases, so does the complexity of the object detection representation, which causes the lack of flexibility handle various numbers of body parts. This increase in complexity leads to a corresponding degradation in the overall detection performance. Furthermore, not all body parts are always visible within a body image, and when they are obscured, the corresponding center offsets become invalid, rendering the method less effective.
BPJDet (Zhou et al.,
This paper introduces novel one-stage mncho-ffree approach for part-body association, called PBADet, which creatively extends anchor-free object detection by introducing single part-to-body center offset. The key innovations of our method are detailed in three primary aspects. First, anchor- free object detection methods (Tian et al.) 2019; Feng et al., (221)) inherently contain information about part-body association, as each location within a bounding box predicts a 4D vector repre- senting distances to the bounding box's sides. Our approach leverages this natural relationship by supplementing with a 2D vector, denoting the distances from the location to the center of the body bounding box, thereby explicitly representing the part-hody association in a logical extension to current techniques. Second, using a single part-body center offset, our method can accommodate any number of body parts without increasing the number of center offsets correspondingly. This scalable design avoids degrading the overall object detection performance and provides an elegant solution for part-body association. Lastly, unlike body-part center offsets, which may involve one- to-many correspondence and become invalid when some parts are invisible, our method guarantees a one-to-one correspondence between each part and a body. The part-body center offset always valid, providing a well-defined ground truth for supervised training. In addition, this one-to-one correspondence simplifies post-processing and ensures precise part-body associations.
In summary, the contributions of this paper are as follows:
We introduce novel extension one-stage anchor-free object detection methods for part-body association. This extension is characterized by incorporating a single part-body center offset, offering a streamlined and intuitive approach to identifying part-body relationships.
Our proposed method serves as a universal framework, capable of supporting multiple parts- to-body associations. The innovative design ensures scalability without compromising the ac- curacy and robustness of the detection process.
Through experiments, we validate that our method delivers state-of-the-an performance, demonstrating its efficacy and potential as new standard in part-body association techniques.
2
RELATED WORK
Body-part Joint Detection. Human body and part detection fall within the broader category of
general object detection (Girshick 2015 Redmon et al. 2016 Liu et al. (016) and have seen signif-
2

---

Published as a conference paper at ICLR 2024
icant exploration and advancement in recent years (Dollar et al.
2011 Liu al. 2019 Deng al.
2020 Bambach et al. 2015). This surge in research has been facilitated by the availability of public (Jin et al.] 2020), COCO-HumanPale
human body and part datasets, such as
Yang et al. 2020), CrowdHuman (Shao et al. 2018), Wider Face (Yang et al. 2016), among others.
In the context of this rich landscape, our paper specifically targets the challenge of human body-pan
joint detection and association.
Previous works have approached the task of body-part joint detection using various strategies. DA- RNN (Zhang et al. 2019) proposes to detect bodies and heads in pairs using a double-anchor RPN to enhance person detection in crowded environments, JointDet (Chi et al., 2020b) incorporates a head-body relationship discriminating module to facilitate relational learning between human bodies and heads, aiming to improve human detection accuracy and reduce head false positives. BFJDet (Wan et al. 2021) introduces a head hook center in object representation and employs an embedding matching loss to associate the body and face of the same person. Hier R-CNN (Yang et al. 2020) extends the Mask R-CNN (He et al. 2017) pipeline, adopting a two-branch system to detect human bodies through the faster branch and human parts through an anchor-free Hier branch utilizing a per-pixel prediction mechanism.
Detector-in-Deeectee (Li et al. (019)) takes similar approach, using Faster R-CNN (Ren et al., (0155) with two detectors to separately focus on the human body and parts in a coarse-to-fine manner. Distinet from these methods, our paper introduces one-stage anchor-free approach for hand-body association.
Human-object Interaction. Human-object interaction (HOI) and body-part association aim to un- derstand spatial relationships but present distinct characteristics. In the realm of HOI, various meth- ods have been proposed to handle different aspects of the problem. GPNN (Wang et al., 2020) treats HOI as a keypoint detection and grouping challenge, whereas HOTR (Kim et al. 202 directly pre- dicts human, object, and interaction triplets from an image, utilizing a transformer encoder-decoder architecture. Additionally, in the specialized field of hand-contact detection, Shan et al. (Shan al.) 2020) developed hand-object detector that furnishes detailed hand attributes such as the location, side, contact state, and bounding box of the interacting object. ContactHands (Narasimhaswamy et al. (2000) builds upon this by localizing hands and deploying another object detector to determine physical contact. However, part-body association focuses solely on the connection between human parts and the body, simplifying the task but requiring specific attention to this nuanced relationship.
pose estimation, exemplified by works such as Jin (et al. (2022); Nie et al. (2019); Newell al. (0117), bears significant resemblance the task of hand-body association. In Jin et al. (2022). the concept of centripetal offsets is introduced to effectively group human joints, thereby facilitating efficient and accurate multi-person pose estima- tion. YOLO- (Maji et al. 2022). an ad vanncmment in the YOLO series, integrates bounding box detection with simultancous 2D pose prediction for multiple individuals within single processing framework. Similarly, ED-Pose (Yang et al. (2023) employs a dual explicit box detection approach to synergize human-leve level and keypoint-level contextual learning. Despite these advancements, multi- person pose estimation methods typically do not provide specific bounding boxes for human parts, an essential element for tasks like hand-body association.
Multi-person Pose Estimation.
Note that while Jin et al. (2022) and our proposed method both utilize part-to-body center offsets, their functionalitics and roles within each framework are different. Jin et al. (2022) leverages these offsets primarily for grouping in pose estimation, whereas our method focuses establishing one-to-one correspondence for accurate part-body association.
Anchor-free Detection. The utilization of anchor boxes, featured prominently in detection archi- tectures such as Faster R-CNN (Ren et al.) 2015), SSD (Liu et al. 2016). and YOLOOv3 (Redmon & Farhadi 2018), has become foundational in many detection frameworks. Nevertheless, these an- necessitating meticulous tuning for optimal per- formance. Furthermore, they present challenges from the imbalance between positive and negative training samples. Crucially, for our specific task of part-body association, the port-body center offset can fall far outside the predefined anchor boxes designated for the parts. This characteristic makes the task of learning port-body association using anchor- machor-based methods particularly challenging.
chor boxes introduce numerous dyper-parameee
Recent years have witnessed the emergence of several anchor-fir free methods. YOLOv1 (Redmon et al. 2016), for example, determines bounding boxes using points proximal to object centers. How-
3

---

Published as a conference paper at ICLR 2024
ever, this approach exhibits a lower recall than its anchor-based counterparts, primarily because it relies solely on near-center points. In a departure from this strategy, FCOS (Tian et al. 2019) lever- ages all points within a ground-truth bounding box, incorporating a centemness branch to suppress suboptimal detected boxes. However, this method performs object classification and localization independently, which may cause a lack of interaction between the two tasks, leading inconsis- tent predictions. Instead of using a centerness branch, TOOD (Feng et al.) (2021) introduces task alignment learning to enhance interaction between the two tasks for high-quality predictions. This refined concept from TOOD serves as the foundation for our part-body association method.
3 METHODOLOGY
3.1 TASK FORMULATION
Given an input image, we consider the task of regressing of bounding boxes, the corresponding classes (i.e., body and each part), and the corresponding association offsets to the bodies.
Archor-free Dense Bounding-box Prediction. Let F. E X represents the feature map at layer i E {1.... } of a backbone CNN, with s; being the total stride up to that layer; H,, W and C, the height, width, and depth of the feature maps respectively, and I. the number of layers whose feature maps are considered. We task a sub-network to regress the bounding-box of each target and another to classify the boxes, i.e., to match the ground-truth targets T = (B,c). Here B {ai, Me, Zr, yb E I denotes the coordinates of the left-top and right-botttom corners of the bounding box; c E 21,2,... for the total number of classes. For example, N 4 when focusing hand, and face. In the anchor-free free paradigm, detection is formulated as a dense inference, i.e., in per-pixel prediction fashion in feature maps. For each position y; (I22, in F,, the detection head regresses a 4D vector (4i, (,, rı, be), which represents the relative offsets from the four sides of a bounding box anchored in pt. Based on the relation (zz, 34) (L= l between feature map locations p; and corresponding locations p = (ar, y) the original image, the predicted values should satisfy the following equations w.r.t. the ground-truth B:
.N¹ specifies the class of the object within the bounding box; N stands the body, left hand, right
= 1;
1/t
9: t.
Ir x
Ub S;
y/
by.
S;
5;
Note that anchor points p. are selected from multi-level feature maps, which aids in detecting objects of varying sizes and enhances the robustness of predictions. Similarly, classification head densely returns& a score vector 02 E RN w.r.t. each class for each position in the feature maps.
Part-to-body Association. In anchor-free approaches, all points enclosed within ground-truth bounding box are treated as positive samples, i.e., only those are used to supervise the bounding box during training (Tian et al. 2019).. We utilize this property to naturally enable part-to-body associativity: any anchor point belonging to part's 's bounding- box should also belong to the body's 's bounding box; and thus should satisfy Equation (T) for both sets of bounding parameters. Hence, we can extend the part detection task to not only predicting the 4D vector corresponding to the part's own bounding box, but also second vector pertaining to the corresponding body's 's bounding box. For a more concise delineation of the part-body association, we define the second vector as only the 2D center offset from part to body (i.e., transitioning from a 4D vector to a 2D vector).
Therefore, we extend the ground-trute -ruth target as T = {B, c, c²} E R x f1,2.... .N} x R2, where ch = {ch c is the center of the body that encloses the part. For an anchor point within the part's bounding box, we thus additionally regress 2D vector (m, ni), representing the offset from the anchor point to the body center to encode the part-body association, which should satisfy:
b y 8,
24 + Am4,
y, + An4.
(2)
where À is scaling factor of m; and n, to control the range of the network outputs. The part-body association prediction is also performed over the multi-level feature maps.
In summary, our per- per-position network predictions can be denoted as o for, Oc: od), where os- (4, tu r.. b,} is the bounding box prediction, 02 = ....... ,,,, } is the classification result, and or = (mi, ni) relates to the part-body association. It is worth noting that our discussion of the human
4

---

Published as a conference paper at ICLR 2024
Dataction Head
calls
Classific ation
Ny
Localization
Displacement
Tep- Displacement
PS
P4
Hy
Detection Head
Detec tion Head
Input
Backbone
Multi-scale features
Detection heads
Output
Figure 2: Illustration of the proposed pipeline.
part-body association serves representative example. However, this formulation stands universal framework, aptly addressing various parts-to-body association challenges (e.g., the wheel- and-car association) without requiring any modifications. Since define the part-body association as 2D vector that denotes the center offset from the part to the body, it can seamlessly accommodate associations involving multiple parts.
3.2 NETWORK ARCHITECTURE
Our approach is founded on one-stage anchor-free detectors, including models like YOLOv5 (Jocher 2020), YOLOv7 (Wang al. 2023a). YOLOv8 (Jocher et al. (023). among oth- ers. While YOLOv5 and YOLOv7 were originally designed as anchor- based methods, they can be conveniently adapted with anchor-free heads. Given the feature maps F; IR M, xW; produced by the backbone network inputs, the detection head comprises three distinct output branches. These branches are responsible for bounding box prediction, class prediction, and part-body associ- ation prediction, respectively. Each branch is constructed using three-layer convolutional network, where the kernel sizes are {3 x 3, 3 X 3, 1 x 1} and the stride
Specifically, the bounding-boo sub-network has the channel structure {C,, L C J. { J. 64) and is followed by a DFL module (Li et al. 2022) to output O2 =E B. (2D map of og predictions), the class branch adopts (Ci. C, C, N} to output O ", and the part-body association branch uses {C,, C.
C:
2} to output O₂ E xd,Wxx22
3.3 LOSS FUNCTIONS
Our method incorporates the task-alignment learning strategy from TOOD (Feng et al. 202 to
supervise bounding box and class predictions, which includes a bounding-bon loU loss Liou. bounding- DFL loss Laft. and a classification loss Cets (refer Feng al. (202I) and Li al. (022) for the explicit definition of these functions). Furthermore, dedicated part-body association loss introduced for our specific objective.
Given the feature maps F; E ". XCx the part-body association detection branch produces output O₂ E R " xx.2 indicating that each anchor point yields 2D vector. Drawing inspiration from the task-alignment learning concept, the anchor assignment for the part-body association mains consistent with those used for bounding-box and class supervision-""a well-aligned anchor point should be able to predict high classification score with precise localization jointly" (Feng al. 2021). The anchor alignment metric is expressed classification score
şa u where and denote an loU value, respectively. a and 3 are hyper-parameters used to control the impact of the two tasks over the anchor alignment metric t. Utilizing the proposed metric t, we choose the top K anchor points for supervision at each training step. The part-body association loss is then articulated as:
Lassoc 11 KP
Σ
- (x.b) Am.bi)/ (mm.)++mmm)) +|| Eb,
| +|1=%b11
A
(3)
(E(L...)
with J/ representing the index list of the top K aligned anchor points for each part and indicating the number of parts in the image. Thus, the overall loss is expressed as:
Niou + Aan + Adt AessocLassoe with Xious Aapts Aetss and Aassos the objective- hyper-parameters.
(4)
5

---

Published as a conference paper at ICLR 2024
3.4 DECODING PART-TO-BODE ASSOCIATIONS
Our model predicts the center offset between a part and its corresponding body, serving as a crucial link in the part-to-body relationship. During inference, we initiate the process by filtering out over- lapping predictions through non-maximum suppression (NMS). This yields refined results for parts and bodies as follows:
O NMS(OO, -Tomf : Toou);
OP = NMS(OP,
10.00.mmmmm.. Feam Flow):
(5)
where conny fon T Too and Fio represent the confidence and loU overlap thresholds for both the body and part in the NMS procedure.
Subsequently, for each part, we compute its anticipated body center using the relationship defined in Equation as:
= + Am;). s, + Amg).
Finally, for each individual part, we determine the Euclidean (l2) distance between the estimated body center and the centers of the bodies that are unassigned and also enclose the part. The body with the smallest distance is chosen the corresponding body for the given part.
The decoding mechanism in our part-to-bddy association is notably more straightforward than the body-to-pate tortart association employed in BPJDet (Zhou et al., 2023b). In BPJDet, complexities arise since each body can be associated with various parts due to occlusions.
EXPERIMENTS
4.1 EVALUATION PROTOCOLS
Datasets. We evaluate our method on two datasets: BodyHands (Narasimhaswamy et al., 2022 and CoCoHumanParts (Yang al. 2020). Additional experiments on CrowdHuman (Shao al. 2018) are provided in Appendix A. BodyHands is large-scale dataset containing unconstrained images with annotations for hand and body locations and correspondences. This dataset consists of 18,861 training images and 1,629 test images. COCOHumanParts is an instance-level human-par dataset with rich-annotated and various scenarios based on COCO 2017 (Lin et al., (014). This dataset consists of 66,808 training images, 64,1 15 test images, and 2,693 validation images.
Evaluation Metric. We report the standard VOC average precision (AP) metric with ToU = 0.5 to evaluate the detection of bodies and parts. We also present the log-average miss rate false positive per image (IPPI) in the range [10-2, 2 10 shortened as MR (Dollar et al. 2011 of the body and its parts. To qualify the part-body association,
report the log- lvg-aeer miss matching rate (mMR 2) on FPPI part-body pairs in [10 2, 10%) which was originally proposed by BFJDet WWan et al. 202I) for exhibiting the proportion of body-face pairs that mismatched. For BodyHands we provide the conditional accuracy and joint AP defined by BodyHands (Narasimhaswamy et al. follow the evaluation protocols defined in Hier R-CNN Yang et al. (020)) and report the detection performance with series of APs (AP .5. 65. AP.s, AM, Ar) as in COCO metrics where subordinate APs represent the part-body association state.
2022). Lastly, for COCOHumanParts.
Implementation Specifies. Our experimentation framework is constructed using PyTorch (aazkee 2019), and the models are trained across Tesla V100 GPUs, leveraging automatic mixed precision (AMP) over a span of 100 epochs. Consistent with YOLOv7 (Wang et al. 2023a). we utilize the same learning rate scheduler, SGD optimizer (Robbins & Monro 1951). and prescribed data augmentations. We followe the same protocol established in BPJDet (Zhou et al. 2023b) that uses pretrained YOLO weights. The input image resolution is 1024 1024, and the batch size is 24. We employ two backbone architectures, YOLOv7 and YOLOv516, to demonstrate the versatility and adaptability of our approach.
In our experiments, we set the loss weights Aio.. 7.5, Xar 1.5, Ante 0.5, Anssoc 0.2, the
scaling factor À = 2.0, and the anchor alignment parameters K = 13, a = 1.0, B = 6.0. During
inference, we set the NMS parameters as: the body thresholds T.omf 0.05, Tho 0.6; while
the part thresholds - 0.05, 7ho 0.6 for BodyHands, and Teomf 0.005, - 0.75 for CaCosumanParts.

---

Published as a conference paper at ICLR 2024
BodyHands
DRJDet (YOLOv5))
(Y
Figure 3: Qualitative results on BodyHands. Red arrows highlight our correct predictions.
Methods
Param (M)
Size
Hand APT Cond. Accuracyt
Joint AP1
OpenPose (0177)
199.0
1536
39.7
74.03
27.81
Keypoint Com.
(2021)
27.3
1536
33.6
71,48
20.71
MaskRCNN=FD
(017)
266.0
1536
84.8
41.38
23.16
MaskRCNN+FS
2017)
266.0
1536
84.8
39.12
23.30
MaskRCNN+LD 2017
266.0
1536
84.8
72.83
50.42
MaskRCNN+IoU
(2017)
266.0
1536
84.8
74.52
51.74
BodyHands (2022)
700.3
1536
84.8
83.44
63.48
BodyHands (2022
700.3
1536
84.8
84.12
63.87
BPJDet (YOLOv5s6)
(2023a)
15.3
1536
84.0
85.68
77.86
BPJDet (YOLOv5m6)
2023a
41.2
1536
85.3
86.80
78.13
BPJDet (YOLOv516)
(2023a)
86.1
1536
85.9
86.91
84.39
Ours (YOLOv7)
36.9
1024
89.1
92,62
85.98
OOurs (YOLOv516)
86.1
1024
88.1
92.71
85.73
Table 1: Quantitative evaluation on BodyHands.
4.2 COMPARISON WITH STATE-OF-THE-ARY
BodyHands. We conduct the hand-body association task the BodyHands dataset Narasimhaswamy et al.] 2022). Notably, this dataset contains only one type of part annotation since it does not distinguish between left and right hands. Table compares PBADet with leading methods. Our approach surpasses competitors across all metrics, including hand AP, condi- tional accuracy, and joint AP, by large margins. In particular, our YOLOv7 model, despite having smaller model size than BPJDet (YOLOv5m6) (Zhou et al.) 2023a), delivers superior performance. Furthermore, YOLOv516 model, which utilizes the same backbone BPJDet (YOLOv516), also demonstrates enhanced results. detection and part-body association detection.
that our method demonstrates the best body and part
association task using the COCOHumanParts dataset (Yang et al. 2020), which includes six distinct body parts: head, face, left hand, right hand, left foot, and right foot. Tables 2 and provide comparative analysis with leading methods. As depicted in Table our method achieves APs that are on par with BJPDet (Zhou al. (023b). Notably, we lead in metrics such APM. APL, and APs specific person and feet detections. Furthermore, Table underscores our method in achieving the lowest average mMR across categories like head, face, both left and right hands, and right foot, emphasizing
COCOHumanParts. We evaluate our approach for the

---

Published as a conference paper at ICLR 2024
Hier R-CNN
BYJDet
(YLOOO
Figure 4: Qualitative results on CoCOHumanParts, Yellow and red arrows highlight other methods' failure cases and our correct predictions, respectively.
All cutegories APs (uriginal subordinate)
Per categories APst
Joint Detect?
Methods
9.96
APs
APar
APL
person head
face
r-hand
1-hand
r-foot
1-foot
Daster-C4-R50 2015
32.0/
$4.91
/
27.2
19.2
19.3
Faster-FPN-R50 2017
34.87 19.1
60/
/5.4/22.4
52.2/33.6
5144
48.7
36.7
31.7
29.7
22.4
22.9
RetinaNet-R50
20178
32.21
54.71
54.51
53.8/
49.7
47.1
33.7
28.7
26.7
19.7
20.2
FCOS-RSO R50 2019
34.1
58.6/
55.11
55.1
51.1
45.7
40.0
29.8
28.1
22.2
21.9
Faster-FPN-XX
2017
36.7
62.8
57.41
55.3/
53.6
49.7
37.3
33.8
32.2
25
25.1
Hier-R50 2020
33.8/33.3
67.7
23.9
47.5/47.1
53.2
50.9
41,5
31.3
29.3
25.5
26.1
Hier-R101
2020
37.2
65.9/
55.1
50.3
54.0
50.4
41.6
31.6
30.1
26.0
26.6
20.20
$6.6
52.3/51.1 /
33.5
27.4
27.9
Hier-R50t 2020
47.6
/
51.5/48.9
X101 2220
71.6/
53.3/50.3
BPIDe (YOLOVv5s))
38.9/38.4
64.5/64.4
59.1
49.7
56.3
53,5
41.9
34.7
33.7
25.5
26.5
BPIDet (V55005m))
42.0/41..7
68.9/68.1
62.3/61.6
54.6/529
59.8
55.7
44.7
38.7
37.6
28.6
29.2
BPJDet (YOLOv516)
43.6/43.3
7
63.8/63.2
58.8/589
61.3
56.6
46.3
40.4
39.6
30.2
30.9
Ours (YOL.vv7) (YOLOv516)
42.7
69.5
65.8
64.4
66.7 66.7
63,
62.1
54.5
44.7
38.7
37.5
30.4
30.9
41.8
70.0/
66.2/65.0
6
39.0
30.9
31.5
Table 2: Quantitative evaluation on COCOHumanParts with all and per categories APs. All cate- gories APs include the original detection APs and subordinate APs. The marker 1 in Hier R-CNN indicates using deformable convolutional layers and multi-scale training (Yang et al. 2020).
method's superior part-body association capabilities. Figure 4 demonstrates that method performs as well as (or better than) BPJDet and much better than Hier R-CNN.
Comparison to Multi-person Pose Estimation. We present a qualitative analysis contrasting our approach with ED-Pose (Yang al. 2023), state-of-the-art multi-person pose estimation method,
as illustrated in Figure (see quantitative restuls in Appendix A.2) Despite utilizing the powerful
Swin-L backbone (Liu al. 2021 ED-Pose occasionally provides imprecise speculative pre-
Methods
Params (M)
Size
head
face
I-hand r-hand
r-foot
I-foot
BPJDet (YOLOv5s6)
15.3
1536
33.4
36.7
50.2
51.4
58.3
58.4
BPIDet (Y0LOv5m6)
41.2
1536
29.7
32.7
42.5
44.2
S3.4
52.3
BPJDet (YOLOv516)
86.1
1536
30.8
31.8
41.6
42.2
49.1
50.4
Ours (YOLOv7)
36.9
1024
29.2
31.3
41.1
42.7
52.6
52.5
Ours YOOL0v516)
86.1
1024
27.9
30.5
36.2
40.8
50.4
47.9
Table 3: Quantitative evaluation on COCOHumanParts with the average mMR
8

---

Published as a conference paper at ICLR 2024
dictions, especially for limbs such as legs and arms. Such unreliable results cloud the discernment of the relationship between a body and its corresponding parts. Our part-body association detention method can effectively address and diminish this issue.
BD Pose (Swin-L)
A
ONES
Figure 5: Qualitative comparison with ED-Pose (Yang al. 2023 circles highlight erroncous predictions.
images from COCO. Yellow
4.3 ABLATION STUDY
Methods
Hand AP1
Cond. Accuracyt Joint AP1
w/o Cassoe (baseline)
89.1
80.78
78.07
w/o Multi-scale
88.8
91.64
85.46
w/o Task-align
89.0
92.08
85.78
Full
89.1
92.62
85.98
Table 4: Ablation experiments on BodyHands with the YOLOv7 backbone architecture.
To better understand the contribution of various components in our approach, we undertake ablation experiments on the BodyHands dataset with the YOLOv7 backbone architecture, summarized in Table 4 and more in Table g in Appendix A.4 Initially. we examine configuration without the association loss, Lassoc. which serves as our baseline for detecting human bodies and parts. For part-body association in this scenario, we utilize Euclidean distances between part center and the unassigned body centers. The body that encloses the part and has the minimal distance is then tagged as the corresponding body for the given part. Compared to the baseline model, the introduction of our association prediction head maintains the accuracy in hand detection, and considerably boosts the efficacy of part-body association.
our subsequent variant, the center offset is defined relation to the multi-scale feature anchor points. When this center offset is interpreted as normalized absolute distance (i.e.. w/o Multi- scale), we observe decline in metrics, including hand prediction accuracy, conditional accuracy, and joint AP. Lastly, method adopts task alignment learning for supervising part-body associ- ations. Disregarding task alignment (Le., w/o Task-align), where all positive samples are used for training, leads to decreased performance in all metrics, in contrast with our full model.
CONCLUSION
The task of human part-body detection and association is critical for wide range of computer vision applications. In this work, we introduced an innovative one-stage, anchor-free methodology for this purpose. By extending the anchor-free object representation across multi-scale feature maps, we incorporate a singular part-to-bodi center offset, effectively bridging parts and their corresponding bodies. Our approach is purposely designed to handle multiple parts-to-body associations without sacrificing detection accuracy or robustness. Experimental results validate that our proposed method sets standards in the domain, surpassing existing state-of-the-art solutions.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sven Bambach, Stefan Lee, David Crandall, and Chen Yu. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. In Proceedings of the IEEE interna- tional conference computer vision, pp. 1949-1957, 2015.
Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. Cross- domain adaptation for animal pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9498-9507,
2019. 2019.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299, 2017.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z Li, and Xudong Zou. Pedhunter: Occlusion robust pedestrian detector in crowded scenes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10639-10646, 2020a.
Cheng Chi, Shifeng Zhang, Junliang Xing. Zhen Lei, Stan Z Li, and Xudong Zou. Relational learning for joint head and human detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10647-10654, 2020b.
Xuangeng Chu, Anlin Zheng, Xiangyu Zhang, and Jian Sun. Detection in crowded scenes: One proposal, multiple predictions. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12214-12223, 2020.
Jiankang Deng, Jia Guo. Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In CVPR, 2020.
Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona. Pedestrian detection: An evaluation of the state of the art. IEEE transactions on pattern analysis and machine intelligence, 34(4): 743-761, 2011.
Chengjian Feng, Yujie Zhong, Yu Gao, Matthew R Scott, and Weilin Huang. Tood: Task-aligned one-stage object detection. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 3490-3499. IEEE Computer Society, 2021.
Ross Girshick. Fast r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Xin Huang, Zheng Ge, Zequn Jie, and Osamu Yoshie. Nms by representative region: Towards crowded pedestrian detection by proposal pairing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10750-10759, 2020.
Lei Jin, Xiaojuan Wang, Xuecheng Nie, Luoqi Liu, Yandong Guo, and Jian Zhao. Grouping by cen- ter: Predicting centripetal offsets for the bottom-up human pose estimation. IEEE Transactions on Multimedia, 2022.
Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole-body human pose estimation in the wild. In Computer Vision-ECCe 2020: 16th European Conference, Glasgow, UK, August 28,28, 2020, Proceedings. Part IX 16, pp. 196-214 Springer, 2020.
Glenn Jocher. y
Ultralytics yolov5, 2020.
URL https:///ithub.comm/
10

---

Published as a conference paper at ICLR 2024
Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, Yonghye Kwon, Kalen Michael, Jia- cong Fang, Zeng Yifu, Colin Wong, Diego Montes, et al. ultralytixsyyoloSS: v7. 0-yolov5 sota realtime instance segmentation. Zenodo, 2022.
Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Ultralytics yolov8, 2023, https::::///tt Com/ http://traaayiisstttaaaaaa
Bumsso Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human-object interaction detection with transformers. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 74 83, 2021.
Xiang Li, Chengqi Lv, Wenhai Wang, Gang Li, Lingfeng Yang, and Jian Yang. Generalized focal loss: Towards efficient representation learning for dense object detection. JEEE transactions on pattern analysis and machine intelligence, 45(3):3139 3153, 2022.
Xiaojie Li, Lu Yang, Qing Song. and Fuqiang Zhou. Detector-in-detector:: Multi-level analysis for human-parts. In Computer Vision-ACCV 2018: 14th Asian Conference Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part ll 14, pp. 228-240. Springer, 2019.
Matthieu Lin, Chuming Li, Xingyuan Bu, Ming Sun, Chen Lin, Junjie Yan, Wanli Ouyang, and Zhidong Deng. Detr for crowd pedestrian detection. arXiv preprint arXiv:2012.06785 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung-Yl Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017b.
Songtao Liu, Di Huang, and Yunhong Wang. Adaptive nms: Refining pedestrian detection crowd. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6459-6468, 2019.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yann Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Debapriya Maji, Soyeb Nagori, Manu Mathew, and Deepak Poddar. Yolo-pose: Enhancing yolo for multi person pose estimation using object keypoint similarity loss. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2637-2646, 2022.
Supreeth Narasimhaswamy, Trung Nguyen, and Minh Hoai Nguyen. Detecting hands and rec- ognizing physical contact in the wild. Advances in neural information processing systems, 33: 7841-7851, 2020.
Supreeth Narasimhaswamy, Thanh Nguyen, Mingzhen Huang, and Minh Hoai Whose hands are these? hand detection and hand-body association in the wild. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 4889-48949 2022.
Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. Advances in neural information processing systems, 30, 2017.

---

Published as a conference paper at ICLR 2024
Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-person pose machines. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 6951-6960, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Joseph Redmon and Ali Farhadi. Yolov3: arXiv:180..02767, 2018.
An incremental improvement.
arXiv preprint
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1869-9878, 2020.
Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhu- man: A benchmark for detecting human in a crowd. arXiv preprint arXiv: /805.00/33, 2018.
Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka. Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 14454-14463, 2021.
Yi Tang, Baopu Li, Min Liu, Boyu Chen, Yaonan Wang, and Wanli Ouyang. Autopedestrian: an automatic data augmentation and loss function search scheme for pedestrian detection. IEEE Transactions on Image Processing, 30:8483-8496, 2021.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In Proceedings of the IEEF/CVF international conference computer vision, pp- 9627-9636, 2019.
Junfeng Wan, Jiangfan Deng, Xiaosong Qiu, and Feng Zhou. Body-face joint detection via embed- ding and head hook. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 2959-2968, 2021.
Chien-Yo Yao Wang, Alexey Bochkovskiy, and Hong-Yual Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023a.
Chien- Yao Wang, Alexey Bochkovskiy, and Hong-Yuse Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7464-7475, 2023b.
Tiancai Wang, Tong Yang, Martin Danelljan, Fahad Shahbaz Khan, Xiangyu Zhang, and Jian Sun. Learning human-object interaction detection using interaction points. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 416441, 2020.
Zixuan Xu, Banghuai Li, Ye Yuan, and Anhong Dang. Beta r-cnn: Looking into pedestrian detection from another perspective. Advances in Neural Information Processing Systems, 33:19953 19953-19963, 2020.
12

---

Published as a conference paper at ICLR 2024
Jie Yang, Ailing Zeng, Shilong Liu, Feng Li, Ruimao Zhang, and Lei Zhang. Explicit box detec- tion unifies end-to-end multi-person pose estimation. In International Conference on Learning Representations, 2023.
Lu Yang, Qing Song, Zhihui Wang, Mengjie Hu, and Chun Liu. Hier r-cnn: Instance-level human parts detection and new benchmark. IEEE Transactions on Image Processing, 30:39-54, 2020.
Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. Wider face: A face detection bench- mark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5525-5533, 2016.
Hang Yu, Yufei Xu, Jing Zhang, Wei Zhao, Ziyu Guan, and Dacheng Tao. Ap-10k: A benchmark for animal pose estimation in the wild. arXiv preprint arXiv:2 /26/7, 2021.
Duncan Zauss. Sven Kreiss, and Alexandre Alahi. Keypoint communities, In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 11057-11066, 2021.
Kevin Zhang, Feng Xiong, Peize Sun, Hu, Boxun Li, and Gang Yu. Double anchor r-cnn for human detection in a crowd. arXiv preprint arXiv: 190.99999999999 2019.
Yuang Zhang, Huanyu He, Jianguo Li, Yuxi Li, John See, and Weiyao Lin. Variational pedestrian detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11622-11631, 2021.
Anlin Zheng, Yuang Zhang, Xiangyu Zhang, Xiaojuan Qi, and Jian Sun. Progressive end-to-end object detection in crowded scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 557-866, 2022.
Huayi Zhou, Fei Jiang, and Ruimin Shen. Who are raising their hands? hand-raiser seeking based on object detection and pose estimation. In Asian Conference on Machine Learning, pp. 470-485. PMLR, 2018.
Huayi Zhou, Fei Jiang, and Hongtao Lu. Body-part joint detection and association via extended object representation. In 2023 IEEE International Conference Multimedia and Expo (ICME), pp. 168-173. IEEE, 2023a.
Huayi Zhou, Fei Jiang, Jiaxin Si, Yue Ding, and Hongtao Lu. Bpjdet: Extended object representation for generic body-part joint detection. arXiv preprint arXiv:2304. 10765, 2023b.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2024
MR 24
MR 21
APT
AP1
mMR 21
Methods
Stage
body
face
body
face
body-face
RetinaNet+PPOS 2017b)
One
52.3
60.1
79.6
58.0
73.7
RetinaNet+BFI 2017b)
One
52.7
59.7
80.0
58.7
63.7
FPN+POS (2017a)
Two
43.5
54.3
87.8
70.3
66.0
FPN+BFJ 2017a)
Two
43.4
53.2
88.8
70.0
52.5
CrowdDet+POS 2020
Two
41.9
54.1
90.7
69.6
64.5
CrowdDet+BFJ 2020
Two
41.9
53.1
90.3
70.5
52.3
BPJDet (YOL.Ovss)) (2023a)
One
41.3
45.9
89.5
80.8
51.4
BPJDet (YOLOv5m6) (2023a)
One
39.7
45.0
90.7
82.2
50.6
BPJDet (YOLOv516) (2023a)
One
40.7
46.3
89.5
81.6
50.1
Ours (YOLOv7)
One
36.5
44.7
91.6
81.6
50.8
Ours (YOLOv516)
One
37.5
45.3
91.4
81.2
50.9
Table 5: Quantitative evaluation CrowdHuman for the face-body association task.
Methods
Joint Detect?
Backbone body MR 24
body AP$
mMR
Adaptive-NMS 2019)
X
CNN
49.7
84.7
PBM (2020)
X
CNN
43.3
89.3
CrowdDet (2020)
X
CNN
41.4
90.7
AEVB (2021)
X
CNN
40.7
AutoPedestrian 2021
X
CNN
40.6
Beta RCNN(KL,))
2020
X
CNN
40.3
88.2
PedHunter (2020a)
X
CNN
39.5
SpNNse-RCNN (2021)
X
DETR
44.8
91.3
Deformable-DEi (2020)
x
DETR
43.7
91.5
PED-DETR (2020)
x
DETR
43.7
91.6
Iter-F2EDETT (2022)
X
DETR
41.6
92.5
DA-RCNN (2019)
CNN
52.3
DA-RCNN + J-NMS (2019)
CNN
51.8
JointDet (2020b)
CNN
46.5
BPJDet (YOLOv556)
(023b)
CNN
41.3
89.5
51.4
BPJDet (YOLOv5m6)
(202bb)
CNN
39.7
90.7
50.6
BPJDet (YOLOv516)
(2023b)
CNN
40.7
89.5
50.1
Ours (YOLOv7)
CNN
36.5
91.6
50.8
Ours (YOLOv516)
CNN
37.5
91.4
50.9
Table 6: The performance comparison of our method for the joint face-body detection task with other crowded person detection methods in the val-set of CrowdHuman.
A APPENDIX
A. EXPERIMENTS ON CROWDHUMAN
We additionally conduct experiments on the CrowdHuman dataset (Shao et al. 2018). CrowdHuman dataset tailored for crowded scenarios, providing individual annotations for each pedestrian. This dataset includes 15,000 training images and 4,375 validation images. The initial annotations include boxes for visible bodies and heads. Labels of faces added by BFIDet (Wan al. 2021 for the body-face association.
In experiments, the input image resolution is 1536 x 1536, and the batch size is 12. The other settings are the same as in the main paper. During inference, we set the NMS parameters as: the body thresholds 7 7 0.05, T 0.6; while the part thresholds T 0.1, Tie 0.3 in BP]Det (Zhou et al. 2023b).
14

---

Published as a conference paper at ICLR 2024
We benchmark our approach for the face-body association task utilizing the CrowdHuman dataset (Shao et al. 2018). The results, as illustrated in Table [ draw a comparison between our proposed PBADet and existing methods. Performance-wise PBADet is comparable with BPJDet, with our PBADez (YOLOv7) achieving the best metrics in MR for both body and face, as well as AP for the body. It's essential to highlight that the face-body association task is singular in nature, involving just one part. Consequently, the advantages of our method over BPJDet are not as pronounced as seen in scenarios with multiple parts-to-body associations, such as on the BodyHands and COCO- HumanPart datasets.
Besides, we also try to compare our joint detector PBADet with other methods specifically designed for crowded person detection, either CNN-based or DETR-base baseed (Carion et al., 2020). As shown in Table apart from obtaining the best MR -2 result for body detection, our PBADet achieves com- parable AP performance with those cumbersome DETR-based detectors such as PED-DETR (Lin al. 2020) and EEEDET (Zheng et al., 2022). Comparing with the counterpart BPJDet, our superiority is still evident. This indicates that our proposed PBADer for addressing the face-body association task can keep better balance between detection and association.
A.2 QUANTITATIVE COMPARISON TO MULTI-PERSON POSE ESTIMATION
ED-Pose (2023)
Ours
Parts
Precision
Recall
Fif
Precisionf
Recallf
F11
lefthand
18.36
30.34
22.87
37.63
60.82
46.50
righthand
19.57
31.59
24.17
38.58
61.26
47.34
leftfoot
29.32
65.28
40.46
41.56
66.24
51.08
rightfoot
29.13
65.58
40.34
40.86
65.55
50.34
all parts
24.09
45.51
31.51
39.41
63.09
48.52
Table 7: Comparison with ED-Pose tion method.
(Yang et al.
2023),
state-of-the-art multi-person pose estima-
We compare our method with ED-Pose (Yang et al. 2023), a current leading method in multi- person
pose estimation. A qualitative comparison with PD-P on images from COCO (Lin et al. 2014) is presented in Figure 5
For a fair and meaningful quantitative comparison, we conducted evaluations on the COCO vali- dation set (Lin et al.) 2014). ED-Pose was also trained the COCO dataset. This validation set contains 2,693 images, each featuring at least one person. To ensure consistency, we utilized the ground truth body-part bounding boxes from COCO-HumanPart which are derived from the original COCO annotations. We excluded facial and head parts due to their inconsistent number of keypoints and focused four challenging and smaller parts: left hand, right hand, left foot, and right foot, which can be determined by one keypoint. For evaluation, we used the ED-Pose model (Swin-L) trained on the COCO training and our PBADet model (YOLOv516) trained the MOCO-HumanPare training set.
In assessing performance, we considered a predicted keypoint by ED-Pose as true positive if fell within ground truth part box. In favor of ED-Pose, we do not use a part detection model but directly use the ground truth part boxes. For PBADet, a predicted part box with successful part-body association was deemed true positive when it achieved an IoU greater than 0.5 with the ground truth box. The results, as shown in the table below, indicate that PBADet demonstrates superior robustness in part-body association compared to ED-Pose, especially in the precision, recall, and Fl score metrics for the selected body parts. This comparative analysis underscores the efficacy of PBADet in handling body associations, affirming its robustness and precision existing methods in this domain.
A.3 EXPERIMENTS ON ANIMAL DATASETS
We expand our research include experiments AnimalPose (Cao al. 2019 and AP-10K YY et al., (02TT) to demonstrate our model's scalability and performance in domains beyond human part-to-body associations.
15

---

Published as a conference paper at ICLR 2024
Figure 6: Qualitative results on AP-10K (Yu et al. 2021
Our study includes five common quadruped animals: dogs, cats, sheep, horses, and cows. These animals are chosen due to the similarity in their anatomical structure, particularly having five iden- tifiable parts: head and four feet. This similarity allows us to treat these diverse animal types as a single class for analysis purposes. The body bounding boxes for these animals are derived from the provided ground truth labels in the datasets, while the part bounding boxes are generated based on keypoint annotations. In our experiments, we use 4,608 images with 6,117 instances of animal 2019) for training and use 2,000 images containing 7.962 body
bodies from AnimalPose (Cao instances from AP-10K (Yu et al., 2021) for evaluation. We train our PBADet (YOLOv516) model with the same experimental setup described in Section 4. of the main paper.
Qualitative results, as illustrated in Figure 6 showcase successful part-body association in these animals. These outcomes affirm that extending our method quadruped animals not only feasible but also yields convincing and meaningful results, demonstrating the potential of our approach for broader applications in part-body association tasks across different domains.
A.4 MORE ABLATION STUDY
Methods
Param (M)
Size
Hand APT Cond. Accuracyt Joint AP1
BPJDet lanchor-based body-to, to-part)
41.2
1536
85.3
86.80
78.13
Ours (anchor-based part-to-body)
36.9
1024
88.4
92.31
85.28
Ours (anchor-free part-to-body)
36.9
1024
89.1
92.62
85.98
Table 8: Comparison of anchor-based vs. anchor-free backbones and body-to-part vs. part-to-body definitions.
We conduct an ablation study to compare our model with the traditional anchor-based YOLOv7 model, using the same association definition of part-to-body center offset in our approach. This comparison has been instrumental in highlighting the advantages of the unchor-free design, espe- cially in the context part-body association tasks. Our findings show that the anchor-free version of YOLOv7 outperforms its anchor-based counterpart, validating the effectiveness of our method. Furthermore, we compared the traditional anchor-basee YOLOv7 model with BPJDet, which also adopts anchor-based approach but utilizes body-to-part association definition. This compar- ison demonstrates that our part-to-body association definition significantly enhances performance, offering clear evidence of the superiority of our approach over previous methods.
A.5 DETAIL RESULTS OF COCOHUMANPARTS
All categories APst (original / subordinate)
AP 55.99
AP.;
AP.
APs
APM
AP,
Ours (YOLOv7)
42.7 / /41.5 69.5 / 67.9 43.5 / 42.2 31.4 / 30.3 65.8 / 64.4 66.7 / 63.3
Ours (YOLOv516)
43.0 41.8 70.0 68.4 43.7 42.4 31.5 / 30.3 66.2 / 65.0 67.2 64.4
Table 9: Evaluation results of the proposed method on COCOHumanParts with all categories APs.
In the original Hier R- R-CNN paper (Yang et al. 2020), small objects are further categorized into small and tiny objects, and AP for small objects is not explicitly provided. Similarly, another comparison
16

---

Published as a conference paper at ICLR 2024
method, BPJDet (Zhou et al. 2023b), also does not offer AP details for small objects. We further include the results of AP, 75 and A in Table transparency and comparison to future work. We method's competitive object detection accuracy relative to
emphasize that Table 2 demonstrates state-of-the-art approaches. More importantly. the critical metric for our task is association accuracy, where, shown in Table 3 our method achieves superior performance.
17


Reviews:
[{"review_id": "ksN27VPJP7", "reviewer": "Reviewer_BLsk", "summary": "-\tIt would be better if more details about the association process could be given.\n-\tHow to choose the value of the hyper-parameter K used in L_{assoc}? The experiments would be more comprehensive if the value of K could be analyzed in the ablation study.\n-\tThe reviewer wonders whether the models are trained from scratch or the pretrained YOLO weights are used. It is not mentioned in the paper.", "questions": "-\tIt would be better if more details about the association process could be given.\n-\tHow to choose the value of the hyper-parameter K used in L_{assoc}? The experiments would be more comprehensive if the value of K could be analyzed in the ablation study.\n-\tThe reviewer wonders whether the models are trained from scratch or the pretrained YOLO weights are used. It is not mentioned in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "-\tCompared to the body-to-part center offset used in the previous one-stage method BPJDet, the part-to-body center offset proposed in this paper has better scalability in terms of the number of parts and avoids degrading the overall object detection performance.\n-\tThe PBADet method proposed in the paper achieves state-of-the-art performance on BodyHands, COCOHumanParts and CrowdHuman datasets.\n-\tThis paper is well organized, clearly presented, and effectively clarifies the differences from previous methods.", "weaknesses": "-\tThe author states that the proposed part-to-body center offset guarantees a one-to-one correspondence between parts and bodies. However, the paper does not elaborate on how the situation is handled when the predicted center offsets of multiple parts with the same category point to the same body. How to define the order for associating multiple parts with the same category? These details are not described in Section 3.4. \n-\tIn Figure 2, it seems that 'P3' and 'P5' in the 'Multi-scale features' are labeled incorrectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DGJm1OoDH8", "reviewer": "Reviewer_AkLB", "summary": "The paper presents a one-stage anchor-free approach (PBADet) for part-body association detection via singular part-to-body center offset that captures the relationship between parts and their parent bodies. It is an extension to one-stage anchor-free object detection methods for part-body association to identify part-body relationships. The approach supports multiple parts-to-body associations without compromising the accuracy and robustness of the detection process. The approach is evaluated on BodyHands, COCOHumanParts, and CrowdHuman datasets and the performance is comparable to the state-of-the-art (SOTA).", "questions": "Please refer to the \"Weakness\" section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is written well and easy to follow. The idea is very good and is inspired by the recent task alignment learning to enhance interaction between the two tasks for high-quality predictions.  \n\nThe rationale behind the anchor-free prediction, and part-to-body associations using models like YOLOv5, YOLOv7, and YOLOV8 is justified. \n\nA thorough experimental evaluation using well-known benchmarked datasets. On each dataset, the performance of the proposed approach is compared to the state-of-the-art and explains the performance gain and its impact on the overall accuracy.\n\nThe importance of the individual module model is experimentally evaluated. \n\nInteresting visualization to show qualitative comparison and highlight the erroneous predictions.", "weaknesses": "The anchor-free object representation uses multi-scale feature maps. How many scales have been used?  Is this the same as the backbone (e.g., YOLOv7) feature representation?\n\nThe approach uses a singular part-to-body center offset. The body center is the bounding box center or something else. Also, how important is accuracy in detecting the body center influencing the overall performance?\n\nIt would be nice to have a section on model capacity and computational complexity (e.g. Params, GFLOPS, per-image inference time, etc) to further improve the article. This should be compared to the other SOTA models.\n\nThe multi-scale module has a significant impact on the performance improvement w.r.t. the baseline. What could be the reason?\n\nThe loss weights ($\\lambda$ and NMS thresholds $\\tau$ values) are optimal for a given dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvwmNwK0hd", "reviewer": "Reviewer_NDUe", "summary": "The paper addresses the problem of human body part detection and association. The authors present a one-stage, anchor-free method to tackle this problem. Specifically, they introduce a part-to-body center offset to capture the relationship between parts and their corresponding bodies. The authors conduct experiments on BodyHands, COCOHumanParts, and CrowdHuman datasets.", "questions": "Please see the Weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- The proposed single-part-body center offset is efficient and can accommodate a large number of body parts without increasing the number of offsets as the number of parts grows. This makes the proposed method more efficient compared to previous methods.\n- The paper presents good experimental evaluations and comparisons with previous methods.\n- The paper also presents experimental evidence to study the benefits of each proposed component in ablation studies.", "weaknesses": "-Table 2 doesn't mention AP for small objects. The proposed method performs better than previous methods on medium and large objects. How does this compare to small objects?\n\n-How do the ablation studies change on the COCOHumanParts dataset? I am curious about the performance of different components of the proposed method on different object sizes: small, medium, and large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RClQJgedvC", "reviewer": "Reviewer_TS1Z", "summary": "This paper presents PBADet, a one-stage, anchor-free object detection model that efficiently identifies and associates object parts, such as human body parts, to their main body. Utilizing multi-scale feature maps and a unique part-to-body center offset, PBADet offers improved accuracy and robustness over existing models. It addresses inefficiencies through a task-aligned learning strategy and a simplified decoding process for part-to-body associations, making it a streamlined and effective solution in object detection.", "questions": "1. How does PBADet address the challenge of occluded parts in the detection process?\n2. How does the model perform in scenarios with multiple overlapping or closely situated objects?\n3. Have you tested the model’s scalability and performance across various domains other than human part-to-body associations?\n4. Could you provide more comparative performance of PBADet against traditional anchor-based detection models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The framework is adaptable for various part-to-body association challenges beyond human body parts.\n2. The model adopts an anchor-free paradigm, potentially improving detection performance for objects of varying sizes.\n3. Incorporation of a task-alignment learning strategy for bounding box, class predictions, and part-body association.", "weaknesses": "1. The paper doesn't explicitly address how the model deals with occluded parts.\n2. The dense, per-pixel prediction approach might lead to increased computational demands.\n3. The performance might be highly sensitive to the tuning of loss function hyper-parameters.\n4. There's a lack of in-depth comparative analysis with traditional anchor-based models.\n5. The model's efficiency in handling images with multiple overlapping objects is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uBHBaocaSG", "reviewer": "Reviewer_vvPE", "summary": "This article presents a method for correlating part to body relationships in a single multi-person RGB image. The method is a single-stage approach that adopts the popular center-based detection architecture. To correlate a subject and multiple detected body parts, a common approach is to estimate center offsets from each body center to the underlying body part. But this may be ineffective since in case of occlusion/close interaction multiple body centers may point to the same body part. Therefore, this paper proposes to simultaneously estimate the offset of the body part position from the detected body part toward the center of the subject. This means that instead of estimating center-to-part offsets, estimating part-to-center offsets for each detected body part will avoid ambiguities in close interaction situations. The proposed method greatly improves the performance of hand association in BodyHands.", "questions": "1. Will the proposed method be open-sourced? It seems hard to re-implement the results with limited details.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1. BodyHands performance improvements are impressive with simple associative design changes. Changing from center-to-part to part-to-center offset greatly alleviates the ambiguity of hand associations in training.\n\n2. Interesting insights. The proposed method can achieve significant performance improvements in hand association on BodyHands. This demonstrates the interesting insight that the position of the hand is more ambiguous than the position of the center of the body. Because more occlusion/interaction occurs in the hand area.", "weaknesses": "1. Writings. It is important to clearly present ideas and implementation. The current presentation is very vague and difficult to understand. It is beneficial to emphasize conceptual-level differences in the introduction. But before that, the paper should at least introduce the method/idea clearly. Additionally, in the Methods and related work section, consider clearly presenting differences compared to previous methods.\n\n2. In Figure 2, please highlight the parts where the proposed approach makes design changes. In Fig. 1, the direction of arrows are not very obvious. Besides, differences in qualitative results are difficult to distinguish. Please consider emphasizing it further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9AZpbVRwzW", "reviewer": "Reviewer_iAuQ", "summary": "This paper propose an association method for part-body detection. Their framework is based on YOLO, which is anchor-free.  In experiments, they show PBADet's good performance.", "questions": "Refer to the weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is easy to read.  \n2. Association for body and parts is  worth studying.  \n3. The method is evaluated in several datasets.", "weaknesses": "1.The main problem of this paper is that part-to-body offsets have been proposed in previous works [1]. Furthermore，in [1], they also try local center offsets to improve the performance. Hence, considering the not new idea and no other designs, the technical learning of this paper is limitted.  \n[1] Jin L, Wang XJ. Grouping by Center: Predicting Centripetal Offsets for the Bottom-up Human Pose Estimation. TMM, 2022.  \n2. \"Single-stage\" is controverial. This method is bottom-up but still need two-stage processing, i.e., detection and grouping. And single-stage method has been studied in several reaseraches [2]. There are  obvious differences between single-stage methods and this paper.  \n[2] Single-Stage Multi-Person Pose Machines, ICCV, 2019.  \n3. To verify the association efficiency, author should compare your association method with other grouping methods, such as associative embeddings and methods in [1], under the same part detection accuracy.    \n[3] Associative Embedding: End-to-End Learning for Joint Detection and Grouping.  \n4. The focus of this paper is association, but the evaluation metric is AP. More metrics on association quality should be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)