Adaptive Hierarchical Aggregation for Federated Object Detection

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 03:46:29

Abstract: In practical object detection scenarios, distributed data and stringent privacy protections significantly limit the feasibility of traditional centralized training methods. Federated learning (FL) emerges as a promising solution to this dilemma. Nonetheless, the issue of data heterogeneity introduces distinct challenges to federated object detection, evident in diminished object perception, classification, and localization abilities. In response, we introduce a task-driven federated learning methodology, dubbed Adaptive Hierarchical Aggregation (FedAHA), tailored to overcome these obstacles. Our algorithm unfolds in two strategic phases from shallow-to-deep layers: (1) Structure-aware Aggregation (SAA) aligns feature extractors during the aggregation phase, thus bolstering the global model's object perception capabilities; (2) Convex Semantic Calibration (CSC) leverages convex function theory to average semantic features instead of model parameters, enhancing the global model's classification and localization precision.  We demonstrate experimentally and theoretically the effectiveness of the proposed two modules respectively. Our method consistently outperforming the state-of-the-art methods across multiple valuable application scenarios. Moreover, we build a real FL system using Raspberry Pis to demonstrate that our approach achieves a good trade-off between performance and efficiency.

Main paper: 
 related work 2.1 object detection: 
 Object detection has always been a critically important and challenging task in the field of computer vision. In recent years, the rapid development of deep learning techniques has greatly promoted the progress of object detection. Initially, deep learning-based object detection networks were two-stage, such as R-CNN [9] and its variants, Fast R-CNN [8] , and Faster R-CNN [36] . Subsequently, end-to-end one-stage networks were proposed. Thanks to their fast detection speed and ease of deployment, they have become the mainstream algorithms in object detection, receiving widespread attention from both the academic and industrial communities. YOLO (You Only Look Once) [34] is a typical representative among them.It was the first one-stage detector in the deep learning era. YOLOv5 [17] , as a successor to YOLO, has become one of the most popular object detection networks due to its high accuracy and speed. Recently, YOLOv7 [41] , a follow-up work from the YOLOv4 [2] team, has been proposed. It outperforms most existing object detectors in terms of speed and accuracy. In addition, with the growing popularity of Transformer, a Transformer-based object detection network, DETR [3] , has been proposed. Transformer discards the traditional convolution operator in favor of attention-alone calculation in order to overcome the limitations of CNNs and obtain a global-scale receptive field. RT-DETR [32] , a real-time end-to-end detector based on the DETR architecture, achieves state-of-the-art (SOTA) performance in both speed and accuracy.McMahan [33] Currently, there has been some preliminary exploration in federated object detection [14, 29, 30, 39, 43] . FedVision [29] proposed an engineering platform to support the development of federated learning powered computer vision applications, which is the first real application of FL in computer vision-based tasks. Hegiste et al. [14] proposed a FL algorithm for object detection in quality inspection tasks using YOLOv5. However, both of them merely integrate the FedAvg algorithm with object detection tasks, lacking attention to the issue of heterogeneity. Yu et al. [43] improved Fe-dAvg based Abnormal Weights Supression, reducing the influence of the weights divergence caused by non-IID and unbalanced data. However, the method of discarding abnormal weights leads to the global model losing some useful information, showing a lack of in-depth solution to the heterogeneity issue. Lu et al. [30] and Su et al. [39] investigated the issue of cross-domain adaptation in object detection. However, the essence of cross-domain problem still lies in data heterogeneity, and they focused solely on the scenario of autonomous driving, neglecting the algorithm's generalizability.OptimizeIn summary, despite the rich progress made in the field of federated learning within computer vision, there is a lack of effective and in-depth solutions to the issue of heterogeneity in the domain of object detection. Compared to related work, we conduct an indepth analysis of the specific challenges that heterogeneity poses to object detection, and subsequently, we propose targeted solutions to address these challenges. Additionally, supported by rigorous experiments and theoretical underpinnings, our method exhibits better performance and generalizability. 
method 3.1 problem setup: 
 Under the standard federated learning setting, there are 𝐾 clients participating in training, alongside a server that coordinates the aggregation process. Each client trains a local model 𝜃 𝑘 using their private dataset 𝐷 𝑘 = {(𝑥 𝑖 𝑘 , 𝑦 𝑖 𝑘 )} 𝑁 𝑘 𝑖=1 , where 𝑁 𝑘 denotes the size of the 𝑘 𝑡ℎ local dataset, and (𝑥 𝑖 𝑘 , 𝑦 𝑖 𝑘 ) represents the inputs and labels, respectively. After every 𝐸 epochs of local training, all the client models will be transmitted to the central server and aggregated to form a global model. Our objective is to develop a global model with a generalized representation, which can be formulated as follows:EQUATIONwhere 𝑁 = 𝑘 ∈𝐾 𝑁 𝑘 , and 𝑤 denotes the parameters of the global model, 𝐿 𝑘 is the loss function of the 𝑘 𝑡ℎ client. In real-world scenarios, private datasets are often heterogeneous, primarily characterized by two factors: an uneven distribution of categories and disparities in data quantities. Given that the server cannot directly access client-side data, to effectively mitigate this issue, similar to FedMD [21] , FedDF [25] and RHFL [7] , we deploy an unlabeled public dataset 𝐷 0 = {𝑥 𝑖 } 𝑁 0 𝑖=1 on the server side as a bridge for communication between client models. The public dataset facilitates the implementation of our algorithm, and in practical scenarios, it can be easily acquired through publicly available datasets, thereby not introducing additional privacy risks. Moreover, we decompose the detection model parameterized by 𝑤 = {𝜃, 𝜙 } into a feature extractor and a detection head. Specifically, the feature extractor extracts shallow texture features of the image and the detection head, given the texture maps, further extracts semantic information and For an in-depth study of the detection task, we visualize the process of the feature extractor extracting texture features and compare it with the classification task, as illustrated in Figure 3 . Specifically, for the classification network, we select ResNet50, which boasts a accuracy of 75.99% on the ImageNet dataset. And for the detection network, we use YOLOv5s, which achieves a 𝑚𝐴𝑃 50 precision of 56.8% on the MS COCO dataset. Both networks are thoroughly trained, and the feature extractor employ the same residual network structure. Through the results of visualization, we note that the detection network encompasses more detailed structured image information. This is because detection networks need to detect as fully objects as possible, rather than simply outputting a single result like in classification tasks, which is exactly the unique challenge for the object detection. Unfortunately, in federated learning with data heterogeneity, this challenge is amplified. The feature extractors of heterogeneous client models lead to instability in the aggregation process. To tackle this challenge, taking into account the characteristics of object detection networks, we introduce the Structure-aware Aggregation. The purpose of the module is to adaptively assign weights to each client model considering their feature extractors, to achieve consistency in model aggregation and mitigate the impacts arising from heterogeneity issues. Inspired by SSIM [42] , through combining structural coefficients and informational measures, we implement the measurement of the consistency of feature extractors:EQUATIONHere, 𝑡 represents the current round of communication, and 𝑀 (𝜃 𝑡 𝑘 ) represents the measure of consistency in feature extraction between the 𝑘 𝑡ℎ client model and the global model, which, in other words, is positively correlated with its corresponding weight.By comparing the feature maps extracted by client models with those of the global model, we can obtain the structural coefficients 𝑆 (𝜃 𝑡 𝑘 ) and informational measures 𝐼 (𝜃 𝑡 𝑘 ), respectively. Suppose x and y are two non-negative feature signals from the global and client models, respectively, which have been aligned with each other (e.g., spatial patches extracted from each feature maps). We associate the two normalized vectors (x -𝜇 𝑥 )/𝜎 𝑥 and (y -𝜇 𝑦 )/𝜎 𝑦 , with the structure of two images, where 𝜇 and 𝜎 respectively represent the mean and standard deviation. The correlation (inner product) between these is a simple and effective measure to quantify the structural similarity. Notice that the correlation between (x -𝜇 𝑥 )/𝜎 𝑥 and (y -𝜇 𝑦 )/𝜎 𝑦 is equivalent to the correlation coefficient between x and y. Thus we design the structural coefficients as follows:EQUATIONwhere the small constant 𝜖 is introduced to avoid division by zero scenarios, specially we set 𝜖 = 0.01. 𝜎 𝑥 𝑦 is the covariance of x and y, which can be estimated:EQUATIONAs for information measure, naturally, we think of utilizing mutual information (MI) in information theory, which quantifies the amount of information one variable provides about another. The MI between x and y can be calculated as:EQUATIONwhere 𝑝 (x) and 𝑝 (y) are the marginal probability distributions of x and y, and 𝑝 (x, y) is their joint probability distribution. By combining the previously obtained structural coefficients and informational measures, we arrive at the final 𝑀 (𝜃 𝑘 ):EQUATIONConsidering the contribution of the number of client datasets to model aggregation, we redefine the aggregation process as follows:EQUATIONEQUATIONTo improve the model's precision in classification and localization, we analyze the problem from the perspective of convex optimization theory and introduce a semantic calibration module.Beyond perceiving objects, the model also needs to process semantic information such as categories and localization. In federated learning, heterogeneous client datasets often lead client models into different local convex domains, which can be formulated as:EQUATIONHowever, since DNN is indeed a highly non-convex model, the weighted arithmetic average of 𝐾 local optima does not yield good results, and even the global model might perform even worse than the individual client models. Nevertheless, the loss functions we commonly use, such as cross entropy loss, square loss, and hinge loss, are convex. We will demonstrate the convex properties of the loss function in object detection models in the next section to provide support for our method. Therefore, averaging the semantic information output by client models, rather than their parameters, can guarantee that the global model's performance is at least as good as their average performance, as shown in Figure 4 . This is because, when 𝐿(•) is a convex function:EQUATIONEQUATIONwhere 𝐹 (𝜃 ; 𝑥) represents the semantic information output by the model. To this end, we propose to ensemble the client models from the perspective of semantic information. In other words, we aim for the semantic information output by the global model to be as close as possible to the average of the semantic information output by the client models:EQUATIONEQUATIONIn order to achieve the effect mentioned above, we design the convex semantic (CS) loss to constrain the semantic information output by the global model as follows:EQUATIONwhere || • || 𝐹 is Frobenius norm. During the aggregation process, the update of the global model can be denoted as:EQUATIONwhere 𝛼 represents the learning rate. In this section, we analyze the loss function in object detection to verify convex function property. Specifically, in object detection models, the loss function generally comprises three parts:EQUATIONHere, Equation14 contains confidence loss 𝐿 𝑜𝑏 𝑗 (judging whether there is an object), classification loss 𝐿 𝑐𝑙𝑠 , and location loss 𝐿 𝑙𝑜𝑐 , and the weights 𝜆 𝑜𝑏 𝑗 , 𝜆 𝑐𝑙𝑠 , and 𝜆 𝑙𝑜𝑐 regulate error emphasis among box coordinates, box dimensions, objectness, no-objectness and classification. We will analyze these three parts separately.Confidence Loss assesses the accuracy of the model's prediction regarding the presence of an object within the predicted bounding box. For each predicted box, the model provides a probability prediction of object presence. Since this is a binary classification problem, the commonly used loss function is the cross-entropy:EQUATIONHere, 𝑦 is the true label, typically 0 or 1, and 𝑝 is the probability of being predicted as the positive class, ranging between 0 and 1. In the binary cross-entropy function, for some fixed 𝑦, the second-order partial derivative with respect to 𝑝 is:EQUATION)This derivative is always positive within the domain of 𝑝, indicating that the function is convex with respect to 𝑝.Classification loss assesses the model's ability to correctly classify objects within bounding boxes. This part of the loss focuses on the capability to accurately label detected objects. The multiclass problem is an extension of the binary classification problem, so the loss function here is quite similar to the previously mentioned confidence loss:EQUATIONwhich 𝑀 represents the number of categories. Therefore, 𝐿 𝑐𝑙𝑠 can be expressed as the sum of 𝑀 convex functions. Assuming 𝑓 1 and 𝑓 2 are two convex functions and 𝑓 = 𝑓 1 + 𝑓 2 , ∀𝜃 ∈ (0, 1) based on MM'24, October 28 -November 1, 2024, Melbourne, Australia. Anonymous Authors the properties of convex functions, we can conclude that:EQUATIONwhich can be easily extended to the sum of 𝑀 functions. This indicates that 𝐿 𝑐𝑙𝑠 , a function composed of 𝑀 convex functions, is also convex.Location loss is used to measure the difference between the predicted bounding boxes and the actual bounding boxes. It includes IoU loss, which measures the degree of overlap between the predicted and actual boxes, and L2 loss, used to minimize the squared error between the coordinates of the predicted and actual boxes. L2 loss is a standard convex function, so we will not expand on its analysis here. Instead, we focus our study on the IoU loss.In the location process, the output of the object detection model includes four normalized parameters, which we denote as x, ỹ, w and h. Here, x and ỹ represent the center coordinates of the predicted bounding box, while w and h represent the width and height, respectively. Inspired by previous research work on IoU loss [37, 46] , we decompose the localization problem into two fine-tuning subproblems: translation and scaling.For the first sub-problem, we focus only on the impact of the outputs w and h. Considering that client models in federated learning converge to local optima, it is ensured that there will be a significant overlap between the predicted bounding boxes and the ground truth. Therefore, we approximate the IoU as the ratio of the area of the two of them, and the IoU is adjusted only within a small range. Next, we need to analyze the two different expressions of IoU separately. When the predicted bounding box is slightly larger than the ground truth, we simplify the loss function based on this assumption as follows:EQUATIONHere, 𝑤 and ℎ represent the width and height of the bounding box in the ground truth, respectively. Assume 𝑝 1 = ( w1 , h1 ) and 𝑝 2 = ( w2 , h2 ) are two points in its domain of definition, and the latter's box is larger than the former's, i.e., w2 > w1 , h2 > h1 .Ignoring terms of third order and higher regarding 𝜃 and (1 -𝜃 ), we can obtain:EQUATION)And when the predicted bounding box is slightly smaller than the ground truth, the loss function can be written as:EQUATION)Following Equation20, we can obtain:EQUATION)Up to this point, we have proven the convex property of the loss function in the first sub-problem under two scenarios.For the second sub-problem, we consider the prediction of the bounding box center, focusing solely on the impact of the outputs x and ỹ. For convenience of description, we denote the offset between the predicted coordinates ( x, ỹ) and the ground truth (𝑥, 𝑦) as (Δ𝑥, Δ𝑦), then we have:EQUATION)Further, we can calculate its Hessian matrix:EQUATIONwhich is a positive-definite matrix. Overall, although the IoU loss is not strictly a convex function, we have demonstrated that it exhibits properties akin to those of a convex function within the context of the problem we considered. 
experiments 4.1 experimental setting: 
 Datasets and Models. We extensively evaluate our method with three public datasets: VOC 2007 [6] , NWPU VHR-10 [4], BCCD [38] . These three datasets originate respectively from the realms of natural imagery, remote sensing imagery, and medical imagery, each representing valuable federated learning application scenario. In partitioning heterogeneous datasets, we refer to the commonly used partitioning methods [23] within the federated learning community and combined them with the characteristics and application scenarios specific to each dataset. Specifically, for the natural imagery VOC 2007, we adopt the 'label distribution skew' partitioning method. This involves partitioning data for each client based on the labels of the samples. With 𝐾 clients and 𝑀 data categories, each client is allocated 𝑀/𝐾 categories, ensuring no overlap of samples among different clients. For the remote sensing imagery NWPU VHR-10, we combine 'feature distribution skew' and 'quantity skew' for data partitioning. Initially, images are categorized and ordered based on the scene of capture (such as airports, industrial areas, residential areas, green zones, oceans, etc.). Subsequently, we introduce the Dirichlet distribution for quantity partitioning, allowing us to flexibly alter the level of imbalance by adjusting the parameter 𝛽. For the medical imagery BCCD, we implement the 'quantity skew' partitioning method using the Dirichlet distribution. In addition, we conduct experiments in two levels of heterogeneous scenarios, setting the number of clients 𝐾, to 4 and 10, respectively, and the parameter 𝛽 in the Dirichlet distribution to 0.5 and 0.1, respectively. Finally, for the models, we select the YOLOv5s and RT-DETER-L, hereinafter abbreviated respectively as YOLOv5 and RT-DETR.Evaluation Metrics. We use mean Average Precision (mAP) to evaluate the detection performance. The mAP is a comprehensive indicator obtained by averaging AP values, which uses an integral method to calculate the area enclosed by the precision-recall curve and coordinate axis of all categories. The mAP can be calculated as:EQUATIONwhere 𝑝 denotes Precision, 𝑟 denotes Recall, and 𝑀 is the number of categories. Typically, the mAP is calculated as the average at a specific Intersection over Union (IoU) threshold. Consequently, we employed two widely used thresholding approaches to calculate the mAP: mAP 50 represents for IoU=0.5 and mAP 50:95 for increasing IoU threshold values, from 0.5 to 0.95 by 0.05. Implementation Details. Our proposed approach is implemented in PyTorch and runs on a workstation with four NVIDIA 3090 GPUs, using PyTorch-1.8, running Ubuntu 18.04. In local learning process, all clients adopt the same hyper-parameter setting. The input image is resized to 512×512. We use the SGD optimizer with an initial learning rate of 𝑙𝑟 = 0.01 and the batch size of 256. For each communication round, each client locally trains for 20 epochs. During the local training, we use linear learning rate decay, and after 20 epochs training, the learning rate decays to the original 0.1.State-of-the-Art Methods. In order to prove the effectiveness of our FedAHA, we compare FedAHA with the heterogeneous FL algorithm: FedProx [24] , SCAFFOLD [19] , FedDF [25] , FedAvg(AWS) [43] and FedDAD [30] . Among them, FedProx, SCAFFOLD, and FedDF are classic algorithms targeted at data heterogeneity, while FedAvg(AWS) and FedDAD are federated object detection algorithms designed to mitigate issues of heterogeneity. It is noteworthy that both FedDF and our method incorporate a public dataset within the framework. To ensure a fair comparison, we set aside 10% of the training data from the clients to form a public dataset on the server, thereby guaranteeing that the total amount of data used across all methods is consistent.Effect of Components. Table 1&2 present the ablation results of our proposed components. Evidently, the SAA and CSC modules each independently contribute to improving the model's performance to a certain extent. A noteworthy observation is that, typically, the use of SAA alone tends to yield a greater enhancement in performance compared to the exclusive use of CSC. This suggests that SAA plays a more fundamental and critical role, aligning with our original design intentions. However, the contribution of the CSC module cannot be overlooked. It fine-tunes the model's classification and localization capabilities, which is reflected in the mAP 50:95 metric. Furthermore, when both modules operate in tandem, they lead to a significant improvement. For instance, when 𝐾 = 10, for the VOC 2007 dataset, the combined action of both modules resulted in a 6.51% increase in mAP 50 , which far exceeds the 3.02%, 1.77% enhancement observed with either module working alone. Thus, the SAA and CSC complement each other effectively. Impact of Heterogeneity. As shown in the Table 1&3 , our method can achieve great improvement under different heterogeneity rates, and the improvement will be more obvious at high heterogeneity rates, especially on the VOC 2007 dataset. When the number of clients is four, our method improves on the original baseline by 2.27% mAP 50 score and 2.81% mAP 50 score for YOLOv5 and RT-DETR, respectively. When the number of clients increased to ten, our method results in a 6.12% and 6.51% mAP 50 enhancement over the baseline for YOLOv5s and RT-DETR, respectively. The addition of the SAA component makes the knowledge of relatively noisy clients be learned less in the process of collaborative learning, so our approach demonstrates more significant effectiveness. This demonstrates that the algorithm we proposed is highly effective in mitigating issues related to heterogeneity. Performance Analysis. We provide comparison results with stateof-the-art methods on two models with three types of data. The comparisons on two heterogeneity rates (𝐾 = 4, 10) are shown in Table 2&4 . Experimental results indicate that our method outperforms existing approaches in most performance metrics under various degrees of heterogeneity. Particularly in scenarios with stronger heterogeneity, our method surpasses other algorithms by a greater margin. Robustness Analysis. Thanks to its task-driven design, our approach has shown both strong model generalization capabilities and dataset generalization abilities. Specifically, our method provides strong support for two mainstream models: the CNN-based YOLOv5 and the Transformer-based RT-DETER. Moreover, our method is capable of handling a variety of datasets, including natural images, remote sensing images, and medical images, which vary significantly in style.Efficiency Analysis. In practical scenarios, the computational capability of clients are typically quite limited. In contrast, servers possess robust computational resource. Therefore, to enhance efficiency in the federated learning process, it is advisable to shift computational overhead to the server as much as possible. Aiming to authentically evaluate the runtime of each algorithm, we utilize the FedML [12] open-source library to establish a smallscale federated learning system composed of Raspberry Pis and Compared to algorithms like FedProx, SCAF-FOLD, and FedDAD, which introduce additional computational or communication overhead on clients, our method exhibits a clear efficiency advantage. For FedDF and FedAvg(AWS), our method can also achieve comparable efficiency performance in the case of significant performance advantages. In summary, our approach achieves a good trade-off between performance and efficiency. 
conclusion: 
 In this paper, we introduce FedAHA, a pioneering federated learning algorithm specifically designed for object detection within heterogeneous scenarios. To address the distinctive challenges encountered in federated object detection, we have meticulously developed two modules: SAA and CSC. Our approach is thoroughly backed by comprehensive experimental analyses and robust theoretical proofs. Extensive testing demonstrates that our method not only outperforms current state-of-the-art methods but also achieves an exceptional balance between performance and efficiency. We are confident that our contributions will significantly broaden the scope of federated learning applications in object detection. 
: 
 Federated learning (FL), a distributed machine learning paradigm that eliminates the need for data transfer, effectively addresses the limitations mentioned above. It was introduced by McMahan et al. along with the FedAvg [33] algorithm and has been extensively explored in the field of computer vision [15, 31, 40, 44, 48] . FL has been an active and challenging research topic and shows promising results in real-world setting [10, 11, 18, 27] . Along with its pilot progress, researches on federated learning are baffled by the key challenge: the issue of data heterogeneity. To address this challenge, some federated learning algorithms based on improvements to FedAvg have been proposed, such as FedProx [24] , SCAFFOLD [19] and FedDF [25] . However, these proposed methods have been validated on classification tasks. Unfortunately, federated object detection tasks have received relatively less attention.Compared to classification tasks, detection tasks face unique challenges when dealing with the issue of data heterogeneity. On one hand, data heterogeneity reduces the global model's ability to perceive objectives. Our experimental analysis has shown that, under task orientation, detection models extract more detailed structural information, which represents the biggest difference from classification models in their patterns of image feature extraction. In FL, heterogeneous data impacts the performance of feature extractors, and further diminishes the global model's capacity to perceive objects. This aspect has been overlooked by many previous researches. On the other hand, the issue of data heterogeneity reduces the model's accuracy in object classification and localization. Beyond perceiving objects, detection networks must not only classify them but also accomplish localization. However, existing federated algorithms lack consideration for localization accuracy. Thus, achieving satisfactory performance in federated object detection remains a challenge. We further explain these challenges in Figure 1 .To tackle the two aforementioned challenges, we propose our efficient federated learning method tailored for object detection with adaptive hierarchical aggregation: 1) Hierarchical aggregation from shallow-to-deep. We divide the model into two parts: the feature extractor and the detection head. For each part, we design targeted aggregation strategies, achieving hierarchical aggregation from shallow to deep layers. Our method achieves a thorough integration from texture features to semantic features, effectively addressing the two challenges posed by heterogeneity in federated object detection. 2) Structure-aware Aggregation. Due to the drift phenomenon [19] between different feature extractors from client models, which leads to a decline in the common model's ability to perceive objects, we introduce a new weighting approach, namely Structure-aware Aggregation, to enhance the consistency of aggregation. Based on image processing and information theory, by combining structural coefficients and informational measures, we design a metric to gauge the consistency among different feature extractors. Through dynamically assigning weights to different client feature extractors based on the consistency measure, our method ensures the stability of the aggregation process, thereby enhancing the global model's ability to perceive objects. 3) Convex Semantic Calibration. To achieve accurate classification and localization, we need to accomplish high-precision semantic aggregation. Inspired by convex function theory, our focus shifts to averaging the semantic outcomes of the detection head, rather than the parameter averages. Simultaneously, we conduct a detailed analysis of the loss function in object detection, proving that our method is supported by solid and reliable theoretical foundations in mathematics. This strategy not only achieves semantic alignment but also effectively circumvents the problem of client models converging towards diverse local-convex domains, thereby enhancing the performance and robustness of the global model. The main contributions of our paper are as follows:• Starting from a task-driven perspective, we investigate the unique challenges posed by data heterogeneity in federated learning to object detection tasks. We propose an adaptive hierarchical federated aggregation algorithm that progresses from shallow-to-deep. • We construct a real federated learning system using Raspberry Pi to authentically verify that our algorithm, by shifting computational overhead to the server side, achieves an excellent trade-off between performance and efficiency. 
 
 Appendix:  


Decision: 3 BA and 1 WA, this paper can be accepted as poster.

Reviews: review: paper_summary: paper_summary: The paper presents a federated learning methodology aimed at addressing the challenges posed by data heterogeneity in federated object detection tasks. The proposed approach, termed Adaptive Hierarchical Aggregation (FedAHA), involves two main phases: Structure-aware Aggregation (SAA) and Convex Semantic Calibration (CSC). The SAA phase aligns feature extractors to enhance the global model's object perception capabilities, while the CSC phase leverages convex function theory to average semantic features, improving the global model's classification and localization precision. The authors demonstrate the effectiveness of these modules through theoretical validation and extensive experiments.
strength_weakness: strength_weakness: Clarity: The paper is well-structured and clearly explains the motivation, methodology, and results. The inclusion of detailed visualizations and ablation studies enhances the comprehensibility of the proposed approach.
Theoretical and Technical Correctness: The use of convex function theory in the CSC module is well-grounded and provides a solid mathematical basis for improving semantic feature aggregation. The theoretical validation of the proposed methods adds robustness to the claims.
limitations: limitations: Lack of Novelty in Baseline Comparison: While the proposed method shows improvements, the baselines used for comparison (e.g., FedAvg, FedProx) are relatively standard. It would strengthen the paper to include comparisons with more recent and advanced federated learning techniques specifically designed for object detection.
Limited Analysis of Computational Efficiency: Although the paper mentions efficiency and even implemented a real federated learning system using Raspberry Pis, a more detailed analysis of the computational cost and scalability of the proposed method compared to other state-of-the-art methods would be valuable. This includes the impact on training time and resource utilization in real-world federated learning deployments.
Logical Flow in Introduction: The introduction section mentions the two challenges of applying federated learning (FL) to object detection, but the logic is unclear. It is recommended that the authors first introduce the key factors affecting the performance of object detection before introducing FL. The impact of data heterogeneity in FL on object detection performance is not clearly articulated.
paper_summary: paper_summary: The paper introduces a task-driven federated learning methodology,  dubbed Adaptive Hierarchical Aggregation (FedAHA), aimed at the task of target detection in distributed data and strict privacy-preserving scenarios. The algorithm unfolds in two strategic phases from shallow-to-deep layers: (1) Structure-aware Aggregation (SAA) aligns feature extractors during the aggregation phase, thus bolstering the global model’s object perception capabilities; (2) Convex Semantic Calibration (CSC) leverages convex function theory to average semantic features instead of model parameters, enhancing the global model’s classification and localization precision. The proposed method consistently outperforms the state-of-the-art methods across multiple valuable application scenarios.
strength_weakness: strength_weakness: (1)	The proposed algorithm is intuitive and easy to understand and use.

(2)	The proposed algorithm solves the data heterogeneity problem in federated scenarios and enhances the global model’s classification and localization precision.

(3)	The proposed method achieves superior performance over the state-of-the-art algorithms, and ablation study on core modules validates the efficacy.
limitations: limitations: (1)	The authors set aside 10% of the training data from the client to form a public dataset, which has privacy leakage issues.

(2)	The aim of the article is to overcome the existing challenges in terms of object perception, classification, and localization abilities in data heterogeneous environments. mAP is the only metric used in the experiments, which is insufficient to evaluate the object perception, classification, and localization capabilities of the model. I suggest the authors to add indicators that are specific to the evaluation of classification or positioning.

(3)	Lacking comparison with the latest baselines. Among the chosen algorithms, only FedDAD was published in 2023, while the rest of the baselines were proposed in 2020 or earlier.
paper_summary: paper_summary: The paper presents "Adaptive Hierarchical Aggregation" (FedAHA), a novel approach to address challenges in federated object detection caused by data heterogeneity. The approach comprises two main strategies: Structure-aware Aggregation (SAA), which aligns feature extractors to improve object perception, and Convex Semantic Calibration (CSC), which uses convex function theory to enhance classification and localization accuracy. The method involves a hierarchical aggregation process from shallow to deep network layers. The effectiveness of this approach is demonstrated through experiments across different datasets, and the paper includes a practical implementation on Raspberry Pi devices to showcase real-world applicability.
strength_weakness: strength_weakness: 1. Novel Methodology: The hierarchical aggregation approach is innovative, addressing specific challenges in federated object detection like feature misalignment and semantic inconsistency due to data heterogeneity.

2. Practical Implementation: The implementation on low-resource devices like Raspberry Pi and the adjustments for computational efficiency make this research highly relevant for real-world applications, particularly in edge computing environments.

3. Theoretical Foundation: The paper provides a solid theoretical foundation for the methodologies used, particularly the application of convex function theory in CSC, which is well-explained and justified.
limitations: limitations: 1. Complexity and Resource Demand: While the paper addresses computational efficiency, the hierarchical and dual-phase nature of the method might still impose significant computational demands, particularly on very large federated networks.

2. Generalization to Other Domains: The experiments are limited to object detection; thus, the generalizability of the approach to other domains or more complex tasks within federated learning remains uncertain.

3. Dependence on Accurate Domain Characterization: The success of the method hinges significantly on the accurate characterization of data heterogeneity and appropriate configuration of the aggregation strategies, which might be challenging in highly dynamic environments.
paper_summary: paper_summary: This paper introduced a task-driven federated learning method called  Adaptive Hierarchical Aggregation (FedAHA) to solve data heterogeneity, non-iid issues for object detection.  To this end, the FedAHA algorithm developed two new techniques namely Structure-aware Aggregation and Convex Semantic Calibration. SAA aligns feature extractors during the aggregation phase, thus bolstering the global model's object perception capabilities. While, CSC leverages convex function theory to average semantic features instead of model parameters, enhancing the global model's classification and localization precision. FedAHA consistently outperforms the state-of-the-art methods across multiple valuable application scenarios from 2.26% to 7.61%.
strength_weakness: strength_weakness: The paper solves an interesting problem of developing an FL algorithm that can work on object detection tasks in non-iid scenarios.  Object detection is often an overlooked task in FL settings. 

The paper is well-written and structured. 

The proposed framework is implemented on Raspberry Pi devices. 

The two techniques named SAA and CSC are novel and are supported by strong theoretical foundations.  

The choice of the datasets, models, and baselines seems appropriate.
limitations: limitations: The requirement to have a public dataset is a major limitation. 

The performance of FedAHA is not evaluated on the Coco dataset. 

It is unclear why evaluation is done on 4 and 10 clients. This also questions the scalability of FedAHA. How will it scale with 50 and 100 clients? 

The reason for FedDF outperforming FedAHA on NWPU VHR-10 in Table 2 and SCAFFOLD in Table 4 is not explained. 

Section 4.3 on efficient analysis is missing many details. For example, which Raspberry PI model was evaluated? How did you calculate the normalized training time? How are these devices communicating with the server? How much run-time memory do the clients consume?