Navigating Data Heterogeneity in Federated Learning: A Semi-Supervised Federated Object Detection
Authors: Taehyeon Kim
Eric Lin
Junu Lee
Christian Lau
Vaikkunth Mugunthan
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Semi-Supervised Learning
Object Detection
TL;DR: Introducing FedSTO within a Semi-Supervised Federated Object Detection framework that effectively overcomes non-IID data shift from labeled server and unlabeled client data, achieving competitive performance with only 20-30% of labels

Abstract: Federated Learning (FL) has emerged as a potent framework for training models across distributed data sources while maintaining data privacy. Nevertheless, it faces challenges with limited high-quality labels and non-IID client data, particularly in applications like autonomous driving. To address these hurdles, we navigate the uncharted waters of Semi-Supervised Federated Object Detection (SSFOD). We present a pioneering SSFOD framework, designed for scenarios where labeled data reside only at the server while clients possess unlabeled data. Notably, our method represents the inaugural implementation of SSFOD for clients with 0% labeled non-IID data, a stark contrast to previous studies that maintain some subset of labels at each client. We propose FedSTO, a two-stage strategy encompassing Selective Training followed by Orthogonally enhanced full-parameter training, to effectively address data shift (e.g. weather conditions) between server and clients. Our contributions include selectively refining the backbone of the detector to avert overfitting, orthogonality regularization to boost representation divergence, and local EMA-driven pseudo label assignment to yield high-quality pseudo labels. Extensive validation on prominent autonomous driving datasets (BDD100K, Cityscapes, and SODA10M) attests to the efficacy of our approach, demonstrating state-of-the-art results. Remarkably, FedSTO, using just 20-30% of labels, performs nearly as well as fully-supervised centralized training methods.

Navigating Data Heterogeneity in Federated Learning: A Semi-Supervised Federated Object Detection
Taehyeon Kim!
Eric Lin KAISS
Junu Lee3
Christian Lau? Vaikkunth Mugunthan" 2
DynamoFF
'The Wharton School potter32ekaist.co ac. kr
Abstract
Federated Learning (FL) has emerged potent framework for training models across distributed data sources while maintaining data privacy. Nevertheless, it faces challenges with limited high-quality labels and non-IID client data, par- ticularly in applications like autonomous driving. To address these hurdles, we navigate the uncharted waters of Semi- Suni--upeeeieed Federated Object Detection (SSFOD). We present pioneering SSFOD framework, designed for scenarios where labeled data reside only at the server while clients possess unlabeled data. Notably, our method represents the inaugural implementation of SSFOD for clients with 0% labeled non-lID data, a stark contrast to previous studies that maintain subset of labels at each client. We propose FedSTO, two-stage strategy compassing Selective Training followed by Orthogonally enhanced full tull-parameter training, to effectively address data shift (e.g. weather conditions) between server and clients. Our contributions include selectively refining the backbone of the detector to avert overfitting, orthogonality regularization to boost representation divergence, and local EMA-driven pseudo label assignment to yield high-quality pseudo labels. Extensive validation on prominent autonomous driving datasets (BDD100K, Cityscapes, and SODA 10M) attests to the efficacy of our approach, demonstrating tf-te-of-the-art results. Remarkably, FedSTO, using just 20-30% of labels, performs nearly as well as fully-supervised centralized training methods.
1 Introduction
Federated Learning (FL. enables decentralized training across distributed data sources, preserving data privacy (27] It has emerged in response to the need for privacy, security, and regulatory compliance such as GDPR (36] and CCPA (31] FI. trains models on local devices and shares only model updates, thereby improving privacy and efficiency. In typical FL cycle, each client updates a shared model with local data, sends the updates to a server for parameter aggregation, and then updates its local model with the newly aggregated global model sent back by the server.
Despite the potential of FL. the assumption fully labeled data restricts its practicality 12 In order acquire high-quality labels, data is often transmitted from edge clients to central server, thereby compromising the privacy assurances provided by FL. Limited labels at the edge necessitate the adoption of transfer learning, self-supervised learning, and semi-supervised learning (SSL) techniques. However, the separation of labeled and unlabeled data complicates the application of these techniques FI., which undermine the system's effectiveness. This issue is amplified in labels-at-server scenarios where only the server possesses labeled data, and clients hold only unlabeled data 10 43 [8 42). In autonomous driving, a novel approach is required to bridge the knowledge gap between labeled and unlabeled data without the need for direct data exchange.
While Supi-SSper Federated Learning (SSFL) has been explored for image classification I8 42 these studies have faced the following challenges:
tasks, 5 10 43
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Server
Upload updated Backbone
Client 1, ....,
Teacher
Unlabrled Lecal
Pseudo Label
Labeled Local Datn
Backbone
Neck Head
iil
EMA Updaate
A.tsa
2393
BackBone Neck
Backbone
Neck
Head
Update
Freeze
Broadcast entire model
Representation Learning with Selective Training
Orthogonal Enhancement
Server
Orthogonal Regularization on Nock & Head
Barkbone
Neck
Head
representation
Amplified embedding space divergence
Client
Client
Client
2. Full Parameter Training with Orthogonal Enhancement
Figure 1: An overview of our FedSTO method within the SSFOD framework with key components: selective training, orthogonal enhancement, and local Exponential Moving Average (EMA)-driven driveen pseudo label assignment, organized into stages. Algorithm steps numbered accordingly.
1. Limited scale and complexity of tasks with datasets such CIFAR and ImageNet, while semi-supervised federated object detection (SSFOD) presents sizably greater difficulties.
2. Non-lID data shift from labeled to unlabeled data. Our investigation stands apart in tackling the most challenging FL situations where clients hold exclusively unlabeled data from different distribution from labeled server data. This acknowledges inherent heterogeneity real-world FL settings, such as diverse weather conditions across clients. For instance, client's dataset may predominantly consist of images captured under cloudy conditions, while others may include images from overcast, rainy, snowy, etc. conditions.
To surmount these inadequately addressed challenges of SSFOD, introduce FedSTO (Federated Selective Training followed by Orthogonally enhanced training), a two-stage training strategy tailored specifically for our SSFOD framework (Figure Ip. Our key contributions include:
Selective Training and Orthogonal Enhancement: FedSTO begins with selective training of the model's 's backbone while other components remain frozen, fostering more consistent representations and establishing robust backbone. This promotes generalization across non-IID) clients, even in the absence of local labels. The subsequent stage involves fine- tuning all parameters with orthogonal regularizations applied to the non-backbone part of the model. This enhancement step is designed to imbue the predictors with resilience against skewed representations induced by local data heterogeneity, thereby promoting representation divergence and robustness.
To prevent deterioration of teacher pseudo labeling models for non-III unlabeled clients, we showcase for the first time an SSFOD framework that applies an alternate training methodology (5), integrated with a Semi-Efficient Teacher (38), driven by local Exponential Moving Average (EMA). Our empirical observations suggest that this personalized EMA-driven model provides superior quality pseudo labels for detection, contrary to the commonly used global model.
SSFL with a Personalized EMA-Driven Semi-Efficient Teacher:

---

mAP
Fully Supervised Training images]
0.605
0.60
0.593
0.558
0.55
New Rarumarr diiiiie
-15K Labeled
0.511
"ISK Unlabeled Images Eightt between non cliced)
0.50
Servac with Labdded Duta
"SK Labeled Images
0.45
Pariilly (Tro ring onk apvled raged)
Verlle Federsted Leorning
Oar FelS1O 55A000 Fornwwork (196 wilhiidd
"K Labeled Images
Trakning
labe mapas
(41). "Partially Supervised Training" shows lower-bound performance using partial labels in a centralized setting. "Vanilla Semi- Semi-Supervised Federated Learning" and "Our FedSTO" demonstrate improved performance with non-llD federated data. FedSTO approaches the "Fully Supervised Training" upper-bound performance under full label use a centralized setting. The x-axis shows the number labeled examples, and the y-axis displays the mean average precision (mAP@0.5) the test set.
Figure 2: Performance comparison on BDD100K dataset
for pseudo labeling in related studies [3. This approach further enhances the quality of the learning process, mitigating potential pitfalls of noisy pseudo labeling.
Performance Improvements: FedSTO achieves 0.082 and 0.035 higher mAP@0.5 when compared to partially supervised and SSFL baselines respectively, nearly matching the fully supervised model's performance (0.012 gap) on BDD100K 41 with a mere 25% of labeled data. We demonstrate similar considerable improvements in model generalization (Figure 2) on rigorous benchmark and ablation experiments with 20k- 120k datapoints from Cityscapes and SODAIOM [9], utilizing the YOLOv5 object detector [13).
Our above contributions present a pioneering approach for utilizing unlabeled data in FL to enhance non-lID detection performance, especially for dynamic objects-an aspect not yet considered in previous research. Despite the paucity of research on SSFOD, we contend that our methods and experiments offer valuable benchmark for future investigations across diverse domains.
Related Works
2.1 Federated Learning (FL): Challenges and Advances
FL has gained significant attention in recent years as a crivacy-preserving approach to harness the Despite the progress made FL., most research has focused primarily on classification tasks, which may limit its applicability and generalizability to broader range of real-world problems. Advanced FL techniques are essential to revolutionize various fields, including autonomous driving, healthcare, and finance, by enabling collaborative learning from distributed data sources 30
potential of distributed data (7 20
28 33
Addressing data heterogeneity of paramount importance in FL., as clients frequently hold data with diverse distributions, which may impact the performance of the global model. To tackle this challenge, researchers have proposed various techniques to handle non-lID data, including adaptive aggregation algorithms and local fine-tuning of models 20 [3. Personalization constitutes another vital aspect of FL, since clients may exhibit unique requirements preferences not entirely captured by the global model [3] 19 39 Methods such as model distillation (23] and meta-learning
have been investigated to facilitate dlient-specife model adaptation and personalization, Finally, communication efficiency is crucial in FL, as exchanging model updates can be resource-intensive. To alleviate this issue, researchers have introduced strategies like quantization [4 sparsification 2 and the utilization of supernet containing multiple subnetworks, with only the selected subnetwork transmitted to the server to reduce communication overhead while preserving model performance [17].

---

2.2 Semi-Supervised Object Detection (SSOD) with YOLO Object Detector
SSOD has been an active research area, focusing on improving the quality of pseudo labels to enhance overall detection performance [35 40]. The evolution of SSL techniques in object detection primarily revolves around using pretrained architectures and applying strong data augmentation strategies to generate consistent and reliable pseudo labels.
Traditional single-stage detectors, such as the family of YOLO detectors, have faced notable chal- lenges in leveraging SSL techniques, often underperforming compared their two-stage counterparts (e.g., Faster RCNN). The limited efficacy of existing SSL methods for single-stage detectors has inspired researchers to develop innovative solutions to overcome these limitations 44 Recently, a novel pipeline incorporating EMA of model weights has exhibited remarkable enhancements in the performance of single-stage detectors like YOLO detectors [38]. By utilizing the EMA model for pseudo labeling, researchers have adeptly addressed the inherent weaknesses in single-stage detectors, substantially elevating their performance in SSL. contexts for object detection tasks.
2.3 Senr-Suuervised Federated Learning (SSFL)
SSFL has emerged as a promising approach to address the challenge of limited labeled data FL scenarios 5
42 SSFL aims to jointly use both labeled and unlabeled data owned by participants to improve FL.. Two primary settings have been explored: Labels-at-Clien and Labels-at-Son -t--errrrrrrr [10 M. In the Labels-at-Chie -lient scenario, clients possess labeled data, while the server only has access to unlabeled data. Conversely, in the Labels-at at-ssvrerrr scenario, the server holds labeled data, and clients have only unlabeled data. Despite the progress in SSFL, there remain limitations in the current research landscape. The majority of existing SSFL research predominantly focuses image classification tasks, leaving other applications relatively unaddressed. In this study, we address these limitations by tackling the more realistic and challenging scenarios with edge clients having (1) no labels and (2) non-IID data (domain shift from the server labeled data), specifically in the context of object detection tasks.
10 43 2 18
Problem Statement
SSFOD We tackle a semi-supervised object detection task involving a labeled dataset S = {æt y 1- and an unlabeled dataset 2 = { focusing on scenarios where Ns < No- In our SSFOD setup, as illustrated in Figure
assume M clients each possessing an unsupervised dataset ".. The server retains the labeled dataset {x",y") and a model parameterized by W*, Each client model, parameterized by W shares the object detection architecture denoted by f: (x, W) + f(x W). We assume that all models share the same object detection architecture, denoted by f: (xx W') - f(x, W), which maps an input æ and parameters W to a set of bounding boxes and their corresponding class probabilities on the K K-dimensional simplex (e.g., using the sigmoid function applied to model outputs unit-wise).
Data Heterogeneity Our study addresses non-III) data resulting from varying weather conditions such cloudy, overcast, rainy, and snow, inspired by feature distribution skew covariate shift (14). We utilize three datasets, BDD100K [41], CityScapes [4], and SODA 10M (9. each displaying class distribution heterogeneity and label density heterogeneity. Our aim is SSFOD framework that can manage this heterogeneity, maintaining performance across diverse conditions and distributions. Data is considered IID when each client exhibits balanced weather condition distribution.
In our framework, we assess the performance of all detection tasks based on mean average precision (mAP@0.5), a standard metric in object detection literature that provides a compre- hensive view of model performance across various object classes and sizes. Importantly, we evaluate the post-training performance of our method by assessing the personalized models of the server and client on their respective datasets. This approach ensures a fair and context-specific evaluation, reflecting the true performance of the personalized models in their intended environments.
Evaluation
Baseline Training Our work explores two principal baselines: "Centralized Training" and "Feder- ated Learning". Depending on the degree of labeled data utilization, we categorize the training into "Partially Supervised" and "Fully Supervised". An ideal situation is one where a fully supervised model is trained in centralized fashion, utilizing all labeled data. In contrast, a more challenging scenario involves a partially supervised model trained solely on the server's limited labeled data. Under our problem setup, initially establish a baseline by performing partial supervision on the

---

Table Performance under different weather conditions for non-IID and IID data splits with server and 3 clients. The results are presented for centralized (Fully Supervised and Partially Supervised) and federated approaches with a pseudo label assigner (Global Model and Local EMA Model).
Type
Method
Non-IID
IID
Cloudy
Overcast
Rainy
Snowy
Cloudy
Overcast
Rainy
Snowy
Fully Supervised
0.600
0.604
0.617
0.597
0.600
0.604
0.617
0.597
Centralized
Partially Supervised
0.540
0.545
0.484
0.474
0.528
0.545
0.533
0.510
Global Model [S]
0.555
0.560
0.497
0.488
0.540
0.551
0.576
0.542
Federated
Local EMA Model
0.560
0.566
0.553
0.553
0.572
0.588
0.593
0.610
Comparison Performance by Weather Condition
0.600
Centralized Learaing
Contralized Learning
Centralized Learning
Centralized Learning
0.625
Perfomance
0.600
0.575
0.500
0:52
0300
0.50
Cloudy
Ra ny
Snowy
CL: Partilly Supervised
CL:55L- EMA Teacher
8908
Fl:: Varil SSFL (Sase ine)
Figure 3: Performance various methods on the BDD100K dataset 441, with the server containing labeled data for the "Cloudy" category and 3 clients having unlabeled data for "Rainy", "Snowy", and "Overcast" categories. Baseline SSFL (red hatched boxes) struggles in comparison to centralized learning (bars in green dotted boxes). "Fully Supervised" and "Partially Supervised" refer to training centralized model with the complete labeled dataset and only the "Cloudy" labeled data, respectively.
server's limited labeled data, which serves as a pretraining step. Following this, each client conducts "Unsupervised learning with unlabeled data". Upon completion, clients transmit their model weights to the server. The server then aggregates these weights and fine-tunes the amalgamated model using its labeled data. The updated model is subsequently disseminated back the clients, culminating one learning round. This cyclical process, known as alternate training in Diao et al. [5), continues effectively. merges the strength of supervised and unsupervised learning to capitalize unlabeled data while preventing model deterioration, thereby optimizing model performance.
Personalized Pseudo Labeling for Unlabeled Clients crucial obstacle in SSFOD lies in precise pseudo label assignment, as improper allotments can result in label inconsistencies, thus negatively impacting mutual learning performance. Building upon the foundation by Xu al. 38 centralized settings, present the first extension of this approach to federated settings, leveraging a personalized Pseudo Label Assigner (PLA) equipped with local EMA models. This technique bifurcates pseudo labels into reliable and unreliable categories using high and low thresholds, thus ensuring robust and precise learning mechanism federated environments. global and local models. However, the global model may fall short in capturing unique features of local data, compromising pseudo label quality. As demonstrated in our evaluation (Table 1), locally updated E.A models outperform global models. While it is feasible to federate the local EMA model, it introduces certain trade-offs, such as increased communication costs and minor performance degradation compared to the local EMA model. Our SSFOD framework, therefore, incorporates a local PLA with a local EMA model, optimally balancing communication efficiency and model stability, ensuring effective learning process for SSOD tasks in distributed environments.
FL, the PLA can applied both
SSFOD with YOLO We utilize the YOLOv5 model, single-stage object detector, in our evaluation. Existing literature shows a scarcity of research on SSFOD within FL like FedAvg (27). particularly for single-stage detectors like YOLO. Figure 3 compares various learning approaches in centralized and federated settings, denoted by green dotted and blue hatched boxes, respectively. We highlight non-IID scenarios with labeled (cloudy) and unlabeled data (overcast, rainy, snowy). In the CL scenario, fully supervised methods noticeably surpass partially supervised ones, and SSL approaches almost match the performance of fully supervised methods. However, baseline training for FL falls substantially short of these high standards, particularly with unlabeled data.
5

---

Table 2: Performance on the BDD dataset with labeled server and unlabeled clients as each element of our FedSTO approach within the SSFOD framework is added. It highlights how each added method contributes to the overall performance under both Non-IID and IID conditions.
Non-III
Method
Cloudy
Overcast
Rainy
Snowy
Total
Cloudy
Overcast
Rainy
Snowy
Total
Partially Supervisod
0.540
0.545
0.484
0.474
0.511
0.528
0.545
0,533
0.510
0.529
SSFL [S] with Local EMA Model
0.560
0.566
0.553
0.553
0.558
0.572
0.588
0.593
0.610
0.591
Selective Training
0.571
0.583
0.557
0.556
0.567
0.576
0.578
0.594
0.599
0.587
FPT with Orthogonal Enhancement 16
0.596
0.607
0.590
0.580
0.593
0.591
0.634
0.614
0.595
0.609
4 Main Method: FedSTO
To mitigate these inherent hurdles presented by FL., we introduce FedSTO, a method that unfolds in two stages, preceded by a warmup stage. The process commences with emphasis robust representation learning for pretraining (Subsection 4.1p, followed by full parameter training (Sub) section 4.2p The initial stage of pretraining integrates a warm-up period utilizing labeled data at the server, transitioning into selective training. This groundwork is fortified by the orthogonal enhancement implemented in the subsequent full parameter training phase.
4.1 Selective Training (ST)
Selective Training (ST) designed address the primary challenge of establishing robust backbone for object detector in FL. The approach unfolds as follows:
1. Labeled dataset training:
All model parameters are updated using a labeled dataset. This step ensures training commences on high quality labeled data, mitigating the potential desta- bilizing effect of noisy, unlabeled data, and heterogeneity from diverse weather conditions.
2. Client-side training with unlabeled dataset: The model, updated in the previous step, is dispatched to the clients. Each client trains the model their local unlabeled dataset. However, during this phase, only the backbonn part of the model is updated, leaving other components frozen. This selective updating procedure fosters more consistent representa- tions by sharing the same non-backbone part (e.g., neck and head), and thus enhances its potential generalization capabilities by concentrating feature extraction.
3. Server-side aggregation: The server aggregates the updated backbone parameters from clients, effectively synthesizing the learned information from diverse unlabeled datasets. The aggregated backbone is then utilized in the first step for further training, repeating the process until performance convergence.
By adhering to this procedure, ST effectively navigates the challenges inherent in the progression of FL while simultaneously accruing substantial benefits. Ensuring stability in semi-supervised object detection tasks paramount. The exposure to heterogeneous unlabeled data, potentially characterized by noise or variable quality, can induce biases into the neck and head components of the model, thereby risking performance degradation by inadvertently generating low-quality or imprecise pseudo annotations. To mitigate this, ST employs selective update strategy specifically targeting the backbone of the model, which is predominantly entrusted with the task of extracting salient features from the input data. By concentrating on the backbone during training, ST aids in the preservation of model stability and the enhancement of its generalization capabilities. Furthermore, in this stage, the communication cost between the server and clients is significantly reduced by uploading only the backbone part from clients to a server. Consequently, it significantly minimizes the deleterious impacts of heterogeneous unlabeled data overall model performance (Tall 2). While ST brings marginal improvements in IID conditions, it presents potent effects under Non-lID circumstances, emphasizing its efficacy in handling heterogeneous data distributions.
4.2 Full Parameter Training (FPT) with Orthogonal Enhancement
Inspired by the critical need for personalized models to exhibit robustness against feature distribution wwwvness-preddmmnnnntte due to diverse weather conditions wwe integrate orthogonality regular- ization presented by Kim et al. [6) which penalizes the symmetric version of the spectral restricted isometry property regularization Eo a0 /) + a (00" f) within the SSFOD framework where a() calculates the spectral norm of the input matrix and 0 a weight matrix from backbacb parts. This regularization is applied during both server and client training stages and targets non-backbone

---

Algorithm 1: FedSTO Algorithm within the SSFOD Framework
INPUT server model parameterized by Ws, the number of rounds for each phase Ti, 72, client
models parameterized by (W,,,, Wa,ar), client backhone part parameterized by
[u,,1= :.. B,
1: W, f WARMUP(Z,,
W.)
// Supervised training at server
1a Phase 1: Selective Training for Pretraining */
2: for + 0+.0..., do
3: S"
SAMPLELLIENTS
4: for each client 5" in parallel do
5: Wu.k f
Bu,k)
// Client-Update
6: end for
7: W. < EkEst P&Wu.. u.
Aggregation
8: W. < SERVER SEERRUUPDAT((A,, Vs, Ws)
// Server-Update
9: end for
1a Phase 2: Full Parameter Training with Orthogonal Enhancement */
10: for t + 0....., T2
do
11:
St --MMPLEECLEENTS
12:
for each client A: 5t in parallel do
Wi, 6 C.IENTTOORTHOGGNAA..PATTTT // Client Shhett-Orthogonalupdatee
W W..k))
14:
end for
15:
// Aggregation W. <- SERVER-ORTHOOGNNAN SIANE--ORRTTOCOOMLLPPNTTIICLL W.) // Serverr-
W. Ekest P&
16:
17: end for
components of the architecture. Our approach promotes generation of diverse, non-redundant, and domain-invariant feature representations, thereby enhancing the model's robustness, reducing noise influence, and significantly augmenting its ability to handle unlabeled data across varied domains.
Incorporating orthogonality regularization into our framework substantially amplifies the divergence in the embedding space, enhancing the model's 's overall detection quality and the reliability of pseudo labels. Importantly, our strategy of embedding orthogonality into the non- non-backbone parts of the model, such as the neck and head, fosters a more balanced and comprehensive training process. This reduces the bias towards specific weather conditions and the heterogeneity of object categories, leading to improved performance as demonstrated in Table 2] Our approach draws upon successful techniques from fine-tuning
26], and transfer learning, and is particularly inspired by meta- learning concepts, (37 (22. In particular, the tendency of the non-backbone components of the model to develop biases prompts us to introduce an orthogonal property to this section. This measure helps counteract these biases, thereby further enhancing the model's robustness and adaptability when confronted with diverse, unlabeled data across multiple domains.
4.3 Main Algorithm: FedSTO
Iillustrates the overall procedure of FedSTO within the SSFOD framework. The server model, parameterized by W., is first trained in supervised fashion during the warm-up phase (Line 1). The algorithm then transitions to Phase 1: Selective Training for Pretraining. This phase involves multiple training iterations (Line 3), where in each iteration, a subset of clients is sampled (Line k. is updated using their local unlabeled datasets 4). The backbone part of each client's model, W Wr,k.
Algorithm Iatilammiilllttaaa
(Line 6). The updated parameters are then aggregated at the server (Line 8), and the server model updated using its labeled dataset (Line 9). Phase Full Parameter Training with Orthogonal Enhancement, the C1.INNT-O C.JENT-- SAATOGGNAA.UPDAAAS and SERVER SERVER-ORTHOGONN OOTTHOOOOOAOI UPDATE METHODS are employed (Lines 14 and 18), introducing orthogonality regularization the training process. This second phase debiases the non-backbone parts of the model, ensuring they have robust predictor across various weather conditions that effectively counterbalance the inherent data heterogeneity.
Experiment
5.1 Experimental Setup
5.1.1 Datasets
BDD100K 41
utilize the BDD100K dataset, which consists of 100,000 driving videos recorded across diverse U.S. locations and under various weather conditions, to evaluate our method. Each
7

---

Table 3: Comparison of FedSTO within the SSFOD framework against the Baselines, SSL, SSFL methods with I server and 3 clients on BDD100K dataset [4]] FedSTO exhibits improvements under various weather conditions on both IID and Non IID cases, and performs close to the centralized fully supervised case. i denotes the SSFL with the local EMA model as pseudo label generator.
Type
Algorithm
Method
Noe-IID
ID
Cloudy
Overcast
Rainy
Snowy
Total
Cloudy
Overcast
Rainy
Snowy
Total
Contralized
S1.
Partially Supervised
Fully Supervised
0.600
0.540
0.601
0.545
0.617
0.484
0.597
0.474
0.605
0.511
0.600
0.528
0.604
0.545
0.617
0.533
0.510
0.597
0.529
0.605
SSL
Unbissed Teacher
[S
0.551
0.550
0.502
0.503
0.546
0.557
0.541
0.533
0.544
EMA Teacher [38
0.598
0.59
0.568
0.568
0.581
0.586
0.570
0.571
0.573
0.575
SFL
Fully Supervised
0.627
0.614
0.607
0.585
0.608
0.635
0.612
0.608
0.595
0.613
FedAvg 2X
0.560
0.566
0.588
0.593
0.610
0.591
Federated
RedDyn 1|
0.508
0.569
0.541
0.522
0.535
0.355
0.414
0.420
0.397
0.400
SSPI
FedOOpt
0.561
0.572
0:565
0.566
0.566
0.591
0.587
0.588
0.577
0.586
FodPAC
0.514
0.532
0.496
0.489
0.508
0.510
0.549
0.547
0.554
0.540
FedSTO
0.596
0.607
0.590
0.580
0,593
0.591
0.634
0.614
0.595
0.609
video, approximately 40 seconds in duration, is recorded at 720p and 30 fps, with GPS/IMU data available for driving trajectories. For our experiments, we specifically select 20,000 data points, distributed across four distinct weather conditions-cloudde rainy, overcast, and snowy. In this study, we primarily focus on five object categories: person, car, bus, truck, and traffic sign. The dataset is partitioned into clients based on these weather conditions, simulating data-heetrogeneous clients. This experimental setup enables us to investigate the influence of data heterogeneity on our framework and to evaluate its robustness under realistic conditions.
We conduct additional experiments using the Cityscapes dataset, which consists of urban street scenes from 50 different cities. Given that this dataset does not provide precise weather information for each annotation, we distribute the data to clients in a uniformly random manner. For our studies, we employ the package, encompassing fine annotations for 3,475 images in the training and validation sets, and dummy annotations for the test set with 1,525 images. We also include the other package, providing an additional 19,998 8-bit images for training.
Cityscape [
To evaluate our approach under diverse conditions, we employ the SODA 10M dataset, which features varied geographies, weather conditions, and object categories. In an IID setup, 20,000 labeled data points are uniformly distributed among one server and three clients. For more realistic setup, the 20,000 labeled data points are kept on the server while 100,000 unlabeled data points are distributed across the clients. This arrangement enables performance evaluation under distinct weather conditions-cled overcast, and rainy showcasing resilience and robustness.
SODAIOM [
5.1.2 Training Details
We conduct our experiments in an environment with one server and multiple clients, depending on the experiment. Both the server and the clients operate on a single local epoch per round. Our training regimen spans 300 rounds: 50 rounds of warm-up, 100 rounds of pretraining (Ti), and 150 rounds of orthogonal enhancement (T2). We use the YOLOv5 Large model architecture with Mosaic, left-right flip, large scale jittering, graying, Gaussian blur, cutout, and color space conversion augmentations. constant learning rate of 0.01 was maintained. Binary sigmoid functions determined objectiveness and class probability with balance ratio of 0.3 for class, 0.7 for object, and an anchor threshold of 4.0. The ignore threshold ranged from 0.1 to 0.6, with an Non-Maximum Suppression (NMS) confidence threshold of 0.1 and an IoU threshold of 0.65, We incorporate an exponential moving average (EMA) rate of 0.999 for stable model parameter representation.
5.2 Results
Table 3 Jillustrates the efficacy of our proposed SSFOD method against various baselines and state-of- the-art approaches on the BDDI00K dataset. FedSTO significantly outperforms other techniques under different weather conditions and data distribution scenarios, i.e., IID and Non-IID, In the CL scenarios, the fully supervised approach yields the highest performance, with SSL methods, such as EMA Teacher [38], demonstrating competitive results. However, the real challenge lies in federated settings, where data privacy and distribution shift become critical considerations. In the SSFOD framework, our FedSTO method consistently surpasses other SSFL techniques. Notably, it achieves superior results even in challenging Non-IID settings, demonstrating its robustness to data distribution shifts. Similar trends hold when increasing the number of clients as shown in the appendix. In IID
8

---

Table 4: Performance under random distributed cases of Cityscapes 4 FedSTO exhibits improve- ments under various object categories, and significantly outperforms the performance for unlabeled clients. denotes the SSFL with the local EMA model as a local pseudo label generator.
Labeled
Unlabeled
Type
Algorithm
Categories
Person
Car
Bus
Truck
Traftic Sign
Person
Car
Bus
Truck
Traffcc Sign
Centralized
SL
Partially Supervised
Fully Supervised
0.569
0.350
0.778
0.683
0.530
0.193
0.307
0.302
0.500
0.246
0.358
0.560
0.648
0.788
0.571
0.343
0.283
0.138
0.255
0,510
SSL
Unhiased Teacher 25
0.391
0.695
0.225
0.320
0.297
0.410
0.689
0.373
0.129
0.354
HMA Teacher
0.475
0.711
0:354
0.347
0.460
0.727
0.436
SFL
Fully Supervised
0.498
0.715
0.357
0.289
0,410
0.492
0.714
0.451
0.251
0.425
Fedenited
FodAvg
22]
0.450
0.697
0.310
0.304
0.356
0.482
0.725
0.425
0.247
0.397
SSFL
FedBN
0.709
0.375
0.618
0.066
FedSTO
0.504
0.720
0.342
0.261
0.415
0.487
0.740
0.460
0.181
0.437
conditions, our method continues to excel, achieving results close to fully supervised centralized approach. These findings highlight the strength of our FedSTO method in leveraging the benefits of FL while mitigating its challenges. The robust performance of our approach across various weather conditions and data distributions underscores its potential for real-world deployment.
When examining the performance on the Cityscapes dataset under uniformly random distributed conditions, the superiority of FedSTO within the SSFOI) framework also remains apparent, as shown in in[aable 4 Compared to other methods, FedSTO consistently demonstrates improved generalization across most object categories, both for labeled and unlabeled data. Intriguingly, the performance of FedSTO surpasses even that of SSL in CL environments.
Evaluation with mAP 0.75 The mAP@0.75 results on the BDD dataset highlight the efficacy of the FedSTO approach (Table 5). In Non-IID settings, while the Fully Supervised centralized method achieve an average mAP of 0.357, FedSTO recorded 0.338, exhibiting comparable performance. However, under IID conditions, FedSTO registers an mAp@0 0.755 of 0.357, closely matching the SFL result of 0.359. These results indicate that FedSTO offers competitive object detection capabilities, even with stricter IoU thresholds.
Table 5: mAP@0.75 on the BDD dataset with labeled server and 3 unlabeled clients.
Type
Algorichm
Method
Non-IID>
M
Cloudy
Overcast
Rainy
Snowy
Total
Clondy
Overcast
Ruiny
Snowy
Total
Centralized
SL.
Fully Supervised
Partially Supervised
0,281
0.351
0.352
0.295
0.368
0.261
0.303
0.356
0.285
0.357
0.351
0.281
0.295
0.352
0.261
0.368
0.303
0.356
0.285
0.357
SFL
Fully Supervised
0.384
0.366
0.357
0.317
0.356
0.377
0.356
0.352
0.349
0.359
Federated
SSFL
FedSTO W/ ST PedSTO
0.321
0.347
0.303
0.265
0.267
0.312
0.289
0.338
0.298
0.343
0.321
0.375
0.361
0.324
0.302
0.350
0.311
0.357
Figure 4a illustrates the performance of our method and other baselines on the SODA dataset. where labeled data synthetically divided in an IID manner across one server and three clients. Our method demonstrates near- near-parity with the fully supervised approach, evidencing its efficacy. FFiuure 4b represents the averaged performance across varying weather conditions on the SODA 10m dataset. Here, all 20k labeled data resides on the server, and 100k unlabeled data points from SODA 10m are distributed across three clients. Despite these variations in conditions, our method consistently outperforms other baselines, confirming its robustness and applicability in diverse environments.
Results on Real World Dataset, SODA 99]
Roalistic Scanario
0.60
0.45
0.50
0.40
Portol Sup Superveed
Fedterg
FediN
Rut Superised
Partil Supertsed
bothert
PedtN
(a) IID Case
(b) Real World Scenario
Figure 4: (a) Performance of various methods the SODA 10m dataset in an IID setting, (b) Average performance across different weather conditions using unlabeled data from the SODA 10m dataset.

---

Varying Number of Clients In non-IlD
Table 6: Performance a non-IlI case of the BDD100K dataset with server and 20 clients.
BDD dataset configuration with server and 20 clients, our proposal advances beyond compet- ing methods, scoring 0.455 and 0.458 on labeled and unlabeled data, respectively. This outcome showcases our method's ' aptitude for tackling intricate real-world circumstances.
Type
Centralized
Wederated-SSFF
Method
Partially Supervised
FodAvg
FodSTO
Labeled
0.3768
0,405
0.405
0.455
Unlabeled
0.3524
0.4311
0.4322
0.458
Varying Sampling Ratio TTale 7 demonstrates the impact of different client sampling ratios on the FedSTO performance using the BDD100K dataset. Notably, even at a lower sampling ratio of 0.1. FedSTO yields commendable results, especially in the unlabeled set for categories like 'Car' (0.738) and 'Bus' (0.573). This underscores that a reduced client sampling still lead to significant performance improvements, emphasizing the efficiency and adaptability of the FL approach.
Table 7: Performance (mAP@0.5) under Non-IID scenarios of BDD100K dataset with server and
100 clients according the changes of client sampling ratio for implementing FedSTO. The term 'Server Only' aligns with the notion of "partillly supervised' in CL settings.
Labeled
Unlabeled
Method
Categorics
Person
Car
Bus
Truck
Traffic Sign
Porson
Car
Bus
Truck
Traffic Sign
Server Only (i.e., client sampling ratio 0.D)
0.378
0.710
0.141
0.425
0.490
0.337
0.707
0.160
0.338
0.491
FedSTO with elient sampling ratio 0.1
0.393
0.714
0,442
0.510
0.540
0.487
0.738
0.573
0.589
0.617
FedSTO with client sampling ratio 0.2
0.458
0.747
0.476
0,521
0.571
0.440
0.731
0.378
0.525
0.573
FedSTO client sampling ratio 0.5
0.444
0.745
0.437
0.502
0.550
0.489
0.730
0,438
0.512
0.538
Efficiency on Network Bandwidth Table 8hiighigght the communication costs over 350 rounds of training involving 100 clients with 0.5 client sampling ratio per round. By removing the neck component of the Yolov5L model, its size is reduced from 181.7MB to 107.13MB. This reduction significantly benefits FedSTO in Phase 1, leading to overall bandwidth savings. When comparing with traditional SSFL methods such as FedAvg and FedProx, (20). FedSTO utilizes only 2,166.23 GB substantial 20.52% reduction in network bandwidth.
Table 8: Communication costs over 350 rounds of training with 100 clients when the client sampling ratio is 0.5 per each round. The total Yolov5L size is 181 181.7MB while the model without the neck part is 107.13MB. Additionally, the model size without BN layers (FedBN (22)) is 181.24 MB. Here, 'Reduction' expresses how much communication cost is reduced compared to using vanilla SSFL
(FedAvg and FedProx (20). )
Warm-up (50 rounds)
Method
Phuse 1 (150 rounds)
Phuse 2 (150 rounds)
Total
Reduction
FedAvg
FedProx
0
100 * 0.50 150 * 181.7 1,362.75 GB
100 * 0.50 150 x 181.7 = 1,362.75 GB
2.725.50 GB
FedBN
0
100 0.50 150 * 181.24 = 1359,30 GB
100 0.50 150 x 181.24 = 1339,30 GB
2.718.60 GB
0.25 %
FedSTO
0.50 107.13 803.48 GB
A 181.7 1,362.75
2,166.23 GB
6 Conclusion
This paper introduces novel Supervisdd Federated Object Detection (SSFOD) framework, featuring a distinctive two-stage training strategy known as FedSTO. Designed to address the chal- lenges of heterogeneous unlabeled data in federated learning, FedSTO employs selective training and orthogonality regularization with personalized pseudo labeling. These mechanisms facilitate robust and diverse feature learning, thereby enhancing object detection performance across multiple weather conditions and data distributions. Empirical results provide compelling evidence of the superiority of FedSTO over established federated and supeviuper learning methodologies. Notably, despite operating with the challenging constraint where non-III clients have no labels, FedSTO successfully counteracts domain shift and achieves performance that is comparable to fully supervised centralized models. This accomplishment constitutes significant strides toward realizing more efficient and prevacy-preserving learning in realistic FL settings. As we venture ahead, we aim to concentrate our research efforts on refining FedSTO and exploring additional strategies for leveraging unlabeled data with various domains and model architectures. We anticipate the work presented in this paper will stimulate continued progress in this rapidly evolving field.
10

---

References
[1] Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N What- mough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint Stdi::2111.022000 2021.
[2] Jieming Bian, Zhu Fu, and Jie Xu. Fedseal: Semi-supervised federated learning with self- ensemble learning and negative learning. arXiv preprint arXiv:2110.07829, 2021.
[3] Hong-You You Chen and Wei-Lun Chao, On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2022.
[4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[5] Enmao Diao, Jie Ding, and Vahid Tarokh. Semifl: Sumi-supervised federated learning for unlabeled clients with alternate training. Advances in Neural Information Processing Systems, 35:17871-17884 2022.
[6] Zhaoyang Du, Celimuge Wu, Tsutomu Yoshinaga, Kok-Lim Alvin Yau, Yusheng Ji, and Jie Li. Federated learning for vehicular internet of things: Recent advances and open issues. IEEE Open Journal of the Computer Society, 1:45-61, 2020.
[7] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: meta-learning approach. arXiv preprint arXiv:2002.07944 2020.
[8] Karim Guirguis, Ahmed Hendawy, George Eskandar, Mohamed Abdelsamad, Matthias Kayser, and Jürgen Beyerer. Cfa: Constraint- based finetuning approach for generalized few-shot object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2022.
Recognition, pages
[9] Jianhua Han, Xiwen Liang, Hang Xu, Kai Chen, Lanqing Hong, Jiageng Mao, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Xiaodan Liang, et al. Sodal0m: a large-scale 2d self/semi-supervisee object detection dataset for autonomous driving, arXiv preprint Xrv: 106. 1118, 2021.
[10] Wonyong Jeong, Jaehong Yoon, Eunho Yang, and Sung Ju Hwang. Federated semi-supervised learning with inter-client consistency & disjoint learning. arXiv preprint arXiv:2006. 12097, 2020.
[11] Yilun Jin, Yang Liu, Kai Chen, and Qiang Yang. Federated learning without full labels: A survey, arXiv preprint S.VV:2203311453 2023.
[12] Yilun Jin, Xiguang Wei, Yang Liu, and Qiang Yang. Towards utilizing unlabeled data in federated learning: A survey and prospective. arXiv preprint arXiv:2002.11545 2020.
[13] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, Kalen Michael, TaoXie, Jiacong Fang. imyhxy, Lorna, Zeng Yifu, Colin Wong, Abhiram V, Diego Montes, Zhiqiang Wang, Cristi Fati, Jebastin Nadar, Laughing, UnglvKitDe, Victor Sonck, tkianai, yxNONG, Piotr Skalski, Adam Hogan, Dhruv Nair, Max Strobel, and Mrinal Jain. ultralyticsyyolovS:: v7.0- YOLOv5 SOTA Realtime Instance Segmentation, November 2022.
[14] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji. Kallista Bonawitz, Zachary Charles, Graham Connode, Rachel Cummings. et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1 1-00, 2021.
[15] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for on-device federated learning, arXiv preprint arXiv: arXi::1910.06378 2019.

---

[16] Tachyeon Kim and Se- Young Yun. Revisiting orthogonality regularization: a study for convolu- tional neural networks in image classification. IEEE Access, 10:69741-69749 2022.
[17] Taehyeon Kim and Se- Young Yun. Supernet training for federated image classification under system heterogeneity, 2022.
[18] Woojung Kim, Keondo Park, Kihyuk Sohn, Raphael Shu, and Hyung-Sin Kim. Federated semi-supervised learning with prototypical networks. arXiv preprint vrxi:2225../3921, 2022.
[19] Kavya Kopparapu, Eric Lin, and Jessica Zhao. Feded: Improving performance non-iid federated learning, arXiv preprint arXiv:2006.096:3 096377 2020.
[20] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv:1812.0612 2018.
[21] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. arXiv preprint arXiv:190..02/89 2019.
[22] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning non-iid features via local batch normalization. arXiv preprint arX:v:2102.07623, 2021.
[23] Tao Lin. Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion federated learning. arXiv preprint #xXv:20006002222 2020.
[24] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[25] Yen-Cheng Liu, Chih-Yas Ma. Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Peter Vajda. Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09489, 2021.
[26] Itzik Malkiel and Lior Wolf. Mml: Maximal multiverse learning for robust fine-tuning of language models. arXiv preprint arXiv: /9/ /806/82, 2019.
[27] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas Communication-cfice -ffficientt learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[28] Mehryar Mohri, Gary Sivek, and Ananda Theertha Suresh. Agnostic federated learning. arXiv preprint 190. 2019.
[29] Vaikkunth Mugunthan, Eric Lin, Vignesh Gokul, Christian Lau, Lalana Kagal, and Steve Pieper. Fedlm:: Federated learning for sparse and personalized lottery ticket networks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XII, pages 69-85. Springer, 2022.
[30] Dinh C Nguyen, Quoc-Viee Pham, Pubudu N Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang. Federated learning for smart healthcare: A survey. ACM Computing Surveys (CSUR), 55(3):1-37, 2022.
[31] State of California Department of Justice. California consumer privacy act. Bov/privacy/ccpa 2018
ca
[32] Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12333-12343, 2021.
[33] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint 2003000033. 2020.
12

---

[34] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: communication--fficcee federated learning method with periodic averaging and quantization. In International Conference Artificial Intelligence and Stafistics, pages 2021- 2031. PMIR, 2020.
[35] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen- Yu Lee, and Tomas Pfis- ter. A simple supi-supereieed learning framework for object detection. arXiv preprint arXiv:2005.04757, 2020.
[36] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, Ist Ed., Cham: Springer International Publishing, 1031122766:10-5555 2017.
[37] Jingjing Wang. Jingyi Zhang, Ying Bian, Youyi Cai, Chunmao Wang, and Shiliang Pu. Self- domain adaptation for face anti-spoofing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2746-27267 2754. 2021.
[38] Bowen Xu, Mingtao Chen, Wenlong Guan, and Lulu Hu. Efficient teacher: Semi-supervised object detection for yolov5. arXiv preprint arXiv:2302.07577, 2023.
[39] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In The Eleventh International Conference on Learning Representations, 2023.
[40] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end remi-supervised object detection with soft teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3060-3069, 2021.
[41] Fisher Yu, Haofeng Chen, Xin Wang. Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2636-2645 2020.
[42] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), pages 1214-1225. IEEE, 2021.
[43] Jessica Zhao, Sayan Ghosh, Akash Bharadwaj, and Chih-Yao Ma. When does the student surpass the teacher? federated semi-supervised learning with teacher-student ema. arXiv preprint 2rX1v:2301.10114 2023.
[44] Huayi Zhou, Fei Jiang, and Hongtao Lu. Soda-yolo: Semi-super domain adaptive yolo for cross-domain object detection. Computer Vision and Image Understanding, 229: 103649, 2023.
13

---

A Overview of Appendix
In this supplementary material, present additional details that not included in the main paper due to the space limit.
B Ethics Statement
In the pursuit of progress within the domain of Federated Learning (FL.), we propose novel method, FedSTO within the SSFOD. This innovation warrants an examination of its ethical ramifications, particularly in regards to privacy, fairesss, environmental impact, and potential for misuse.
Privacy and Data Security Analogous to established FL methodologies, SSFOD is architected to preserve the privacy of client data. By conducting computations locally and transmitting solely model updates, the risks associated with raw data transmission are mitigated. Nonetheless, the potential threat of adversarial actions, such as model inversion attacks, underscores the imperative for ongoing efforts to bolster the robustness of FL methods against such vulnerabilities.
Fairness The deployment of FedSTO into SSFOD either amplify alleviate existing fairness concerns within FL., contingent on the particulars its application. Should SSFOD predominantly utilize data stemming from specific demographic cohorts, the model predictions risk acquiring an inadvertent bias. In contrast, SSFOD's s capacity to manage large-scale, real-world scenarios may engender diverse data inclusion, thereby fostering
equitable model.
Environmental Impact Similar to its FL counterparts, our approach diminishes the requirement for centralized data storage and computation, thereby potentially reducing associated carbon footprints. However, the energy expenditures of localized computation and communication for model updates necessitate judicious management uphold environmental sustainability.
Potential Misuses Although our approach is designed to enhance the robustness of FI. methodolo- gies in large-scale, real-world settings, the potential for misuse remains. For instance, malevolent entities could exploit FedSTO with SSFOD framework to deliberately introduce bias or disinforma- tion into models. Consequently, the implementation of protective measures against such misuse is of paramount importance.
In summary, while our method represents a significant contribution to the FL field, its potential ethical implications mandate thoughtful application. We strongly endorse ongoing discourse and scrutiny to ensure that its deployment is in alignment with the principles of privacy, fairness, and social responsibility.
C Limitations
While our SSFOD method offers significant advancements in Semi- comi-Supervised Federated Learning for Object Detection, is important to recognize accompanying limitations:
1. Performance with Highly Imbalanced Data Our SSFOD method exhibits robustness across range of data hetcrogeneity. However, its performance in scenarios involving severe data imbalance across clients necessitates additional exploration. Imbalance here refers to disparities in data class distribution across clients. Instances of extreme skewness could lead to biased learning outcomes, with the model excelling in classes with abundant samples and faltering in those with fewer. Although SSFOD incorporates mechanisms to offset effects of data heterogeneity, it is not explicitly designed to manage extreme data imbalance.
2. Computational Overhead Despite its effectiveness in bolstering model robustness in large- scale, real-worke scenarios, SSFOD introduces additional computational overhead. This primarily due to the algorithmic complexities and computational requirements intrinsic to our method, which might be a constraint for resource-limited devices often participating in federated learning. This could potentially limit the scalability and applicability of our method in real-world FL. scenarios. Therefore, improving the computational efficiency of SSFOD without compromising its efficacy is an important direction for future research.
14

---

3. Sensitivity to Varying Weather Conditions SSFOD has been designed to tackle the challenge of varying weather conditions in autonomous driving. However, real-world scenarios, there are other types of environmental changes that can equally affect the learning process. For instance, varying lighting conditions or changes in road surfaces might influence the input data. Since our SSFOD method primarily focuses on weather conditions, it might not exhibit the same level of efficiency when dealing with other environmental factors. Future iterations of SSFOD could explore these areas to provide more comprehensive solution to environmental handling in FL.
D Future Directions
Despite the aforementioned limitations, our SSFOD method lays the groundwork for several promis- ing future research directions:
1. Handling Other Environmental Factors: Future work could extend the SSFOD method to handle other environmental factors efficiently. This would make a more comprehensive solution for real-world federated learning scenarios where different environmental factors coexist.
2. Adaptation for Imbalanced Data: Investigating and enhancing the performance of SSFOD with highly imbalanced data distribution would be valuable future direction. Techniques like adaptive resampling or cost-sensitive learning could be integrated with our method to tackle this challenge.
3. Optimization of Computational Efficiency: Future research could focus on optimizing the computational efficiency of the SSFOD method. Reducing the computational overhead without compromising the robustness in large-scale, real-world scenarios would make our method more practical for real-world scenarios.
4. Robustness Against Adversarial Attacks: As the FL domain evolves, adversarial attacks pose an increasing threat to model robustness. Future work could explore how to bolster the SSFOD method (and FL methods, general) to ensure robustness against adversarial attacks.
By addressing these limitations and exploring these future directions, we can continuously refine and evolve the SSFOD method to better serve the ever-growing demands of federated learning.
E Detailed Data Heterogeneity in Federated Object Detection
Heterogeneity, prevalent attribute in object detection datasets, often arises from three crucial aspects:
Weather-induced feature distribution skew: Within outdoor scenarios like autonomous driving, weather variations significantly impact the visual representation of objects. Differing conditions such sunny, rainy, foggy, or snowy can alter an object's appearance, causing the weather-induced feature distribution skew. Moreover, sensor diversity, including RGB cameras and infrared sensors, contributes to this skew as they respond uniquely to various weather conditions. This complex scenario creates challenging task for an object detection system that must generalize across diverse conditions.
Class distribution heterogeneity: This heterogeneity refers to the uneven representation of various classes within a dataset. In many cases, certain classes are far more prevalent than others. For instance, in an autonomous driving scenario, 'cars' or "pedestrians' may be much more frequent than 'bicycles' or 'motorcycles. This imbalance can cause learning algorithms to develop a bias towards more common classes. Moreover, in a federated learning scenario, the class distribution may vary among clients; rural area client might capture more "animal' instances compared to an urban area client.
Label density heterogeneity: This form of heterogeneity pertains to the variation in the quantity of annotated objects per image. An image of crowded scene may contain far more objects than sparser image. This variability can influence the performance of detection models, particularly those that rely fixed number of anchors proposals per image.
15

---

Furthermore, it can also impact the training process as images with more objects provide more "training signal" per image than those with fewer objects. a federated learning context, certain clients might possess more densely labeled data than others, which could affect the learning process.
While current work primarily addresses the issue of weather-induced feature distribution skew, the other forms of heterogeneity, i.e., class distribution heterogeneity and label density heterogeneity. also require careful consideration. They present unique challenges within the federated learning environment and significantly influence model performance. It is crucial to extend our methodologies account for these factors, fostering more robust and versatile object defection systems capable of handling the intricate realities of real-world scenarios. Consequently, future work should aim at developing comprehensive solutions that cater all these facets of data heterogeneity in the context of federated learning.
F Exploring the Impact of Orthogonal Enhancement under Data Heterogeneity in SSFOD
F1
Theoretical Analysis
This section presents straightforward bounding strategy for the loss in the context of data hetero- geneity. Our primary focus is to ascertain the Mean Squared Error (MSE) of two-layered loyered neural network, encompassing both 'head' and 'backbone'. Analyzing this model's behavior under slightly perturbed data distribution is intended to offer insights into worst-case loss scenarios when orthogonal weights are employed.
We consider a multivariate distribution, denoted by PP, with a E Red and y E I being drawn from this distribution. We have i.i.d. samples from P, constituting D = (I,, y:): (æ,, y.) ~ IP, sis n. For the sake of this analysis, our model is expressed as f(x) = W Bx, where B E R Rexd and W E R are the backbone and head weight matrices, respectively. The MSE is given by Ez,yl y f(x) representing the expectation of the f²-norm of the difference vector between the ground truth and the prediction.
dataset
Under the presumption that / has been finely optimized on the training sample distribution (using the dataset 'D) to yield an MSE of value I,, we next examine a perturbed data distribution, denoted by P', characterized as follows:
(x'.y) P
(x', (xc + c. y)
(1)
Here, (x. v) ~ P and ~ N (a Σ) is Gaussian noise, characterized by covariance matrix Σ, independent of (x, y). The MSE loss under IP, represented by L', can be bound as:
Eg' ' l// - T Bx 2 = Ex.g. lly W B(x + E)12
Ex,y, y W Bæ W Bell
Eawlly W Bxl E. W Bell2
(2)
+ Laveel T Bx) TW Be]
+ En..ee T B W T Bx)]
= L Be Bel
This calculation introduces a nonnegative penalty of E | Bell to the out-of-sample MSE.
The derived penalty opens the possibility for further analysis, particularly when considering struc- tural assumptions on the head, backbone, or noise vector. The Gaussian assumption for the noise vector is quite enlightening, given that the squared (3-norm is tantamount to the quadratic form T'BT WWT Be. By incorporating a recognized principle about the expectation of Gaussian vector's 's quadratic form, we obtain:
E||W T Bel =EleT B WW BE] WW BΣ).
(3)
16

---

Assuming WWT = I, an identity matrix, the trace simplifies to tr(B' BE). In the event the model has undergone orthogonality regularization, resulting in a semi-orthogonal head W (precisely, WWT Π) we obtain a concise formulation for the penalty that hinges solely on the backbone and the fundamental noise parameter.
F.2 Insights from Theoretical Analysis
The insights gained from our theoretical analysis significantly enrich our understanding of the FedSTO approach and how it can navigate challenges specific to FL.
One major challenge FL data heterogeneity - the data across different clients (or devices) can vary significantly. By bounding the MSE for our model under slightly shifted distributions, we learn how our model's performance could change in the presence of such data heterogeneity. This is akin to having a 'sensitivity" measure for our model, showing us how 'sensitive" the model is to changes in data distributions. The better we understand this sensitivity, the more effectively we can tailor our model to handle the challenges of federated learning.
The analysis also highlights the significance of having orthogonal weights in our model. The investigation reveals that when the head of our model is semi- semi-orthogonal, the penalty on the out-of- sample MSE simplifies depend only on the backbone and the noise term, effectively isolating the effect of elient-specifer data perturbations. This insight is of particular importance because it provides clear and quantifiable understanding of the role and benefit of orthogonal weights in controlling model sensitivity to data heterogeneity.
Furthermore, our analysis provides valuable directions for future research in FL.. The relationship discovered between the orthogonality of weights and the bound on MSE opens up new avenues for investigation. It could, for instance, prompt more in-depth research into regularization techniques that aim to achieve orthogonality, improving model stability across diverse data distributions.
In simpler terms, theoretical analysis is much like 'roadmap', helping understand how our model reacts to changes data distributions, the role of orthogonal weights in this context, and where could focus future research efforts to improve approach. This roadmap makes the journey of applying FedSTO to federated learning more navigable, ultimately leading to more effective and robust models.
G Detailed Experimental Settings
YOLOv5 Architecture The YOLOv5 object detection model is based on the YOLO algorithm. which divides an image into a grid system and predicts objects within each grid. YOLOvS uses a convolutional neural network (CNN) to extract features from the image and then uses these features to predict the location of the bounding boxes (x.y,height, The YOLOv5 architecture consists of three main parts: the backbone, the neck, and the head. The backbone is responsible for extracting features from the image. The neck combines the features from the backbone and passes them to the head. The head then interprets the combined features predict the class of an image. Here a more detailed description of each part of the YOLOY5 architecture:
the scores, and the objects classes.
Backbone:
The backbone of YOLOv5 is based on the CSPDarknet53 architecture. The CSPDarknet53 architecture a modified version of the Darknet53 architecture that uses cross stage partial connection (CSP) module to improve the performance of the network. The CSP module allows the network learn more complex features by sharing information across different layers.
Neck: The neck of YOLOv5 is a simple convolutional layer that combines the features from the backbone.
Head: The head of YOLOv5 is composed of three convolutional layers that predict the location of the bounding boxes, the scores, and the objects classes. The bounding boxes are predicted using regression model, while the scores and classes are predicted using a classification model.
17

---

YOLOv5 has been shown to be effective for object detection, achieving good results on variety of object detection benchmarks. It also fast, making a good choice for real-time object detection applications.
Annotations The YOLO object detection algorithm divides an image into grid system and predicts objects within each grid. The annotations for YOLO are stored in text file, with each line representing object in the image. The format of cach line is as follows:
[object_id]] [x_ x__eenter,, ly_center], [width], [height], [score]
where object_id is the ID of the object, x_center is the x-coordinate of the center of the object's bounding box. y_center is the y-coordinate of the center of the object's bounding box, width the width of the object's bounding box, height is the height of the object's bounding box, and score is the confidence score of the object detection.
Loss Functions YOLOv5 is an object detection model that is trained using a loss function that combines three terms: a class loss, box loss, and confidence loss. The class loss is cross-entropy loss that measures the difference between the predicted class probabilities and the ground truth class labels. The box loss is smooth LI loss that measures the distance between the predicted bounding boxes and the ground truth bounding boxes. The confidence loss is a binary cross-entropy loss that measures the difference between the predicted confidence scores and the ground truth binary labels indicating whether or not an object is present in the image. The overall loss function minimized using stochastic gradient descent. Here is more detailed explanation of each loss term:
Class loss Lats: The cross-entropy loss a measure the difference between two probability distributions. In the case of YOLOv5, the two probability distributions are the predicted class probabilities and the ground truth class labels. The cross-entropy loss minimized when the predicted class probabilities are identical to the ground truth class labels.
Box loss Loby: The smooth L1 loss is measure of the distance between two sets of numbers. In the case of YOLOvv, the two sets of numbers are the predicted bounding boxes and the ground truth bounding boxes. The smooth I.1 loss is minimized when the predicted bounding boxes are identical to the ground truth bounding boxes.
Confidence loss Cionf: ; The binary cross-entropy loss a measure of the difference between two binary distributions. In the case of YOLOv5, the two binary distributions are the predicted confidence scores and the ground truth binary labels indicating whether or not an object is present in the image. The binary cross-entropy loss is minimized when the predicted confidence scores are identical to ground truth binary labels.
H Discussions on Semi-Supervised Object Detection
We implement similar strategy the pseudo label assigner of the semi-efficient teacher 38 We aim to provide a comprehensive description of our training method and pseudo label assignment approach in this section.
H.1 Adaptive Loss
The adaptive loss function in our SSFOD framework is comprised of two key parts: supervised loss (L,), calculated from labeled data, and an unsupervised loss (L.), generated from unlabelled instances.
The supervised loss, I.,, adheres to conventional practices used object detection tasks. This incor- porates the amalgamation of cross-entropy loss, responsible for classification, and CloU (Complete Intersection over Union) loss, accounting for bounding box regression:
(hiw)) L, Σ [CE(X") (h,ws (h.w)) C'oUU CX (h.w), y(hhwl) CEEX ob x h
.Ydss
(4)
18

---

where CE is the cross- cross-enropyy loss function, (hh,r) signifies the model's output, and ((())) corresponds to the sampled results of pseudo label assigner (i.e., local EMA updated model).
The unsupervised loss component, Tuu, is an addition that leverages the pseudo labels generated by the local EMA updated model. 'The objective of Lu is to guide the model exploit beneficial information embedded within unlabelled data. It is computed as:
L. Las + Lrey + Loby"
where Las Leg and Labj denote the losses associated with classification, bounding box regression, and objectness score, respectively. Each of these losses uses pseudo labels and is precisely tailored to foster reliable and efficient learning from unlabeled data.
Classification Loss, Ly hones the model's ' class-specific accuracy. It is calculated only for pseudo labels whose scores are above predefined high threshold T2, using cross-entropy loss function. The divergence is gauged between the model's s predicted classification scores and the class scores from the pseudo labels provided by the Pscudo Label Assigner.
Regression Loss, I ere: scrutinizes the precision of the bounding box predictions. This loss applies to pseudo labels with scores above 72 or objectness scores surpassing a value typically set 0.99. A CloU loss function employed to measure the discrepancy between the predicted bounding box regressions and those associated with the pseudo labels.
Objectness Loss. Loby- evaluates the model's 's objectness prediction capability, essentially the model's s confidence in a given bounding box containing an object of interest. All pseudo labels contribute this loss, though the computation varies depending on the pseudo label score. For scores below 71, the loss is computed against zero. For those above T2, the loss compares against the objectness of the pseudo label. Scores between TI and Ta result in the loss calculated against the soft objectness score of the pscudo labels.
This strategic amalgamation of losses forms the underpinning of the unsupervised loss function, fostering efficient and reliable learning from unlabeled data. By harnessing the potential information embedded within pseudo labels, our model aims to significantly enhance semi-supervised object detection in an FL setup.
In centralized setting, the supervised and unsupervised losses would typically be trained jointly. However, our federated scenario, where the server possesses only labeled data and the client holds exclusively unlabeled data, we adopt an alternate training approach. The server focuses optimizing the supervised loss I,, with its labeled data, while the client concurrently refines the unsupervised loss L, using its unlabeled data. This methodical division of labor allows us to leverage the distinctive characteristics of both types of data and facilitates efficient learning federated environment.
H.2 Local EMA Model for the Pseudo Labeler
In SSFOD framework, we employ a Local Exponential Moving Average (EMA) model for generating pseudo labels, enabling the judicious utilization of unlabeled data at each client's 's disposal. The concept of the EMA model hinges on an infinite impulse response filter that assigns weightages decreasing exponentially, offering a balanced consideration of both historical and immediate data points for reliable predictions.
Fach client maintains a local EMA model in our method. The weights of this local EMA model are a weighted average of its own weights and the weights of the global model. This relationship can be described mathematically as follows:
W (t) EMAationck
(t((t-1)
+ (1 - ar) ((()
(6)
EMA.tienck
client
In Eq. (6). ((())
gMA. symbolizes the weights of client k's local EMA model round and (-(t)
client
represents the weights of the client k's model at round I. is the EMA decay rate dictating the pace of depreciation of the client's weights' influence. Typically, the decay rate resides the 0.9 to 0.999 range, subject to the specific application.
19

---

Table Performance on the BDD dataset with labeled server and 3 unlabeled clients. It highlights how each added method contributes to the overall performance under both Non-IID and IID condi- tions.
Non-IID
III
Method
Cloudy
Overcast
Rainy
Snowy
Total
Cloudy
Overcast
Rainy
Snowy
Total
Vanilla
0.560
0.566
0.553
0.553
0.558
0.572
0.588
0.593
0.610
0.591
Freezing Only Head
0,531
0.603
0.567
0.565
0.567
0.558
0.613
0.614
0.593
0.595
Freezing Nock & Head
0.571
0,583
0.557
0.556
0,567
0,576
0.578
0.594
0.599
0.587
In pursuit of consistent basis across all clients, the weights of the Local EMA model are reinitialized with the global model's weights post each server broadcast. This particularity our framework ensures harmonious coexistence of global model consistency and local data awareness.
The system enables the local EMA model of each client to gradually tune to the unique local data characteristics, by integrating updates from the client's 's unlabeled data. As consequence, we achieve a potent blend of personalization and model performance a FL. environment, the while alleviating communication overheads. The design acknowledges the distinct data distributions that each client may harbor, allowing the model to adapt accordingly, thereby augmenting the learning process's efficacy and efficiency.
Following the weight updates, the local EMA model functions the pscudo labeler for each client. This approach ensures the generation of stable and reliable pseudo labels, despite the limited interactions with the global model. the Federated Learning scenario, where communication costs carry substantial significance, this feature is crucial.
Our local EMA strategy presents several benefits. guarantees independent pseudo label generation at each client's end without necessitating frequent server updates. In addition, it enhances the learning process's robustness, the local EMA model remains largely unaffected by the client model's noisy gradient updates. As result, the system produces more trustworthy pseudo labels, ultimately culminating in improved performance.
I Discussions on Selective Training
In our SSFOD framework, Selective Training plays pivotal role in achieving superior performance and enhanced computational efficiency. Traditionally, in selective training, all components beyond the backbone, typically referred as -on-Backbone parts, are frozen reduce computational complexity and improve training efficiency. However, It may not be necessary to freeze all of the non-backbone parts.
In our preliminary investigations, we discover that freezing only the head, a component of the Non- -on-Backbone parts, can yield similar performance levels as freezing the entire observation opens the door to potentially more efficient and flexible models in the context of selective training, it suggests that selective freezing could be just as effective complete freezing Table 9p.
This
Nonetheless, in consideration of the inherent nature of FL, we ultimately decide to continue freezing both the neck and head components. We reach this decision primarily due to the communication cost considerations, which carry considerable weight in a federated learning setting. Despite the promising results observed when only freezing the head, freezing both the neck and head does not result in any noticeable performance decrement, while does markedly reduce the communication cost.
The implications of this choice are particularly relevant in the context of FL environments, where minimizing communication costs is paramount for practical deployment. We posit that this finding may inform future strategies for selective training, suggesting that selective freezing of specific Non- Bac-Baacbon parts can effectively balance performance and efficiency in FL environments.
20

---

Table 10: Performance under random distributed cases of Cityscapes FedSTO exhibits improve- ments under various object categories, and significantly outperforms the performance for unlabeled clients.
Labeled
Unlabeled
Method
Categories
Person
Truck
Traffic Sign
Person
Traffic Sign
ORN from scratch Freezing only Head + FPT with ORN
0.470
0.694
0.449
0.074
0,393
0.437
0.720
0.507
0.100
0.378
0.504
0.697
0,353
0.239
0.411
0.486
0.740
0.420
0.078
0.415
Proezing Nock & Head FPT with ORN
0,504
0.720
0.342
0.261
0,415
0,487
0.740
0,460
0.181
0.437
J Further Discussions on Full Parameter Training (FPT) with Orthogonal Enhancement
As part of our ongoing evaluation of our SSFOD framework, we conduct comprehensive experiments involving different training strategies, in particular, Full Parameter Training (FPT) with Orthogonal Regularization (ORN). Through these experiments, we compare three distinct approaches:
1. ORN employed right from scratch without any pre-training or frozen components.
2. A two-stage process where the model head is frozen during the pre-training phase, followed by fine-tuning across the entire model with ORN.
3. A more nuanced strategy where both the neck and head components of the model are initially frozen, subsequently transitioning to ORN-driven FPT.
Our empirical results decisively point the superiority of the third approach (Tale 10P. Despite the initial freezing of the neck and head components, yields performance that surpassed the other two strategies. This counter-intuitive finding underscores the effectiveness of the careful balancing of selective parameter freezing and ORN application.
The mere marginal performance difference in mAP between the second and third approaches suggests room for further investigation. An exploration the specific conditions under which one might outperform the other could shed light new avenues for model improvement. This represents an exciting direction for future research.
However, for the scope of the present study, we opt for the third strategy of freezing both the neck and head components simultaneously. This decision not only influenced by the slightly higher performance but also by the added advantage of reduced communication costs. This approach resonates with the primary objective of our research to realize nigh-performing efficient, and cost-effective FL. frameworks for SSOD tasks.
K
Examination of Loss Functions within SSFOD Framework
An integral part of our study the SSFOD framework involves examination of the influence of various loss functions on model performance. One of the notorious challenges in object detection tasks is dealing with class imbalance. To counter this, decided to evaluate the effectiveness of Focal Loss, which is widely used for its capability to handle such imbalances, in comparison with the conventional Cross-Entroopy (CE) Loss, which refer vanilla Loss in the paper.
Focal Loss 24 has been designed to tackle class imbalance by reducing the contribution of casy instances, thus allowing the model to concentrate on challenging, misclassified samples. Despite its recognized efficacy in a variety of settings, we observed unexpected performance under our specific framework.
Contrary to our initial assumptions, our empirical findings suggest that the Vanilla Loss proved to be more potent safeguard against the class imbalance issue within our SSFOD framework (Table I1). We compare the performance by changing the loss functions the baseline SSFOD training. The underlying reasons for this surprising result are potentially multifaceted and certainly merit additional exploration. However, these observations underscore the importance empirical testing in choosing the appropriate loss functions tailored for specific frameworks and tasks.
21

---

Table 11: Performance of CE loss and focal loss on the BDD dataset with 1 labeled server and unlabeled clients. It highlights how each added method contributes to the overall performance under both Non-IID and IID conditions.
Non-IID
IID
Method
Cloudy
Overcast
Rainy
Snowy
Total
Cloudy
Overcast
Rainy
Snowy
Total
CE loss
0.560
0.566
0.553
0.553
0.558
0.572
0.588
0.593
0.610
0.591
Focal loss
0.364
0.462
0.438
0.446
0.428
0.371
0.464
0.469
0.482
0.447
L Implementation
To provide a comprehensive overview of our study, we provide the details of our implementa- tion, including the reproduction of various existing FL methods such as FedDyn [1 FedBN [22]. FedOpt [3], and FedPAC (39].
FedDyn [ For the implementation of FedDyn, we adhere strictly the training methodol- ogy as stipulated in the original paper.
FedBN [2]: In implementing the FedBN method, we make a conscientious choice to leave out the statistics and bias of the batch normalization (BN) layer from the aggregation step.
FedOpt K3 FedOpt implementation aligns with Vanilla SSFL,, albeit with slight mod- ification in the server model's optimization procedure. The key differentiation lies in the utilization of pscudo learning rate of 0.9 during server model optimization, which was chosen through empirical evaluations to maximize performance. Thus, while preserving the core attributes of the FedOpt framework, we adapt it our specific task requirements.
FedPAC 39 The execution of the FedPAC framework, primarily architected for classifi- cation tasks, necessitates thoughtful adaptation suit our object detection scenario. This adaptation involves the appropriation of the YOLOv5 head, used as stand-in classifier, which allows us to transpose our specific needs onto the FedPAC blueprint. This process not only honors the training techniques prescribed by FedPAC but also satisfies our model's unique requirements. As an integral part of our approach, the local EMA model, a cor- nerstone of our broader learning strategy, is employed as a pscudo labeler in this scheme. Furthermore, our model's 's backbone representation is equated with the output from YOLO's backbone. This bespoke modification offers a perspective to exploit the central principles of FedPAC while tailoring them to suit the idiosyncrasies of our model.
22

---

M Experimental Results with 100 Clients
In the setting of extensive FL environment with a network of 100 clients and 0.1 client sampling ra- tio, we embark on an empirical investigation to ascertain the robustness and scalability of the proposed method. This scenario closely mirrors real- world cross-device situations, inherently characterized by widespread client distribution. Intriguingly, each client in our experiment is associated with data corresponding to a single weather condition. Notwithstanding this, our method exhibits remarkable resilience and efficacy. As illustration, the model outperforms baseline that is trained solely on server's labeled data, as demonstrated in (Table 12 This superior performance underscores the model's capability in effectively leveraging the heterogeneity intrinsic distributed data, thereby enhancing the overall performance. These empirical findings provide compelling evidence of the adaptability and effectiveness of our approach within large-scale FL contexts. With its ability to maintain high performance coupled with computational efficiency, our method exhibits promising potential in managing data heterogeneity across extensive federated networks.
Table 12: Performance under Non-IID scenarios of BDD100k dataset with server and 100 clients. FedSTO exhibits improvements under various object categories, and significantly outperforms the performance for unlabeled clients. The term 'Server Only Scenario' aligns with the notion of 'partially supervised" in CL settings.
Labeled
Unlabeled
Method
Categories
Car
Bus
Truck
Trafiic Sign
Person
Truck
Traffic Sign
Server Only Scenario
0.378
0.710
0.141
0.425
0.490
0.337
0.707
0.160
0.338
0.491
FedSTO within SSFOD Framework
0.393
0.714
0.442
0,510
0.540
0.487
0.738
0.573
0.589
0.617
23


Reviews:
[{"review_id": "qXLilC03Lh", "reviewer": "Reviewer_2CR4", "summary": "This paper presents a Semi-Supervised Federated Object Detection (SSFOD) framework featuring a two-stage training strategy, FedSTO, designed to address the challenges of heterogeneous unlabeled data in federated learning. The proposed framework employs selective training and orthogonality regularization with personalized EMA-driven pseudo-labeling to facilitate robust and diverse feature learning, enhancing object detection performance across multiple weather conditions and data distributions. The empirical results provide evidence of the merits of FedSTO over existing federated and semi-supervised learning methodologies. Notably, despite non-IID clients having no labels, FedSTO achieves performance comparable to fully supervised centralized models.", "questions": "Please see the weaknesses mentioned above.", "limitations": "The paper does not have an explicit section for \"limitations and negative social impact\", but that is not a major concern for the research topic studied in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe paper introduces a solution for SSFL for object detection with a one-stage detector in non-IID weather conditions. \n•\tThe paper outlines a new approach to SSFOD, combining existing methods and adapting them to the federated learning setting.\n•\tThe paper is well-structured, has clear headings and subheadings, and is easy to follow.", "weaknesses": "1. The inclusion of a federated setting with fully labeled data on at least one dataset would have provided a valuable comparison to the proposed approach. Also, while the comparison with the partially supervised baseline helps establish the lower bound of server pretraining, it is not a fair comparison due to the significant difference in training data volume.\n2. Although the models are evaluated on their respective datasets, it would have been beneficial to assess their performance on a global test set as well. Context-specific evaluations are necessary, but examining the models' generalization can help avoid the issue of overly specialized clients, which is a concern in personalization.\n3. An interesting analysis would have been to investigate how the performance gap of the proposed method changes with increasing amounts of data. The reported performance of the Fully Supervised Centralized Yolo-v5 Large in the paper is notably lower compared to other papers on full-scale datasets (e.g., YoloV5s achieves 77.2 in https://arxiv.org/pdf/2108.11250v7.pdf). This raises the question of whether the low performance and the closure in performance with the centralized setting is due to the limited training data. It would be valuable to compare the proposed approach against these scenarios with more data available or if possible employing a full server pretraining.\n4. Regarding Cityscape, since the dataset does not provide precise weather information for each annotation, the data is distributed uniformly at random, and it's not clear how it is non-IID. Also, the non-IID aspect mentioned in the paper is limited to addressing the skew in feature distribution induced by weather variations. The authors could have utilized foggy-Cityscape and KITTI datasets o obtain a more realistic non-IID setting. \n5. The proposed SSFOD problem setting is similar to unsupervised domain adaptative object detectors and test-time domain adaptation. Even, the ingredients of the proposed solution such as EMA are also known in the semi-supervised OD and domain adaptation literature.  To me, the only difference is that the model parameters are updated on local clients instead of central weight update.   Please clarify the differences and advantages of the proposed solution compared to similar domain-adapted object detection methods. \n6. Many key components such as FPT with orthogonal regularization are adapted from existing literature such as [16].  Although I agree that the proposed problem setting is unique and novel, it is also important to clarify the novel technical contributions of the proposed solution. \n7. What is the rationale behind using Yolov5?  It will be beneficial to show that the proposed solution is generalizable on other families of object detectors such as Faster-R CNN and recent transformer-based object detectors such as DETR/Deformable DETR. \n8. mAP@0.5 is not an ideal evaluation metric to qualify localization capabilities. Please report mAP@0.75 and COCO-style mAP for a better understanding of the precise localization capabilities of the model. \n9. EMA, pseudo-label assignment on unlabelled data needs to be better explained. What are the training loss functions used for backbone weight update during the selective training step? \n10. It is mentioned that the proposed solution uses augmentations such as Mosaic, left-right flip, large-scale jittering. But, it is not clear how are the corresponding ground truth box positions determined in the case of unlabeled images.\n11.  As the authors mention that SSFOD introduces additional computational overhead, it is desired to quantify the computational resources employed. \n12. The paper contains some errors, such as a missing dot in Line 139 and the use of \"IID\" in Table 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback in helping refine our work.\n### W1. Results of fully labeled FL & W8. mAP@0.75\n- As you suggested, we added fully-supervised IID and non-IID FL in Table A of our Global response's PDF as well as comprehensive evaluations using mAP@0.75. These results demonstrate that our FedSTO approach achieves competitive performance even when it only has access to 25% of the labels.\n- The partially supervised baseline establishes lower bounds of server pretraining and shows utility of unlabeled data. We agree that the strength of FedSTO is highlighted by 3 other baselines — significant improvements against vanilla SSFL and close performance to fully-supervised centralized and FL results.\n\n### W2. Global performance\n- We concur in analyzing the generalizability of our models to other data distributions. Table E of our Global response's PDF shows FedSTO's competitive global performance.\n\n### W3. Lower performance compared to the previous work\n- Upon inspection, the Yolov5's 0.772 in your referenced paper (https://arxiv.org/pdf/2108.11250v7.pdf) is only for the Car class. As listed in Table 4 of our paper, our centralized model reaches 0.788 and FedSTO reaches 0.740 for the Car class. Thus, our paper's performance is comparable (especially since we only have access to 25% of the labels).\n\n### W4. Non-IIDness on Cityscapes\n- Your observation is correct that Cityscapes was distributed uniformly at random (stated in  line 260 of our main text). As you suggested, we would like to incorporate experiments with Foggy Cityscapes, space permitting. As our paper pioneers the exploration of SSFOD, we chose to demonstrate the efficacy of our approach on standard settings of datasets and anticipate future extensions to more nuanced settings.\n\n### W5. Differentiation from previous domain-adapted object detection (OD) methods\n- Although inspired by past unsupervised domain adaptation methods, there are several major differences in our work. A novel challenge we address is the separation of labeled and unlabeled data. Conventional domain-adapted OD methods use labeled data instances to conduct SSL. Our scenario, where each client only holds unlabeled data and cannot access labels, presents a unique and complex problem.\n- To overcome this, FedSTO uses personalized pseudo labelers and orthogonal enhancements:\n  - Go against conventionally used global models and demonstrate that local EMA models are stronger pseudo labelers.\n  - Stabilize training with only unlabeled data by using the alternate server and local pseudo labelers.\n  - Introduce warm-up and alternate training phases which yield personalization and generalization.\n  - Mitigate instability from training exclusively with unlabeled data by employing selective training.\n- Theoretical analysis is included in the Appendix.\n\n### W6. Novel technical contribution\n- Although we build on prior research, their context and application differ significantly. A detailed theoretical explanation is in **Section F of our Appendix**. The greater challenge in using single-stage detectors, along with strategic loss application, signifies novel technical contributions.\n- For example, even though the penalizing loss may appear similar to existing literature such as [16] cited in our main paper, the intent behind its usage differs vastly. The focus of [16] was to enhance DNNs by applying across various layers of the architecture. In contrast, we have strategically applied the loss to the neck and head of our architecture with a specific aim: robustness of detection quality to heterogeneities among client datasets. Similarly, components like FPT with orthogonal regularization were primarily tested on image classification tasks, whereas we have adapted it to the more challenging domain of object detection.\n\n### W7. Rationale behind Yolov5\n- Given our problem setting's novelty, we focused on single-stage detectors as they jointly achieve high performance, faster inference times, and small model sizes, which are crucial for real-world applications such as autonomous driving. Among single-stage detectors, Yolo models are a de facto standard. In addition, the nature of FL demands a focus on models that do not incur excessive communication costs. The suggested models, unlike Yolo, do not necessarily satisfy the fast inference or low communication cost criteria. \n\n### W9. EMA pseudo label assignment + training loss\n- Thank you for your interest in the pseudo label mechanism and training loss. Due to space constraints, we provide explanations in **Appendix G and H**. As a brief summary, EMA constructs an enhanced local model that smoothens fluctuations during model updates. The EMA-based model serves as a local pseudo labeler to label unlabeled data held by each client. This labeling is essential for performing learning tasks on the unlabeled data. We employ the combination of objectiveness, bounding box, and classification losses included in the standardized Yolov5 loss. We train only on datapoints whose pseudo labels have a high confidence score.\n\n### W10. Ground truth boxes for augmented unlabeled images\n- Ground truth box positions for unlabeled images are determined through predictions provided by our EMA pseudo labeler. They are subjected to data augmentation in parallel with image data. Thus, we generate augmented image label pairs from unlabeled images, which are then used for further training.\n\n### W11. Computational resources\n- In our study, while SSFOD has inherent complexities due to pseudo labeling, it did not significantly increase memory use compared to fully-supervised FL; we utilized the same V100 GPU throughout. Furthermore, FedSTO reduces communication costs by 20.52% by freezing the model's neck, as seen in Table D of our response pdf. Going forward, we will aim for an optimal balance between performance and efficiency.\n\n### Limitations\n- Due to space constraints, potential limitations and societal implications are in the Appendix."}, {"review_id": "jLIvQAM5mU", "reviewer": "Reviewer_qq85", "summary": "This paper explores Semi-Supervised Federated Object Detection (SSFOD), a pioneering framework for distributed data sources with limited high-quality labels and non-IID client data, particularly in applications like autonomous driving. The authors present a two-stage strategy, FedSTO, encompassing Selective Training followed by Orthogonally Enhanced full-parameter training, to address data shift while representing the first implementation of SSFOD for clients with 0% labeled non-IID data. The proposed approach includes selective refinement of the detector backbone to avert overfitting, orthogonality regularization to enhance representation divergence, and local EMA-driven pseudo label assignment to produce high-quality pseudo labels. ", "questions": "see weaknesses", "limitations": "The authors did not discuss limitation and ethical issues in the main body.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides helpful figures, explores the valuable direction of semi-supervised Federated Object Detection (SSFOD), and presents an effective two-stage strategy called FedSTO for addressing data shift in a distributed data source. The proposed approach achieves state-of-the-art results in multiple datasets. Additionally, the paper provides a clear problem statement that is easy to understand.", "weaknesses": "However, there are a few areas for improvement before the paper can be considered for publication. First, the references list is incomplete as some essential references are missing, such as \"Federated learning with label distribution skew via logits calibration.\" ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your comments below.\n### W1. Additional references  (such as `Federated learning with label distribution skew via logits calibration')\n- Thank you for pointing out the omission in our references list and for suggesting the inclusion of \"Federated learning with label distribution skew via logits calibration.\" We appreciate your recommendation and acknowledge the significance of this work in the domain of federated learning.\n- However, we would like to kindly note that while both our research and the suggested reference operate within the realm of federated learning, the specific challenges and methodologies they address are quite distinct. The cited paper delves into addressing label heterogeneity in federated image classification through logits calibration. In contrast, our work primarily targets federated object detection, placing particular emphasis on weather-conditioned heterogeneity. Moreover, the inherent complexity of label skewness in object detection – which encompasses annotation, objects per image, and class heterogeneity – sets it apart from image classification. The nuances of the problems and the intricate differences between the two contexts may not make the suggested reference directly applicable to our study.\n- Nevertheless, we recognize the value of drawing connections between related yet distinct works in federated learning. Should we be granted additional pages for the camera-ready version, we are fully committed to incorporating the recommended reference into **Section 2.1 \"Federated Learning (FL): Challenges and Advances\"**, highlighting its relevance and distinction from our approach.\n- Once again, thank you for your valuable feedback and suggestions. We appreciate the time and effort you have dedicated to reviewing our paper.\n\n### Limitations and ethical issues are not discussed\n- Thank you for pointing out the lack of an explicit \"limitations and negative social impact\" section in the main body of the manuscript.\n- Due to space constraints, and in an effort to maintain the coherence of the main text, we have extensively discussed the potential limitations and negative social implications **in the Appendix**. We understand the importance of addressing such concerns in contemporary AI research and have made a dedicated effort to ensure that these topics are addressed in detail, albeit in the supplementary section.\n- We believe that it is essential for readers and practitioners to be aware of the possible pitfalls, limitations, and societal implications of the proposed methods. We hope that the provided discussion in the Appendix serves this purpose. In subsequent versions of this paper, we'll strive to make such sections more prominent by referencing it in the main text. Thank you for your understanding and feedback."}, {"review_id": "z6VGK60laf", "reviewer": "Reviewer_8JqK", "summary": "This work focuses on a practical application of federated learning, federated semi-supervised learning for object detection. It assumes that the server has labeled data and the clients only have unlabeled data. The proposed method is two-fold: selective training and orthogonal enhancement.", "questions": "- Line 3 of the Algorithm conducts client sampling, while it seems that the experiments do not conduct client sampling.\n- The server and client are assumed to be from the dataset. What would be the impact if the dataset in the server and clients are not from the same dataset? It would be more practical as we are unable to assume that we can collect the data in a server similar to the clients.", "limitations": "Limitations are not discussed in the paper. It could contains some of the aspects such as the scale of the experiments.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper integrates multiple existing techniques for federated semi-supervised learning. It is technically sound.\n- The paper is generally well-written, clear, and easy to follow.\n- Experiments on three object detection datasets demonstrate the effectiveness of the proposed method.", "weaknesses": "- The novelty is somewhat limited. The novelty lies in the integration of existing methods and making them work on the target use case, while the key algorithms are more or less adopted from existing works.\n- The evaluation scale can be extended to a larger number of clients. The majority of the experiments are run with 3 clients and only one experiment is run with 20 clients. The autonomous driving use case is more like cross-device FL. It’s important to evaluate on more clients with client sampling.\n- Some existing works with related techniques are not discussed: [1] and [2] fixed the head and only trains the backbone like the selective training. [3] and [4] also uses EMA in local training.\n- Since the first stage is to learn representation, a comparison could be done with federated self-supervised learning methods [3][4] for learning visual representations.\n- The organization of section 3 is not very straightforward. For example, 'Personalized Pseudo Labeling for Unlabeled Clients' is more suitable in Section 4 instead of problem statement.\n\n[1] Fedbabu: Towards enhanced representation for federated image classification, ICLR’22. \n\n[2] Spherefed: Hyperspherical federated learning. ECCV’22\n\n[3] Collaborative unsupervised visual representation learning from decentralized data. ICCV’21\n\n[4] Divergence-aware federated self-supervised learning. ICLR’22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. We address each below.\n### W1. Novelty\n- We would like to respectfully disagree with the assertion that the novelty of our work is limited. When trying to naively apply existing techniques to the SSFOD setting, we observed notably poor performance, which led to our novel formulations. As we have pointed out in our paper, related FL research tends to focus on image classification and small image sizes, like in CIFAR. In contrast, our work tackles object detection, which is a more complex problem with larger and noisier images.\n- To elaborate on the novelty in the general response, we propose the first SSFOD framework, where the server has labeled data while clients only have unlabeled data. To the best of our knowledge, this is a novel problem formulation that has not yet been addressed in the literature. Rather than simply predicting classes in image classification, pseudo labeling in object detection involves predicting ground-truth boxes, applying non-maximum suppression, and assigning class labels to the predicted boxes. Because each step is intricate, constructing and training a useful pseudo labeler—especially in the presence of data heterogeneity—is far from simple (theoretical and ablation analysis is provided in the **Appendix**).\n- In the centralized learning setting, semi-supervised object detection usually refers to labeled data to correct the training of unlabeled data and perform domain adaptation with every batch. However, in our federated learning scenario, the clients are assumed to have no access to labeled data. Simply applying existing techniques in this context would result in suboptimal and unstable performance (**Alternate Training part in Table A** of the attached pdf & **Table 1 in the main paper**). We believe FedSTO is a novel contribution and hope that this explanation will reinforce this viewpoint.\n\n### W2. Evaluation on a larger number of clients, client sampling & Q1. Client sampling\n- We agree that it is crucial to test our framework in a setting with a larger number of clients. To support this, we have conducted experiments with 1 server and 100 clients, as detailed in **the Appendix Section M & Table C in the attached pdf**.\n- We would like to stress that our experiments are not merely theoretical or synthetic. They are conducted in a real-world setting, with genuine network communications occurring between 100 AWS virtual machines. We do this to underscore the practicality and applicability of our findings, and to convince the community that our proposed approach to personalized, semi-supervised FL is viable.\n\n### W3. Discussions for related techniques\n- Thank you for highlighting related works [1]-[4]. Regarding the freezing in [1] and [2], we have evaluated similar methods, especially those in \"Personalized Federated Learning with Feature Alignment and Classifier Collaboration, ICLR 2023.”, which emphasizes enhancing feature alignment through freezing the head. Though similar, they focus on image classification while our work addresses the unique challenges of object detection. We list more results and the discussions of the ablation study about these methods [1, 2] in **Table 2, Appendix Section I & Appendix Table 7**.\n- For the EMA in local training as in [3] and [4], we acknowledge the similarities in terms of leveraging EMA models. However, while [3] and [4] seem to be inclined towards using pseudo labels or representations directly from the EMA model in the image classification task, our method involves generating annotations after the non-maximum suppression step using the EMA model's predictions in the object detection task. \n- **Given acceptance, we'll incorporate the references you provided in our final version.** This will certainly strengthen the connections to related methods.\n\n### W4. 1st stage for self-supervised learning\n- Thank you for pointing out the potential comparison with federated self-supervised learning methods [3], [4]. While our approach has similarities with conventional self-supervised methods, our main focus was to progressively develop SSFOD, given the nascent state of research in this topic. We recognize the value of the cited works; however, they primarily target image classification. Directly applying their techniques to federated object detection introduces unique challenges not present in their original context. We are nonetheless enthusiastic about exploring a federated self-supervised backbone in our future research, and appreciate your insightful suggestion.\n\n### W5. Not Straightforward Organization of section 3\n- Thank you for your keen observation on the organization of Section 3. We concur that \"Personalized Pseudo Labeling for Unlabeled Clients\" plays a pivotal role in our problem formulation. Given its significance, especially when juxtaposed with \"data heterogeneity\", we believe its initial discussion is warranted in Section 3. Our proposal is to retain an introductory segment on personalization in Section 3 and defer the deeper, algorithmic details to Section 4. This restructuring aims to strike a balance, addressing the problem's essence while streamlining technical content.\n\n### Q2. Server and Client have different dataset\n- Thank you for the insightful comment. We concur that the server and client data from differing datasets introduce a valuable layer of heterogeneity.\n- Our current focus revolves around addressing complexities like weather-induced feature distribution skew and label density heterogeneity, particularly in applications like autonomous driving, detailed in **Appendix Section E**.\n- While the scenario you highlight—which could encompass out-of-distribution data or introduce new classes—is not directly addressed in our work, we anticipate our methods, such as personalized pseudo labeling, might provide some adaptability. However, this is a rich area warranting further exploration. In the future work, we will explore these scenarios more comprehensively. "}, {"review_id": "drb0UlvcNf", "reviewer": "Reviewer_AM8c", "summary": "This paper introduces a novel framework called Semi-Supervised Federated Object Detection (SSFOD) to tackle the problem of object detection in a federated learning setting. In this framework, the server possesses labeled data, while the clients hold unlabeled data from different distributions. The proposed approach consists of two stages: selective training and orthogonal enhancement. In the selective training stage, the focus is on updating only the backbone parameters on the clients to establish a robust backbone for the object detector. This selective approach helps improve the model's generalization capabilities across different distributions. The orthogonal enhancement stage follows, where all parameters are fine-tuned with orthogonal regularization. This regularization promotes representation divergence and robustness, further enhancing the model's performance. The paper also introduces a personalized pseudo label assigner based on a local exponential moving average (EMA) model. This assigner generates high-quality pseudo labels for object detection tasks, facilitating the training process in the semi-supervised setting. To evaluate the proposed SSFOD framework, this paper conducts experiments on three datasets: BDD100K, Cityscapes, and SODA10M. The results demonstrate that the proposed method achieves state-of-the-art performance when compared to existing approaches in both semi-supervised and federated learning domains.", "questions": "Please refer to the Weakness.", "limitations": "Please refer to the Weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, This paper is well-motivated. In practical applications, not all data on clients are labeled, and how to leverage unlabeled data is important for FL.  \n2, The writing is clear and easy to follow.  \n3, The paper performs extensive experiments on three diverse datasets, encompassing varying scales, complexities, and domains. Moreover, the proposed method is compared against multiple baselines as well as state-of-the-art techniques. The experimental results consistently demonstrate improvements across different metrics, object categories, weather conditions, and data distributions. This comprehensive evaluation reinforces the effectiveness and robustness of the proposed method, highlighting its superiority over existing approaches in various scenarios.  \n", "weaknesses": "1, This paper lacks discussion on the communication efficiency and scalability aspects of the proposed method, which are important considerations for its practical implementation. Specifically, it does not address the communication overhead associated with uploading only the backbone parameters or utilizing local exponential moving average (EMA) models. Furthermore, the paper does not investigate the performance of the method as the number of clients or the size of unlabeled data increases.  \n2, It could be better to compare or relate the proposed method with existing works on semi-supervised or self-supervised FL [1,2,3,4]. What are the advantages of the proposed method over these existing methods? \n[1] Zhuang, Weiming, Yonggang Wen, and Shuai Zhang. \"Divergence-aware federated self-supervised learning.\" arXiv preprint arXiv:2204.04385 (2022).\n[2] Zhang, Fengda, et al. \"Federated unsupervised representation learning.\" arXiv preprint arXiv:2010.08982 (2020).\n[3] Wu, Yawen, et al. \"Federated contrastive learning for volumetric medical image segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2021.\n[4] Dong, Nanqing, and Irina Voiculescu. \"Federated contrastive learning for decentralized unlabeled medical images.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1. Communication efficiency and scalability aspect + # of clients and size of unlabeled data increases\n- We thank the reviewer for this thoughtful comment. Communication efficiency and scalability are indeed crucial considerations for any FL implementation. In our work, we are mindful of these aspects, and we have attempted to address them in the following ways:\n\n1. **Communication Efficiency**: Our method communicates only the backbone parameters rather than the entire model, substantially reducing communication payload. As such, our selective communication. As the backbone typically constitutes a fraction of the model parameters, selective communication indeed reduces overhead. We have uploaded the relevant costs **in the attached pdf of the general response (Table D)**.\n2. **Local EMA models**: The utilization of local EMA models helps mitigate communication constraints as it enables each client to independently generate pseudo labels. This reduces the need for frequent server updates, further improving communication efficiency.\n3. **Scalability**: We appreciate your insight, and we agree that it is crucial to test our framework in a setting with a larger number of clients. To support this, we have conducted experiments with 1 server and 100 clients, as detailed in the supplementary material (**Appendix Section M**). The results presented in the appendix were obtained with a client sampling ratio of 0.1. Additionally, we put more results in **Table C in the attached pdf**.\n - We would like to stress that our experiments are not merely theoretical or synthetic. They are conducted in a real-world setting, with genuine network communications occurring between 100 AWS virtual machines. We do this to underscore the practicality and applicability of our findings, and to convince the community that our proposed approach to personalized, semi-supervised FL is viable.\n- As for detailed discussions and evaluations regarding these aspects, due to space limitations in the main paper, we have allocated this content to the **Appendix**. But, if extra pages are granted for the camera-ready version, we will certainly incorporate more details about these practical aspects in the main text.\n- We hope this clarifies the reviewer's concern, and we appreciate the suggestion to focus more on these practical aspects. It will undoubtedly make our work more robust and closer to real-world applicability.\n\n### W2. What are the advantages of the proposed method?\n- We appreciate the reviewer's suggestion to compare and relate our method with existing works on semi-supervised or self-supervised FL. Indeed, these are relevant and important works in the domain of federated learning. Here is how our method differentiates and improves over these works:\n1. **Domain-Specificity**: Our work primarily targets object detection tasks in the FL setup, specifically focusing on the challenges presented by real-world scenarios such as autonomous driving. While the works suggested primarily concentrate on image classification or medical image segmentation, we are operating in a different problem domain with its unique challenges.\n2. **Methodological Advances**: Our SSFOD (Semi-Supervised Federated Object Detection) framework proposes a novel federated semi-supervised learning approach tailored to object detection tasks. It integrates techniques like Personalized Pseudo Labeling, Orthogonal Enhancement, and Selective Communication to effectively leverage unlabeled data, enhance feature representation, and reduce communication overhead, respectively.\n3. **End-to-End Solution**: Our work presents an end-to-end solution: problem formulation, benchmark setting, improved methods, and experimental results. We first formalize the problem, set up a benchmark for semi-supervised federated object detection, and finally propose a novel method to solve it.\n\n- That said, the suggested works do offer valuable insights and techniques in their respective fields. Some of their methodologies might be applicable and potentially beneficial to our work. For instance, Contrastive Learning techniques from [3] and [4] could be incorporated into our framework to further improve the feature representation capabilities of our model.\n\n- We appreciate the reviewer pointing out these references, and **we will certainly consider adding a discussion regarding these works in our paper**, focusing on how they relate to our work and how their methodologies could potentially be integrated into our framework. This will also serve to highlight the unique contributions of our work. We believe such comparisons would enrich the context of our paper and make it more comprehensive."}, {"review_id": "eVnANbWIGC", "reviewer": "Reviewer_1XGa", "summary": "To solve the challenges with limited high-quality labels and non-IID client data in federated learning, the authors present a pioneering SSFOD framework, designed for scenarios where labeled data reside only at the server while clients possess unlabeled data. Meanwhile, they propose the FedSTO, which consists of selective training, orthogonal enhancement, and personalized EMA-driven semi-efficient teacher. Finally, FedSTO achieves 0.082 and 0.035 higher mAP@0.5 when compared to partially supervised and SSFL baselines respectively. ", "questions": "1. What do T1 and T2 in Algorithm 1 represent?", "limitations": "Yes, the authors have described limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Good motivation. On the one hand, selective training can address the primary challenge of establishing a robust backbone for object detectors in FL. Specifically, it fosters more consistent representations by sharing the same non-backbone part. On the other hand, orthogonal enhancement reduces the bias towards specific weather conditions and the heterogeneity of object categories, leading to improved performance.\n2. Many experiments have also verified the effectiveness of the proposed FedSTO.", "weaknesses": "1. The authors do not detailed introduce Personalized Pseudo Labeling for Unlabeled Clients, which is not easy to understand. \n2. I don't understand what Fig.3 wants to express, why there will be Fully supervised in Unlabeled overcast, rainy, and snowy.\n3. In Fig.1, I don't know how the orthogonal enhancement works on the neck and head. Besides, I can not understand how to generate pseudo labels and assign them to the client models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### W1. No explanations for personalized pseudo-labeling for unlabeled clients\n- Thank you for your question. We apologize if the presentation of the “Personalized Pseudo Labeling for Unlabeled Clients” was not clear in the main body of the paper due to space constraints. We have provided a comprehensive explanation in **Appendix H.2** which offers more details on this aspect.\n- In our SSFOD framework, we adopt a local Exponential Moving Average (EMA) model for generating pseudo labels, which facilitates efficient utilization of each client's unlabeled data. The EMA model operates on an infinite impulse response filter that assigns decreasing exponential weights. This arrangement ensures a balanced consideration of both historical and immediate data points, aiding in making reliable predictions.\n- Each client maintains a local EMA model whose weights are a weighted average of the client's own weights and the weights of the global model. This correlation is depicted in **Eq. (6) in the Appendix**. After each server broadcast, the weights of the Local EMA model are reinitialized with the global model’s weights, ensuring global model consistency along with local data awareness.\n- The client's local EMA model gradually tunes to the unique characteristics of its local data, integrating updates from the client's unlabeled data. Consequently, we create a balance of personalization and model performance in a FL environment while alleviating communication overheads.\n- Following the weight updates, the local EMA model functions as the pseudo labeler for each client. This method ensures the generation of stable and reliable pseudo labels, despite the limited interactions with the global model. The local EMA model guarantees independent pseudo label generation at each client’s end without requiring frequent server updates. It also enhances the learning process’s robustness, as the local EMA model remains largely unaffected by the client model’s noisy gradient updates.\n- We believe this explanation provides a better understanding of our approach. We hope to edit the main text for clarity and to add more details if we get extra pages for the camera-ready version of the paper. **For now, please refer to Appendix H.2 for a detailed understanding.**\n\n### W2. The purpose of Fig 3\n- Thank you for your question about Figure 3. We apologize if it caused any confusion. In this figure, “Fully supervised” refers to the model trained with labeled data across all weather conditions—Cloudy, Overcast, Rainy, and Snowy. “Partially supervised” signifies the model trained only on labeled data from the “Cloudy” condition, while “Vanilla SSFL” implies the model trained on labeled data from 'Cloudy' and unlabeled data from “Overcast”, “Rainy”, and “Snowy” conditions. The performance of these models was then evaluated across all weather conditions, regardless of their training status.\n- We understand that the labeling of the figure might be a bit confusing, and we appreciate your feedback. We will work on clarifying the labeling in the figure to avoid any such confusion in the future.\n\n### W3. Overview of the method\n- Thank you for your inquiry regarding Fig.1 and the processes involved.\n- The Orthogonal Enhancement functions by imposing an orthogonality regularization on the neck and head of the model (line 217). This process penalizes the kernel weight matrices to be orthogonal, being robust towards the feature bias (theoretically supported in the Appendix Section F).\n- Regarding the generation of pseudo labels, this is accomplished using a pseudo labeler. As outlined in Appendix Section G, the pseudo labeler generates predictions which then undergo a non-maximum suppression step to form pseudo annotations. These pseudo annotations are subsequently used in conjunction with the loss function described in the same section of the appendix, guiding the learning of the client models.\n- Due to space constraints in the main body of the paper, we have detailed these implementation specifics in the appendix. We appreciate your understanding and welcome any further queries you might have.\n\n### Q1. What are T1 and T2 in Algorithm 1?\n- T1 and T2 in Algorithm 1 represent distinct phases in the learning process. Specifically, T1 refers to the number of pretraining rounds, which means “Representation Learning with Selective Training”. This phase aids in establishing a decent starting point for subsequent learning. On the other hand, T2 denotes the number of rounds dedicated to Orthogonal Enhancement. This phase is primarily designed to increase the diversity of feature learning by discouraging the kernel weight matrices from becoming orthogonal.\n- We will further clarify it by explicitly mentioning T1 and T2."}]

Author rebuttal:
We extend our gratitude to all the reviewers for providing comprehensive and thoughtful feedback on our manuscript. We appreciate your valuable insight into the strengths and areas for improvement of our work.

### Summary of Strengths cited by Reviewers
- **Novelty in Approach and Framework:** We are encouraged by the observations of **Reviewers 2CR4** and **Reviewer qq85** noting the significant strength of our proposed SSFOD and FedSTO approaches, particularly in non-IID weather conditions. While it may seem prior methods tackled similar problems for image classification, these methods perform poorly in SSFOD's substantially more complex and practically relevant setting. The novel components of FedSTO — two-stage selective training and high-quality personalized pseudo-labelers — substantially boost performance compared to simply applying prior work to SSFOD (**we present ablation studies and theoretical analysis in Tables 1 & 2 of the main text and Sections F - J of the Appendix**). Indeed, FedSTO nears the performance of centralized learning models even with only having access to 25% of the labels and non-IID data.
- **Impact**: We appreciate **Reviewers AM8c, Reviewer 1XGa**, and **Reviewer qq85** for noting the important motivation and impact of our work. FedSTO reaches competitive performance while also exhibiting crucial real-world benefits of not sharing data to a central server and not requiring labeling at the edge. Preserving user privacy and reducing training cost are necessary improvements for feasibility in the real-world for a wider range of applications. 
- **Clarity, Structure, and Presentation:** We are pleased to note the collective positive feedback on the clarity and presentation of our work, as acknowledged by **Reviewers 2CR4**, **Reviewer qq85**, **Reviewer 8JqK**, and **Reviewer AM8c**.
- **Technical Soundness and Integration:** **Reviewers 8JqK** and **Reviewer AM8c**'s emphasis on our paper's technical solidity is greatly appreciated. Presenting multiple techniques for semi-supervised FL and providing theoretical analysis (in the Appendix) while ensuring the coherence of the overarching framework was one of our primary objectives.
- **Extensive Experiments and Comparative Analysis:** We are encouraged by the feedback from **Reviewers AM8c** and **Reviewer 1XGa** regarding our empirical approach. We conducted comprehensive experiments across diverse datasets and made comparisons against multiple baselines—as well as SOTA techniques—to showcase the efficacy and robustness of FedSTO. **Reviewer 1XGa** specifically acknowledges the effectiveness of selective training in establishing a robust backbone for object detectors in FL, particularly its ability to reduce biases and heterogeneities.
- **Relevance to Practical Applications:** The observations made by **Reviewer AM8c** highlight our work's significance in real-world scenarios, where not all client data may be labeled. Leveraging unlabeled data in FL remains an imperative challenge, and our work attempts to provide tangible solutions in this direction. FedSTO also reduces communication cost by 20.52% in comparison to conventional FL algorithms, as **detailed in the attached PDF of tables**.

### Core Contributions of Our Work
- **Semi-Supervised Federated Object Detection (SSFOD)**: We are glad that **Reviewers 2CR4**, **Reviewer qq85**, **Reviewer 1XGa**, and **Reviewer AM8c** acknowledged the novelty of SSFOD, especially the challenging landscape of FL for distributed data sources with limited labeled data and non-IID client data. Our approach is positioned to significantly benefit applications such as autonomous driving. Not requiring labels in non-IID settings promotes not only cost reductions in training but also importantly preserves privacy as data is retained at the edge.
- **FedSTO**: As pointed out by **Reviewers 2CR4**, **Reviewer qq85**, and **Reviewer 1XGa**, the hallmark of our paper is the introduction of the two-stage training strategy, FedSTO, specifically tailored for clients with 0% labeled non-IID data. This strategy, with its selective training and orthogonally enhanced full-parameter training, is designed to tackle data shifts and represents a pioneering implementation of SSFOD.
- **Selective Training & Orthogonal Enhancement**: **Reviewers AM8c** and **Reviewer 8JqK** have emphasized our distinctive approach of selective training, aimed at refining the detector backbone to prevent overfitting, and orthogonal enhancement, which fosters representation diversity and robustness. These strategies collectively enable our framework to improve generalization capabilities across diverse data distributions.
- **Personalized Pseudo-labeling through EMA**: Both **Reviewers qq85** and **Reviewer AM8c** brought attention to our personalized EMA-driven pseudo-label assignment, a novel contribution that ensures the generation of high-quality pseudo labels for object detection tasks. This component elevates the performance of object detectors in a semi-supervised, non-IID FL context.
- **Empirical Validation and Superior Performance:** As recognized by **Reviewer 2CR4**, **Reviewer AM8c**, and **Reviewer 1XGa**, the empirical validation of our method across multiple datasets (BDD100K, Cityscapes, and SODA10M) demonstrates the superiority of our proposed methodology over existing federated and semi-supervised learning methods. The notable improvements in mAP@0.5 metrics when compared to baseline models further accentuates the efficacy of our approach. 

### Things in Pdf
In the attached PDF, we included the following results:
- Results for fully supervised FL
- mAP@0.75 results
- Global model's performance
- Results with 100 clients and various sampling ratio

In light of the feedback, we are committed to refining our manuscript further, addressing any lingering queries, and incorporating your valuable insights into the final version of our paper.

Meta-review: The reviewers are positive on the technical contributions and impact of the proposed federated learning framework for object detection. During the review process, issues like novelty and proper comparisons/evaluation were raised, which were later addressed by the authors during the rebuttal. Since all reviewers lean toward the acceptance of this paper, I think it is above the threshold for publication.
Decision: Accept (poster)