SlowGuess/NeurIPS_2023_OCR huggingface dataset:
components of the architecture. Our approach promotes generation of diverse, non-redundant, and domain-invariant feature representations, thereby enhancing the model's robustness, reducing noise influence, and significantly augmenting its ability to handle unlabeled data across varied domains.
Incorporating orthogonality regularization into our framework substantially amplifies the divergence in the embedding space, enhancing the model's overall detection quality and the reliability of pseudo labels. Importantly, our strategy of embedding orthogonality into the non-backbone parts of the model, such as the neck and head, fosters a more balanced and comprehensive training process. This reduces the bias towards specific weather conditions and the heterogeneity of object categories, leading to improved performance as demonstrated in Table 2. Our approach draws upon successful techniques from fine-tuning [8, 26], and transfer learning, and is particularly inspired by meta-learning concepts,[37, 32]. In particular, the tendency of the non-backbone components of the model to develop biases prompts us to introduce an orthogonal property to this section. This measure helps counteract these biases, thereby further enhancing the model's robustness and adaptability when confronted with diverse, unlabeled data across multiple domains.
4.3 Main Algorithm: FedSTO
Algorithm 1 illustrates the overall procedure of FedSTO within the SSFOD framework. The server model, parameterized by  $W_{s}$ , is first trained in a supervised fashion during the warm-up phase (Line 1). The algorithm then transitions to Phase 1: Selective Training for Pretraining. This phase involves multiple training iterations (Line 3), where in each iteration, a subset of clients is sampled (Line 4). The backbone part of each client's model,  $W_{u,k}$ , is updated using their local unlabeled datasets (Line 6). The updated parameters are then aggregated at the server (Line 8), and the server model is updated using its labeled dataset (Line 9). In Phase 2: Full Parameter Training with Orthogonal Enhancement, the CLIENT-ORTHOGONALUPDATE and SERVER-ORTHOGONALUPDATE METHODS are employed (Lines 14 and 18), introducing orthogonality regularization to the training process. This second phase debiases the non-backbone parts of the model, ensuring they have a robust predictor across various weather conditions that effectively counterbalance the inherent data heterogeneity.
BDD100K [41] We utilize the BDD100K dataset, which consists of 100,000 driving videos recorded across diverse U.S. locations and under various weather conditions, to evaluate our method. Each
Table 3: Comparison of FedSTO within the SSFOD framework against the Baselines, SSL, SSFL methods with 1 server and 3 clients on BDD100K dataset [41]. FedSTO exhibits improvements under various weather conditions on both IID and Non IID cases, and performs close to the centralized fully supervised case.  $\dagger$  denotes the SSFL with the local EMA model as a pseudo label generator.
Type	Algorithm	Method	Non-IID	IID
Cloudy	Overcast	Rainy	Snowy	Total	Cloudy	Overcast	Rainy	Snowy	Total
Centralized	SL	Fully Supervised	0.600	0.604	0.617	0.597	0.605	0.600	0.604	0.617	0.597	0.605
Partially Supervised	0.540	0.545	0.484	0.474	0.511	0.528	0.545	0.533	0.510	0.529
SSL	Unbiased Teacher [25]	0.551	0.550	0.502	0.503	0.527	0.546	0.557	0.541	0.533	0.544
EMA Teacher [38]	0.598	0.59	0.568	0.568	0.581	0.586	0.570	0.571	0.573	0.575
Federated	SFL	Fully Supervised	0.627	0.614	0.607	0.585	0.608	0.635	0.612	0.608	0.595	0.613
\( SSFL^† \)	FedAvg [27]	0.560	0.566	0.553	0.553	0.558	0.572	0.588	0.593	0.610	0.591
FedDyn [1]	0.508	0.569	0.541	0.522	0.535	0.355	0.414	0.420	0.397	0.400
FedOpt [33]	0.561	0.572	0.565	0.566	0.566	0.591	0.587	0.588	0.577	0.586
FedPAC [39]	0.514	0.532	0.496	0.489	0.508	0.510	0.549	0.547	0.554	0.540
FedSTO	0.596	0.607	0.590	0.580	0.593	0.591	0.634	0.614	0.595	0.609
video, approximately 40 seconds in duration, is recorded at  $720\mathrm{p}$  and 30 fps, with GPS/IMU data available for driving trajectories. For our experiments, we specifically select 20,000 data points, distributed across four distinct weather conditions—cloudy, rainy, overcast, and snowy. In this study, we primarily focus on five object categories: person, car, bus, truck, and traffic sign. The dataset is partitioned into clients based on these weather conditions, simulating data-heterogeneous clients. This experimental setup enables us to investigate the influence of data heterogeneity on our framework and to evaluate its robustness under realistic conditions.
Cityscape [4] We conduct additional experiments using the Cityscapes dataset, which consists of urban street scenes from 50 different cities. Given that this dataset does not provide precise weather information for each annotation, we distribute the data to clients in a uniformly random manner. For our studies, we employ the package, encompassing fine annotations for 3,475 images in the training and validation sets, and dummy annotations for the test set with 1,525 images. We also include the other package, providing an additional 19,998 8-bit images for training.
SODA10M [9] To evaluate our approach under diverse conditions, we employ the SODA10M dataset, which features varied geographies, weather conditions, and object categories. In an IID setup, 20,000 labeled data points are uniformly distributed among one server and three clients. For a more realistic setup, the 20,000 labeled data points are kept on the server while 100,000 unlabeled data points are distributed across the clients. This arrangement enables performance evaluation under distinct weather conditions—clear, overcast, and rainy—showcasing resilience and robustness.
5.1.2 Training Details
We conduct our experiments in an environment with one server and multiple clients, depending on the experiment. Both the server and the clients operate on a single local epoch per round. Our training regimen spans 300 rounds: 50 rounds of warm-up, 100 rounds of pretraining  $(T_{1})$ , and 150 rounds of orthogonal enhancement  $(T_{2})$ . We use the YOLOv5 Large model architecture with Mosaic, left-right flip, large scale jittering, graying, Gaussian blur, cutout, and color space conversion augmentations. A constant learning rate of 0.01 was maintained. Binary sigmoid functions determined objectiveness and class probability with a balance ratio of 0.3 for class, 0.7 for object, and an anchor threshold of 4.0. The ignore threshold ranged from 0.1 to 0.6, with an Non-Maximum Suppression (NMS) confidence threshold of 0.1 and an IoU threshold of 0.65. We incorporate an exponential moving average (EMA) rate of 0.999 for stable model parameter representation.
Table 3 illustrates the efficacy of our proposed SSFOD method against various baselines and state-of-the-art approaches on the BDD100K dataset. FedSTO significantly outperforms other techniques under different weather conditions and data distribution scenarios, i.e., IID and Non-IID. In the CL scenarios, the fully supervised approach yields the highest performance, with SSL methods, such as EMA Teacher [38], demonstrating competitive results. However, the real challenge lies in federated settings, where data privacy and distribution shift become critical considerations. In the SSFOD framework, our FedSTO method consistently surpasses other SSFL techniques. Notably, it achieves superior results even in challenging Non-IID settings, demonstrating its robustness to data distribution shifts. Similar trends hold when increasing the number of clients as shown in the appendix. In IID
Table 4: Performance under random distributed cases of Cityscapes [4]. FedSTO exhibits improvements under various object categories, and significantly outperforms the performance for unlabeled clients.  $\dagger$  denotes the SSFL with the local EMA model as a local pseudo label generator.
Type	Algorithm	Method	Labeled	Unlabeled
Categories
Person	Car	Bus	Truck	Traffic Sign	Person	Car	Bus	Truck	Traffic Sign
Centralized	SL	Fully Supervised	0.569	0.778	0.530	0.307	0.500	0.560	0.788	0.571	0.283	0.510
Partially Supervised	0.380	0.683	0.193	0.302	0.246	0.358	0.648	0.343	0.138	0.255
SSL	Unbiased Teacher [25]	0.391	0.695	0.225	0.320	0.297	0.410	0.689	0.373	0.129	0.354
EMA Teacher [38]	0.475	0.711	0.354	0.347	0.379	0.460	0.727	0.436	0.144	0.378
Federated	SFL	Fully Supervised	0.498	0.715	0.357	0.289	0.410	0.492	0.714	0.451	0.251	0.425
\( SSFL^† \)	FedAvg [27]	0.450	0.697	0.310	0.304	0.356	0.482	0.725	0.425	0.247	0.397
FedBN [22]	0.488	0.709	0.325	0.285	0.411	0.375	0.618	0.046	0.031	0.286
FedSTO	0.504	0.720	0.342	0.261	0.415	0.487	0.740	0.460	0.181	0.437
conditions, our method continues to excel, achieving results close to fully supervised centralized approach. These findings highlight the strength of our FedSTO method in leveraging the benefits of FL while mitigating its challenges. The robust performance of our approach across various weather conditions and data distributions underscores its potential for real-world deployment.
When examining the performance on the Cityscapes dataset under uniformly random distributed conditions, the superiority of FedSTO within the SSFOD framework also remains apparent, as shown in Table 4. Compared to other methods, FedSTO consistently demonstrates improved generalization across most object categories, both for labeled and unlabeled data. Intriguingly, the performance of FedSTO surpasses even that of SSL in CL environments.
Evaluation with mAP@0.75 The mAP@0.75 results on the BDD dataset highlight the efficacy of the FedSTO approach (Table 5). In Non-IID settings, while the Fully Supervised centralized method achieve an average mAP of 0.357, FedSTO recorded 0.338, exhibiting comparable performance. However, under IID conditions, FedSTO registers an mAP@0.75 of 0.357, closely matching the SFL result of 0.359. These results indicate that FedSTO offers competitive object detection capabilities, even with stricter IoU thresholds.
Table 5: mAP@0.75 on the BDD dataset with 1 labeled server and 3 unlabeled clients.
Type	Algorithm	Method	Non-IID	IID
Cloudy	Overcast	Rainy	Snowy	Total	Cloudy	Overcast	Rainy	Snowy	Total
Centralized	SL	Fully Supervised	0.351	0.352	0.368	0.356	0.357	0.351	0.352	0.368	0.356	0.357
Partially Supervised	0.281	0.295	0.261	0.303	0.285	0.281	0.295	0.261	0.303	0.285
Federated	SFL	Fully Supervised	0.384	0.366	0.357	0.317	0.356	0.377	0.356	0.352	0.349	0.359
SSFL	FedSTO W/O ST	0.321	0.303	0.265	0.267	0.289	0.298	0.321	0.324	0.302	0.311
FedSTO	0.347	0.351	0.341	0.312	0.338	0.343	0.375	0.361	0.350	0.357
Results on Real World Dataset, SODA10m [9] Figure 4a illustrates the performance of our method and other baselines on the SODA10m dataset, where labeled data is synthetically divided in an IID manner across one server and three clients. Our method demonstrates near-parity with the fully supervised approach, evidencing its efficacy. Figure 4b represents the averaged performance across varying weather conditions on the SODA10m dataset. Here, all 20k labeled data resides on the server, and 100k unlabeled data points from SODA10m are distributed across three clients. Despite these variations in conditions, our method consistently outperforms other baselines, confirming its robustness and applicability in diverse environments.
(a) IID Case
(b) Real World Scenario
Figure 4: (a) Performance of various methods on the SODA10m dataset in an IID setting, (b) Average performance across different weather conditions using unlabeled data from the SODA10m dataset.
Varying Number of Clients In a non-IID BDD dataset configuration with 1 server and 20 clients, our proposal advances beyond competing methods, scoring 0.455 and 0.458 on labeled and unlabeled data, respectively. This outcome showcases our method's aptitude for tackling intricate real-world circumstances.
Table 6: Performance on a non-IID case of the BDD100k dataset with 1 server and 20 clients.
Type	Centralized	Federated-SSFL
Method	Partially Supervised	FedAvg	ST	FedSTO
Labeled	0.3768	0.405	0.405	0.455
Unlabeled	0.3524	0.4311	0.4322	0.458
Varying Sampling Ratio Table 7 demonstrates the impact of different client sampling ratios on the FedSTO performance using the BDD100k dataset. Notably, even at a lower sampling ratio of 0.1, FedSTO yields commendable results, especially in the unlabeled set for categories like 'Car' (0.738) and 'Bus' (0.573). This underscores that a reduced client sampling can still lead to significant performance improvements, emphasizing the efficiency and adaptability of the FL approach.
Table 7: Performance (mAP@0.5) under Non-IID scenarios of BDD100k dataset with 1 server and 100 clients according to the changes of client sampling ratio for implementing FedSTO. The term 'Server Only' aligns with the notion of 'partially supervised' in CL settings.
Method	Labeled	Unlabeled
Categories
Person	Car	Bus	Truck	Traffic Sign	Person	Car	Bus	Truck	Traffic Sign
Server Only (i.e., client sampling ratio 0.0)	0.378	0.710	0.141	0.425	0.490	0.337	0.707	0.160	0.338	0.491
FedSTO with client sampling ratio 0.1	0.393	0.714	0.442	0.510	0.540	0.487	0.738	0.573	0.589	0.617
FedSTO with client sampling ratio 0.2	0.458	0.747	0.476	0.521	0.571	0.440	0.731	0.378	0.525	0.573
FedSTO with client sampling ratio 0.5	0.444	0.745	0.437	0.502	0.550	0.489	0.730	0.438	0.512	0.538
Efficiency on Network Bandwidth Table 8 highlights the communication costs over 350 rounds of training involving 100 clients with a 0.5 client sampling ratio per round. By removing the neck component of the Yolov5L model, its size is reduced from 181.7MB to 107.13MB. This reduction significantly benefits FedSTO in Phase 1, leading to overall bandwidth savings. When comparing with traditional SSFL methods such as FedAvg and FedProx,[20], FedSTO utilizes only 2,166.23 GB - a substantial  $20.52\%$  reduction in network bandwidth.
Table 8: Communication costs over 350 rounds of training with 100 clients when the client sampling ratio is 0.5 per each round. The total Yolov5L size is 181.7MB while the model without the neck part is 107.13MB. Additionally, the model size without BN layers (FedBN [22]) is 181.24 MB. Here, 'Reduction' expresses how much communication cost is reduced compared to using vanilla SSFL (FedAvg and FedProx [20]).
Method	Warm-up (50 rounds)	Phase 1 (150 rounds)	Phase 2 (150 rounds)	Total	Reduction
FedAvg FedProx	0	100 * 0.50 * 150 * 181.7 = 1,362.75 GB	100 * 0.50 * 150 * 181.7 = 1,362.75 GB	2,725.50 GB	-
FedBN FedSTO	0	100 * 0.50 * 150 * 181.24 = 1359.30 GB	100 * 0.50 * 150 * 181.24 = 1359.30 GB	2,718.60 GB	0.25 %
0	100 * 0.50 * 150 * 107.13 = 803.48 GB	100 * 0.50 * 150 * 181.7 = 1,362.75 GB	2,166.23 GB	20.52 %
This paper introduces a novel Semi-Supervised Federated Object Detection (SSFOD) framework, featuring a distinctive two-stage training strategy known as FedSTO. Designed to address the challenges of heterogeneous unlabeled data in federated learning, FedSTO employs selective training and orthogonality regularization with personalized pseudo labeling. These mechanisms facilitate robust and diverse feature learning, thereby enhancing object detection performance across multiple weather conditions and data distributions. Empirical results provide compelling evidence of the superiority of FedSTO over established federated and semi-supervised learning methodologies. Notably, despite operating with the challenging constraint where non-IID clients have no labels, FedSTO successfully counteracts domain shift and achieves performance that is comparable to fully supervised centralized models. This accomplishment constitutes significant strides toward realizing more efficient and privacy-preserving learning in realistic FL settings. As we venture ahead, we aim to concentrate our research efforts on refining FedSTO and exploring additional strategies for leveraging unlabeled data with various domains and model architectures. We anticipate the work presented in this paper will stimulate continued progress in this rapidly evolving field.
[34] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jabbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pages 2021-2031. PMLR, 2020.
[35] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757, 2020.
[36] Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10-5555, 2017.
[37] Jingjing Wang, Jingyi Zhang, Ying Bian, Youyi Cai, Chunmao Wang, and Shiliang Pu. Self-domain adaptation for face anti-spoofing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 2746-2754, 2021.
[38] Bowen Xu, Mingtao Chen, Wenlong Guan, and Lulu Hu. Efficient teacher: Semi-supervised object detection for yolov5. arXiv preprint arXiv:2302.07577, 2023.
[39] Jian Xu, Xinyi Tong, and Shao-Lun Huang. Personalized federated learning with feature alignment and classifier collaboration. In The Eleventh International Conference on Learning Representations, 2023.
[40] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft teacher. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3060-3069, 2021.
[41] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645, 2020.
[42] Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Improving semi-supervised federated learning by reducing the gradient diversity of models. In 2021 IEEE International Conference on Big Data (Big Data), pages 1214-1225. IEEE, 2021.
[43] Jessica Zhao, Sayan Ghosh, Akash Bharadwaj, and Chih-Yao Ma. When does the student surpass the teacher? federated semi-supervised learning with teacher-student ema. arXiv preprint arXiv:2301.10114, 2023.
[44] Huayi Zhou, Fei Jiang, and Hongtao Lu. Ssda-yolo: Semi-supervised domain adaptive yolo for cross-domain object detection. Computer Vision and Image Understanding, 229:103649, 2023.
In this supplementary material, we present additional details that are not included in the main paper due to the space limit.
In the pursuit of progress within the domain of Federated Learning (FL), we propose a novel method, FedSTO within the SSFOD. This innovation warrants an examination of its ethical ramifications, particularly in regards to privacy, fairness, environmental impact, and potential for misuse.
Privacy and Data Security Analogous to established FL methodologies, SSFOD is architected to preserve the privacy of client data. By conducting computations locally and transmitting solely model updates, the risks associated with raw data transmission are mitigated. Nonetheless, the potential threat of adversarial actions, such as model inversion attacks, underscores the imperative for ongoing efforts to bolster the robustness of FL methods against such vulnerabilities.
Fairness The deployment of FedSTO into SSFOD can either amplify or alleviate existing fairness concerns within FL, contingent on the particulars of its application. Should SSFOD predominantly utilize data stemming from specific demographic cohorts, the model predictions risk acquiring an inadvertent bias. In contrast, SSFOD's capacity to manage large-scale, real-world scenarios may engender a more diverse data inclusion, thereby fostering a more equitable model.
Environmental Impact Similar to its FL counterparts, our approach diminishes the requirement for centralized data storage and computation, thereby potentially reducing associated carbon footprints. However, the energy expenditures of localized computation and communication for model updates necessitate judicious management to uphold environmental sustainability.
Potential Misuses Although our approach is designed to enhance the robustness of FL methodologies in large-scale, real-world settings, the potential for misuse remains. For instance, malevolent entities could exploit FedSTO with SSFOD framework to deliberately introduce bias or disinformation into models. Consequently, the implementation of protective measures against such misuse is of paramount importance.
In summary, while our method represents a significant contribution to the FL field, its potential ethical implications mandate thoughtful application. We strongly endorse ongoing discourse and scrutiny to ensure that its deployment is in alignment with the principles of privacy, fairness, and social responsibility.
While our SSFOD method offers significant advancements in Semi-Supervised Federated Learning for Object Detection, it is important to recognize its accompanying limitations:
1. Performance with Highly Imbalanced Data Our SSFOD method exhibits robustness across a range of data heterogeneity. However, its performance in scenarios involving severe data imbalance across clients necessitates additional exploration. Imbalance here refers to disparities in data class distribution across clients. Instances of extreme skewness could lead to biased learning outcomes, with the model excelling in classes with abundant samples and faltering in those with fewer. Although SSFOD incorporates mechanisms to offset effects of data heterogeneity, it is not explicitly designed to manage extreme data imbalance.
2. Computational Overhead Despite its effectiveness in bolstering model robustness in large-scale, real-world scenarios, SSFOD introduces additional computational overhead. This is primarily due to the algorithmic complexities and computational requirements intrinsic to our method, which might be a constraint for resource-limited devices often participating in federated learning. This could potentially limit the scalability and applicability of our method in real-world FL scenarios. Therefore, improving the computational efficiency of SSFOD without compromising its efficacy is an important direction for future research.
3. Sensitivity to Varying Weather Conditions SSFOD has been designed to tackle the challenge of varying weather conditions in autonomous driving. However, in real-world scenarios, there are other types of environmental changes that can equally affect the learning process. For instance, varying lighting conditions or changes in road surfaces might influence the input data. Since our SSFOD method primarily focuses on weather conditions, it might not exhibit the same level of efficiency when dealing with other environmental factors. Future iterations of SSFOD could explore these areas to provide a more comprehensive solution to environmental handling in FL.
Despite the aforementioned limitations, our SSFOD method lays the groundwork for several promising future research directions:
1. Handling Other Environmental Factors: Future work could extend the SSFOD method to handle other environmental factors efficiently. This would make it a more comprehensive solution for real-world federated learning scenarios where different environmental factors coexist.
2. Adaptation for Imbalanced Data: Investigating and enhancing the performance of SSFOD with highly imbalanced data distribution would be a valuable future direction. Techniques like adaptive resampling or cost-sensitive learning could be integrated with our method to tackle this challenge.
3. Optimization of Computational Efficiency: Future research could focus on optimizing the computational efficiency of the SSFOD method. Reducing the computational overhead without compromising the robustness in large-scale, real-world scenarios would make our method more practical for real-world FL scenarios.
4. Robustness Against Adversarial Attacks: As the FL domain evolves, adversarial attacks pose an increasing threat to model robustness. Future work could explore how to bolster the SSFOD method (and FL methods, in general) to ensure robustness against adversarial attacks.