SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. ORN employed right from scratch without any pre-training or frozen components.
2. A two-stage process where the model head is frozen during the pre-training phase, followed by fine-tuning across the entire model with ORN.
3. A more nuanced strategy where both the neck and head components of the model are initially frozen, subsequently transitioning to an ORN-driven FPT.
Our empirical results decisively point to the superiority of the third approach (Table 10). Despite the initial freezing of the neck and head components, it yields performance that surpassed the other two strategies. This counter-intuitive finding underscores the effectiveness of the careful balancing of selective parameter freezing and ORN application.
The mere marginal performance difference in mAP between the second and third approaches suggests room for further investigation. An exploration of the specific conditions under which one might outperform the other could shed light on new avenues for model improvement. This represents an exciting direction for future research.
However, for the scope of the present study, we opt for the third strategy of freezing both the neck and head components simultaneously. This decision is not only influenced by the slightly higher performance but also by the added advantage of reduced communication costs. This approach resonates with the primary objective of our research - to realize high-performing, efficient, and cost-effective FL frameworks for SSOD tasks.
K Examination of Loss Functions within SSFOD Framework
An integral part of our study on the SSFOD framework involves an examination of the influence of various loss functions on model performance. One of the notorious challenges in object detection tasks is dealing with class imbalance. To counter this, we decided to evaluate the effectiveness of Focal Loss, which is widely used for its capability to handle such imbalances, in comparison with the conventional Cross-Entropy (CE) Loss, which we refer to as vanilla Loss in the paper.
Focal Loss [24] has been designed to tackle class imbalance by reducing the contribution of easy instances, thus allowing the model to concentrate on challenging, misclassified samples. Despite its recognized efficacy in a variety of settings, we observed unexpected performance under our specific framework.
Contrary to our initial assumptions, our empirical findings suggest that the Vanilla Loss proved to be a more potent safeguard against the class imbalance issue within our SSFOD framework (Table 11). We compare the performance by changing the loss functions in the baseline SSFOD training. The underlying reasons for this surprising result are potentially multifaceted and certainly merit additional exploration. However, these observations underscore the importance of empirical testing in choosing the appropriate loss functions tailored for specific frameworks and tasks.
Table 11: Performance of CE loss and focal loss on the BDD dataset with 1 labeled server and 3 unlabeled clients. It highlights how each added method contributes to the overall performance under both Non-IID and IID conditions.
Method	Non-IID	IID
Cloudy	Overcast	Rainy	Snowy	Total	Cloudy	Overcast	Rainy	Snowy	Total
CE loss	0.560	0.566	0.553	0.553	0.558	0.572	0.588	0.593	0.610	0.591
Focal loss	0.364	0.462	0.438	0.446	0.428	0.371	0.464	0.469	0.482	0.447
L Implementation
To provide a comprehensive overview of our study, we provide the details of our implementation, including the reproduction of various existing FL methods such as FedDyn [1], FedBN [22], FedOpt [33], and FedPAC [39].
FedDyn [1]: For the implementation of FedDyn, we adhere strictly to the training methodology as stipulated in the original paper.
FedBN [22]: In implementing the FedBN method, we make a conscientious choice to leave out the statistics and bias of the batch normalization (BN) layer from the aggregation step.
- FedOpt [33]: FedOpt implementation aligns with Vanilla SSFL, albeit with a slight modification in the server model's optimization procedure. The key differentiation lies in the utilization of a pseudo learning rate of 0.9 during server model optimization, which was chosen through empirical evaluations to maximize performance. Thus, while preserving the core attributes of the FedOpt framework, we adapt it to our specific task requirements.
- FedPAC [39]: The execution of the FedPAC framework, primarily architected for classification tasks, necessitates thoughtful adaptation to suit our object detection scenario. This adaptation involves the appropriation of the YOLOv5 head, used as a stand-in classifier, which allows us to transpose our specific needs onto the FedPAC blueprint. This process not only honors the training techniques prescribed by FedPAC but also satisfies our model's unique requirements. As an integral part of our approach, the local EMA model, a cornerstone of our broader learning strategy, is employed as a pseudo labeler in this scheme. Furthermore, our model's backbone representation is equated with the output from YOLO's backbone. This bespoke modification offers a perspective to exploit the central principles of FedPAC while tailoring them to suit the idiosyncrasies of our model.
M Experimental Results with 100 Clients
In the setting of an extensive FL environment with a network of 100 clients and 0.1 client sampling ratio, we embark on an empirical investigation to ascertain the robustness and scalability of the proposed method. This scenario closely mirrors real-world cross-device situations, inherently characterized by widespread client distribution. Intriguingly, each client in our experiment is associated with data corresponding to a single weather condition. Notwithstanding this, our method exhibits remarkable resilience and efficacy. As an illustration, the model outperforms a baseline that is trained solely on the server's labeled data, as demonstrated in Table 12. This superior performance underscores the model's capability in effectively leveraging the heterogeneity intrinsic to distributed data, thereby enhancing the overall performance. These empirical findings provide compelling evidence of the adaptability and effectiveness of our approach within large-scale FL contexts. With its ability to maintain high performance coupled with computational efficiency, our method exhibits promising potential in managing data heterogeneity across extensive federated networks.
Table 12: Performance under Non-IID scenarios of BDD100k dataset with 1 server and 100 clients. FedSTO exhibits improvements under various object categories, and significantly outperforms the performance for unlabeled clients. The term 'Server Only Scenario' aligns with the notion of 'partially supervised' in CL settings.
Method	Labeled	Unlabeled
Categories
Person	Car	Bus	Truck	Traffic Sign	Person	Car	Bus	Truck	Traffic Sign
Server Only Scenario	0.378	0.710	0.141	0.425	0.490	0.337	0.707	0.160	0.338	0.491
FedSTO within SSFOD framework	0.393	0.714	0.442	0.510	0.540	0.487	0.738	0.573	0.589	0.617