SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By addressing these limitations and exploring these future directions, we can continuously refine and evolve the SSFOD method to better serve the ever-growing demands of federated learning.
E Detailed Data Heterogeneity in Federated Object Detection
Heterogeneity, a prevalent attribute in object detection datasets, often arises from three crucial aspects:
- Weather-induced feature distribution skew: Within outdoor scenarios like autonomous driving, weather variations significantly impact the visual representation of objects. Differing conditions such as sunny, rainy, foggy, or snowy can alter an object's appearance, causing the weather-induced feature distribution skew. Moreover, sensor diversity, including RGB cameras and infrared sensors, contributes to this skew as they respond uniquely to various weather conditions. This complex scenario creates a challenging task for an object detection system that must generalize across diverse conditions.
- Class distribution heterogeneity: This heterogeneity refers to the uneven representation of various classes within a dataset. In many cases, certain classes are far more prevalent than others. For instance, in an autonomous driving scenario, 'cars' or 'pedestrians' may be much more frequent than 'bicycles' or 'motorcycles.' This imbalance can cause learning algorithms to develop a bias towards more common classes. Moreover, in a federated learning scenario, the class distribution may vary among clients; a rural area client might capture more 'animal' instances compared to an urban area client.
- Label density heterogeneity: This form of heterogeneity pertains to the variation in the quantity of annotated objects per image. An image of a crowded scene may contain far more objects than a sparser image. This variability can influence the performance of detection models, particularly those that rely on a fixed number of anchors or proposals per image.
Furthermore, it can also impact the training process as images with more objects provide more "training signal" per image than those with fewer objects. In a federated learning context, certain clients might possess more densely labeled data than others, which could affect the learning process.
While our current work primarily addresses the issue of weather-induced feature distribution skew, the other forms of heterogeneity, i.e., class distribution heterogeneity and label density heterogeneity, also require careful consideration. They present unique challenges within the federated learning environment and significantly influence model performance. It is crucial to extend our methodologies to account for these factors, fostering more robust and versatile object detection systems capable of handling the intricate realities of real-world scenarios. Consequently, future work should aim at developing comprehensive solutions that cater to all these facets of data heterogeneity in the context of federated learning.
F Exploring the Impact of Orthogonal Enhancement under Data Heterogeneity in SSFOD
F.1 Theoretical Analysis
This section presents a straightforward bounding strategy for the loss in the context of data heterogeneity. Our primary focus is to ascertain the Mean Squared Error (MSE) of a two-layered neural network, encompassing both 'head' and 'backbone'. Analyzing this model's behavior under a slightly perturbed data distribution is intended to offer insights into worst-case loss scenarios when orthogonal weights are employed.
We consider a multivariate distribution, denoted by  $\mathbb{P}$ , with  $\pmb{x} \in \mathbb{R}^d$  and  $\pmb{y} \in \mathbb{R}^m$  being drawn from this distribution. We have  $n$  i.i.d. samples from  $\mathbb{P}$ , constituting our dataset  $\mathcal{D} = (\pmb{x}_i, \pmb{y}_i) \colon (\pmb{x}_i, \pmb{y}_i) \sim \mathbb{P}, 1 \leq i \leq n$ . For the sake of this analysis, our model is expressed as  $f(\pmb{x}) = W^\top B\pmb{x}$ , where  $B \in \mathbb{R}^{k \times d}$  and  $W \in \mathbb{R}^{k \times m}$  are the backbone and head weight matrices, respectively. The MSE is given by  $\mathbb{E}_{\pmb{x},\pmb{y}}\| \pmb{y} - f(\pmb{x})\|_2^2$ , representing the expectation of the  $\ell^2$ -norm of the difference vector between the ground truth and the prediction.
Under the presumption that  $f$  has been finely optimized on the training sample distribution (using the dataset  $\mathcal{D}$ ) to yield an MSE of value  $L$ , we next examine a perturbed data distribution, denoted by  $\mathbb{P}'$ , characterized as follows:
$$
\left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \sim \mathbb {P} ^ {\prime} \Longleftrightarrow \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}\right) \sim (\boldsymbol {x} + \boldsymbol {\epsilon}, \boldsymbol {y}) \tag {1}
$$
Here,  $(\pmb{x},\pmb{y})\sim \mathbb{P}$  and  $\epsilon \sim \mathcal{N}_d(0,\Sigma)$  is Gaussian noise, characterized by covariance matrix  $\Sigma$  independent of  $(\pmb{x},\pmb{y})$ . The MSE loss under  $\mathbb{P}'$ , represented by  $L'$ , can be bound as:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {x} ^ {\prime}, \boldsymbol {y} ^ {\prime}} \| \boldsymbol {y} ^ {\prime} - W ^ {\top} B \boldsymbol {x} ^ {\prime} \| _ {2} ^ {2} = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \epsilon} \| \boldsymbol {y} - W ^ {\top} B (\boldsymbol {x} + \epsilon) \| _ {2} ^ {2} \\ = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \boldsymbol {\epsilon}} \| \boldsymbol {y} - W ^ {\top} B \boldsymbol {x} + W ^ {\top} B \boldsymbol {\epsilon} \| _ {2} ^ {2} \\ = \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}} \| \boldsymbol {y} - W ^ {\top} B \boldsymbol {x} \| _ {2} ^ {2} + \mathbb {E} _ {\boldsymbol {\epsilon}} \| W ^ {\top} B \boldsymbol {\epsilon} \| _ {2} ^ {2} \tag {2} \\ + \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \epsilon} [ (\boldsymbol {y} - W ^ {\top} B \boldsymbol {x}) ^ {\top} W ^ {\top} B \boldsymbol {e} ] \\ + \mathbb {E} _ {\boldsymbol {x}, \boldsymbol {y}, \epsilon} [ \boldsymbol {\epsilon} ^ {\top} B ^ {\top} W (\boldsymbol {y} - W ^ {\top} B \boldsymbol {x}) ] \\ = L + \mathbb {E} \| W ^ {\top} B \boldsymbol {\epsilon} \| _ {2} ^ {2}. \\ \end{array}
$$
This calculation introduces a nonnegative penalty of  $\mathbb{E}\| W^{\top}B\epsilon \| _2^2$  to the out-of-sample MSE.
The derived penalty opens the possibility for further analysis, particularly when considering structural assumptions on the head, backbone, or noise vector. The Gaussian assumption for the noise vector is quite enlightening, given that the squared  $\ell^2$ -norm is tantamount to the quadratic form  $\epsilon^\top B^\top WW^\top B\epsilon$ . By incorporating a recognized principle about the expectation of a Gaussian vector's quadratic form, we obtain:
$$
\mathbb {E} \| W ^ {\top} B \boldsymbol {\epsilon} \| _ {2} ^ {2} = \mathbb {E} [ \boldsymbol {\epsilon} ^ {\top} B ^ {\top} W W ^ {\top} B \boldsymbol {\epsilon} ] = \operatorname {t r} (B ^ {\top} W W ^ {\top} B \Sigma). \tag {3}
$$
Assuming  $WW^{\top} = I$ , an identity matrix, the trace simplifies to  $\mathrm{tr}(B^\top B\Sigma)$ . In the event the model  $f$  has undergone orthogonality regularization, resulting in a semi-orthogonal head  $W$  (precisely,  $WW^{\top} = I$ ), we obtain a concise formulation for the penalty that hinges solely on the backbone and the fundamental noise parameter.
F.2 Insights from Theoretical Analysis
The insights gained from our theoretical analysis significantly enrich our understanding of the FedSTO approach and how it can navigate challenges specific to FL.
One major challenge in FL is data heterogeneity – the data across different clients (or devices) can vary significantly. By bounding the MSE for our model under slightly shifted distributions, we learn how our model's performance could change in the presence of such data heterogeneity. This is akin to having a 'sensitivity' measure for our model, showing us how 'sensitive' the model is to changes in data distributions. The better we understand this sensitivity, the more effectively we can tailor our model to handle the challenges of federated learning.
The analysis also highlights the significance of having orthogonal weights in our model. The investigation reveals that when the head of our model is semi-orthogonal, the penalty on the out-of-sample MSE simplifies to depend only on the backbone and the noise term, effectively isolating the effect of client-specific data perturbations. This insight is of particular importance because it provides a clear and quantifiable understanding of the role and benefit of orthogonal weights in controlling model sensitivity to data heterogeneity.
Furthermore, our analysis provides valuable directions for future research in FL. The relationship discovered between the orthogonality of weights and the bound on MSE opens up new avenues for investigation. It could, for instance, prompt more in-depth research into regularization techniques that aim to achieve orthogonality, improving model stability across diverse data distributions.
In simpler terms, our theoretical analysis is much like a 'roadmap', helping us understand how our model reacts to changes in data distributions, the role of orthogonal weights in this context, and where we could focus future research efforts to improve our approach. This roadmap makes the journey of applying FedSTO to federated learning more navigable, ultimately leading to more effective and robust models.
G Detailed Experimental Settings
YOLOv5 Architecture The YOLOv5 object detection model is based on the YOLO algorithm, which divides an image into a grid system and predicts objects within each grid. YOLOv5 uses a convolutional neural network (CNN) to extract features from the image and then uses these features to predict the location of the bounding boxes (x,y,height,width), the scores, and the objects classes. The YOLOv5 architecture consists of three main parts: the backbone, the neck, and the head. The backbone is responsible for extracting features from the image. The neck combines the features from the backbone and passes them to the head. The head then interprets the combined features to predict the class of an image. Here is a more detailed description of each part of the YOLOv5 architecture:
- Backbone: The backbone of YOLOv5 is based on the CSPDarknet53 architecture. The CSPDarknet53 architecture is a modified version of the Darknet53 architecture that uses a cross stage partial connection (CSP) module to improve the performance of the network. The CSP module allows the network to learn more complex features by sharing information across different layers.
- Neck: The neck of YOLOv5 is a simple convolutional layer that combines the features from the backbone.
- Head: The head of YOLOv5 is composed of three convolutional layers that predict the location of the bounding boxes, the scores, and the objects classes. The bounding boxes are predicted using a regression model, while the scores and classes are predicted using a classification model.
YOLOv5 has been shown to be effective for object detection, achieving good results on a variety of object detection benchmarks. It is also fast, making it a good choice for real-time object detection applications.
Annotations The YOLO object detection algorithm divides an image into a grid system and predicts objects within each grid. The annotations for YOLO are stored in a text file, with each line representing an object in the image. The format of each line is as follows:
[object_id], [x_center], [y_center], [width], [height], [score]
where object_id is the ID of the object, x_center is the x-coordinate of the center of the object's bounding box, y_center is the y-coordinate of the center of the object's bounding box, width is the width of the object's bounding box, height is the height of the object's bounding box, and score is the confidence score of the object detection.
Loss Functions YOLOv5 is an object detection model that is trained using a loss function that combines three terms: a class loss, a box loss, and a confidence loss. The class loss is a cross-entropy loss that measures the difference between the predicted class probabilities and the ground truth class labels. The box loss is a smooth L1 loss that measures the distance between the predicted bounding boxes and the ground truth bounding boxes. The confidence loss is a binary cross-entropy loss that measures the difference between the predicted confidence scores and the ground truth binary labels indicating whether or not an object is present in the image. The overall loss function is minimized using stochastic gradient descent. Here is a more detailed explanation of each loss term:
- Class loss  $\mathcal{L}_{cls}$ : The cross-entropy loss is a measure of the difference between two probability distributions. In the case of YOLOv5, the two probability distributions are the predicted class probabilities and the ground truth class labels. The cross-entropy loss is minimized when the predicted class probabilities are identical to the ground truth class labels.
- Box loss  $\mathcal{L}_{obj}$ : The smooth L1 loss is a measure of the distance between two sets of numbers. In the case of YOLOv5, the two sets of numbers are the predicted bounding boxes and the ground truth bounding boxes. The smooth L1 loss is minimized when the predicted bounding boxes are identical to the ground truth bounding boxes.
- Confidence loss  $\mathcal{L}_{conf}$ : The binary cross-entropy loss is a measure of the difference between two binary distributions. In the case of YOLOv5, the two binary distributions are the predicted confidence scores and the ground truth binary labels indicating whether or not an object is present in the image. The binary cross-entropy loss is minimized when the predicted confidence scores are identical to the ground truth binary labels.
H Discussions on Semi-Supervised Object Detection
We implement a similar strategy to the pseudo label assigner of the semi-efficient teacher [38]. We aim to provide a comprehensive description of our training method and pseudo label assignment approach in this section.
H.1 Adaptive Loss
The adaptive loss function in our SSFOD framework is comprised of two key parts: a supervised loss  $(L_{s})$ , calculated from labeled data, and an unsupervised loss  $(L_{u})$ , generated from unlabelled instances.
The supervised loss,  $L_{s}$ , adheres to conventional practices used in object detection tasks. This incorporates the amalgamation of cross-entropy loss, responsible for classification, and CIoU (Complete Intersection over Union) loss, accounting for bounding box regression:
$$
L _ {s} = \sum_ {h, w} \left(C E \left(X _ {c l s} ^ {(h, w)}, Y _ {c l s} ^ {(h, w)}\right) + C I o U \left(X _ {r e g} ^ {(h, w)}, Y _ {r e g} ^ {(h, w)}\right) + C E \left(X _ {o b j} ^ {(h, w)}, Y _ {o b j} ^ {h, w}\right)\right), \tag {4}
$$
where  $CE$  is the cross-entropy loss function,  $X^{(h,w)}$  signifies the model's output, and  $Y^{(h,w)}$  corresponds to the sampled results of pseudo label assigner (i.e., local EMA updated model).
The unsupervised loss component,  $L_{u}$ , is an addition that leverages the pseudo labels generated by the local EMA updated model. The objective of  $L_{u}$  is to guide the model to exploit beneficial information embedded within unlabelled data. It is computed as:
$$
L _ {u} = L _ {c l s} ^ {u} + L _ {r e g} ^ {u} + L _ {o b j} ^ {u}, \tag {5}
$$
where  $L_{cls}^{u}, L_{reg}^{u}$ , and  $L_{obj}^{u}$  denote the losses associated with classification, bounding box regression, and objectness score, respectively. Each of these losses uses pseudo labels and is precisely tailored to foster reliable and efficient learning from unlabeled data.
- Classification Loss,  $L_{cls}^{u}$ , hones the model's class-specific accuracy. It is calculated only for pseudo labels whose scores are above a predefined high threshold  $\tau_{2}$ , using a cross-entropy loss function. The divergence is gauged between the model's predicted classification scores and the class scores from the pseudo labels provided by the Pseudo Label Assigner.
- Regression Loss,  $L_{reg}^{u}$ , scrutinizes the precision of the bounding box predictions. This loss applies to pseudo labels with scores above  $\tau_{2}$  or objectness scores surpassing a value typically set at 0.99. A CIoU loss function is employed to measure the discrepancy between the predicted bounding box regressions and those associated with the pseudo labels.
- Objectness Loss,  $L_{obj}^{u}$ , evaluates the model's objectness prediction capability, essentially the model's confidence in a given bounding box containing an object of interest. All pseudo labels contribute to this loss, though the computation varies depending on the pseudo label score. For scores below  $\tau_{1}$ , the loss is computed against zero. For those above  $\tau_{2}$ , the loss compares against the objectness of the pseudo label. Scores between  $\tau_{1}$  and  $\tau_{2}$  result in the loss calculated against the soft objectness score of the pseudo labels.
This strategic amalgamation of losses forms the underpinning of the unsupervised loss function, fostering efficient and reliable learning from unlabeled data. By harnessing the potential information embedded within pseudo labels, our model aims to significantly enhance semi-supervised object detection in an FL setup.
In a centralized setting, the supervised and unsupervised losses would typically be trained jointly. However, in our federated scenario, where the server possesses only labeled data and the client holds exclusively unlabeled data, we adopt an alternate training approach. The server focuses on optimizing the supervised loss  $L_{s}$  with its labeled data, while the client concurrently refines the unsupervised loss  $L_{u}$  using its unlabeled data. This methodical division of labor allows us to leverage the distinctive characteristics of both types of data and facilitates efficient learning in a federated environment.
H.2 Local EMA Model for the Pseudo Labeler
In SSFOD framework, we employ a Local Exponential Moving Average (EMA) model for generating pseudo labels, enabling the judicious utilization of unlabeled data at each client's disposal. The concept of the EMA model hinges on an infinite impulse response filter that assigns weightages decreasing exponentially, offering a balanced consideration of both historical and immediate data points for reliable predictions.
Each client maintains a local EMA model in our method. The weights of this local EMA model are a weighted average of its own weights and the weights of the global model. This relationship can be described mathematically as follows:
$$
W _ {\mathrm {E M A} _ {\text {c l i e n t k}}} ^ {(t)} = \alpha W _ {\mathrm {E M A} _ {\text {c l i e n t k}}} ^ {(t - 1)} + (1 - \alpha) W _ {\text {c l i e n t k}} ^ {(t)} \tag {6}
$$
In Eq. (6),  $W_{\mathrm{EMA}_{\mathrm{client}k}}^{(t)}$  symbolizes the weights of client k's local EMA model at round  $t$ , and  $W_{\mathrm{client}k}^{(t)}$  represents the weights of the client k's model at round  $t$ .  $\alpha$  is the EMA decay rate dictating the pace of depreciation of the client's weights' influence. Typically, the decay rate resides in the 0.9 to 0.999 range, subject to the specific application.
Table 9: Performance on the BDD dataset with 1 labeled server and 3 unlabeled clients. It highlights how each added method contributes to the overall performance under both Non-IID and IID conditions.
Method	Non-IID	IID
Cloudy	Overcast	Rainy	Snowy	Total	Cloudy	Overcast	Rainy	Snowy	Total
Vanilla	0.560	0.566	0.553	0.553	0.558	0.572	0.588	0.593	0.610	0.591
Freezing Only Head	0.531	0.603	0.567	0.565	0.567	0.558	0.613	0.614	0.593	0.595
Freezing Neck & Head	0.571	0.583	0.557	0.556	0.567	0.576	0.578	0.594	0.599	0.587
In pursuit of a consistent basis across all clients, the weights of the Local EMA model are reinitialized with the global model's weights post each server broadcast. This particularity of our framework ensures a harmonious coexistence of global model consistency and local data awareness.
The system enables the local EMA model of each client to gradually tune to the unique local data characteristics, by integrating updates from the client's unlabeled data. As a consequence, we achieve a potent blend of personalization and model performance in a FL environment, all the while alleviating communication overheads. The design acknowledges the distinct data distributions that each client may harbor, allowing the model to adapt accordingly, thereby augmenting the learning process's efficacy and efficiency.
Following the weight updates, the local EMA model functions as the pseudo labeler for each client. This approach ensures the generation of stable and reliable pseudo labels, despite the limited interactions with the global model. In the Federated Learning scenario, where communication costs carry substantial significance, this feature is crucial.
Our local EMA strategy presents several benefits. It guarantees independent pseudo label generation at each client's end without necessitating frequent server updates. In addition, it enhances the learning process's robustness, as the local EMA model remains largely unaffected by the client model's noisy gradient updates. As a result, the system produces more trustworthy pseudo labels, ultimately culminating in improved performance.
I Discussions on Selective Training
In our SSFOD framework, Selective Training plays a pivotal role in achieving superior performance and enhanced computational efficiency. Traditionally, in selective training, all components beyond the backbone, typically referred to as Non-Backbone parts, are frozen to reduce computational complexity and improve training efficiency. However, It may not be necessary to freeze all of the non-backbone parts.
In our preliminary investigations, we discover that freezing only the head, a component of the Non-Backbone parts, can yield similar performance levels as freezing the entire Non-Backbone. This observation opens the door to potentially more efficient and flexible models in the context of selective training, as it suggests that selective freezing could be just as effective as complete freezing (Table 9).
Nonetheless, in consideration of the inherent nature of FL, we ultimately decide to continue freezing both the neck and head components. We reach this decision primarily due to the communication cost considerations, which carry considerable weight in a federated learning setting. Despite the promising results observed when only freezing the head, freezing both the neck and head does not result in any noticeable performance decrement, while it does markedly reduce the communication cost.
The implications of this choice are particularly relevant in the context of FL environments, where minimizing communication costs is paramount for practical deployment. We posit that this finding may inform future strategies for selective training, suggesting that selective freezing of specific Non-Backbone parts can effectively balance performance and efficiency in FL environments.
Table 10: Performance under random distributed cases of Cityscapes [4]. FedSTO exhibits improvements under various object categories, and significantly outperforms the performance for unlabeled clients.
Method	Labeled	Unlabeled
Categories
Person	Car	Bus	Truck	Traffic Sign	Person	Car	Bus	Truck	Traffic Sign
ORN from scratch	0.470	0.694	0.449	0.074	0.393	0.437	0.720	0.507	0.100	0.378
Freezing only Head + FPT with ORN	0.504	0.697	0.353	0.239	0.411	0.486	0.740	0.420	0.078	0.415
Freezing Neck & Head + FPT with ORN	0.504	0.720	0.342	0.261	0.415	0.487	0.740	0.460	0.181	0.437
J Further Discussions on Full Parameter Training (FPT) with Orthogonal Enhancement
As part of our ongoing evaluation of our SSFOD framework, we conduct comprehensive experiments involving different training strategies, in particular, Full Parameter Training (FPT) with Orthogonal Regularization (ORN). Through these experiments, we compare three distinct approaches: