FOOGD: Federated Collaboration for Both Out-of-distribution Generalization and Detection
Authors: Xinting Liao
Weiming Liu
Pengyang Zhou
Fengyuan Yu
Jiahe Xu
Jun Wang
Wenjie Wang
Chaochao Chen
Xiaolin Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: Federated learning; Out-of-distribution;
Primary area: privacy
TL;DR: We devise a FL framework that adapts to wild data, which coexists with non-IID in-distribution (IN) data, covariate-shift (IN-C) data, and semantic-shift  (OUT) data.

Abstract: Federated learning (FL) is a promising machine learning paradigm that collaborates with client models to capture global knowledge. However, deploying FL models in real-world scenarios remains unreliable due to the coexistence of in-distribution data and unexpected out-of-distribution (OOD) data, such as covariate-shift and semantic-shift data. Current FL researches typically address either covariate-shift data through OOD generalization or semantic-shift data via OOD detection, overlooking the simultaneous occurrence of various OOD shifts. In this work, we propose FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL process. Firstly, SM3D in FOOGD estimates score model for arbitrary distributions without prior constraints, and detects semantic-shift data powerfully. Then SAG in FOOGD provides invariant yet diverse knowledge for both local covariate-shift generalization and client performance generalization. In empirical validations, FOOGD significantly enjoys three main advantages: (1) reliably estimating non-normalized decentralized distributions, (2) detecting semantic shift data via score values, and (3) generalizing to covariate-shift data by regularizing feature extractor. The project is open in https://github.com/XeniaLLL/FOOGD-main.git.

FOOGD: Federated Collaboration for Both Out-of-distribttion Generalization and Detection
Xinting Liao', Weiming Liu', Pengyang Zhou', Fengyuan Yu', Jiahe Xu',
Jun Wang', Wenjie Wang', Chaochao Chen', Xiaolin Zheng
Zhejiang University, OPPO Research Institute, 3 National University of Singapore (xintingliao, 21831010, zhoupy, zjuccc, xlzheng) edu. en, junwang lu@gmail. com, wenj senjiewang96@gail.. com
Abstract
Federated learning (FL) is promising machine learning paradigm that collaborates with client models capture global knowledge. However, deploying FL models real-world scenarios remains unreliable due to the coexistence of in nn-distribution data and unexpected citoo--distribution (OOD) data, such as covariate-shif and semantic-shift data. Current FL. researches typically address either covariate-shift data through OOD generalization or semantic-shift data via OOD detection, over- looking the simultaneous occurrence of various OOD shifts. In this work, we propose FOOGD, a method that estimates the probability density of each client and obtains reliable global distribution as guidance for the subsequent FL pro- cess. Firstly, SKID in FOOGD estimates score model for arbitrary distributions without prior constraints, and detects semantic-shif data powerfully. Then SAG in FOOGD provides invariant yet diverse knowledge for both local covariate-shin generalization and client performance generalization. In empirical validations, FOOGD significantly enjoys three main advantages: (1) reliably estimating non- normalized decentralized distributions, (2) detecting semantic shift data via values, and (3) generalizing to covariate-shift data by regularizing feature extractor. The prejoct open in /ttts://gittubl www.oo/XeniiaaLLLL/
git
1
Introduction
Federated learning (FL) [56 provides distributed machine learning paradigm, which collaboratively models decentralized data resources. Specifically, each client models its data locally and server improves model performance by aggregating client models, which indirectly shares knowledge among clients and preserves privacy. FL further makes efforts adapt real-world scenarios, i.e., adapting non-independent and identical distribution (non-IID)
39 30
Beyond non-llID issues, deploying FL models in real- world also encounters different tasks of out-of- distribution (OOD) shift (69 26, (), e.g., tackling covariate shifts (OOD generalization) and handling semantic shifts (OOD detection). In FL, OOD generalization task is devised to capture the invariant data-label relationships of covariate-shin data intra- and inter-client, which offers the potential of
adapting unseen clients [ 60 70, 84] The OOD detection task in FL aims to find semantic-shift data samples that do not belong to any known categories of all client data during FL training [83] Both OOD generalization and detection simultancously exist in FL, hindering the deployment of FL methods. Nevertheless, the existing work only tackles each OOD task in isolation. SCONE B proposes unified basged-based framework realize OOD generalization and OOD detection tasks
"Coresponning author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

in centralized machine learning. But it is infeasible to FL due to two reasons, i.e., being non- con-trivial in searching for consistent margin among non-IID distribution, and requiring outlier exposure of data. This motivates us to a crucial yet unexplored question:
Can we devise a framewok that adapts to wild data, which coexists with non-IID in-distribution (IN) data, covariate-shift (IN-C) data, and semantic-hhift (OUT) data?
In this work, we simultaneously promote OOD generalization and detection by collaborating with clients in FL. The objectives of OOD gen- eralization and detection vary among different clients due to their non normalized and hetero- geneous probability densities. This motivates us to build systematic and global guidance to distin- guish IN, IN-C, and OUT data. As depicted in Fig. T for non-IlI client distributions, we first estimate the probability density in each local client and then compose these local estimations for global distribution in server. Once a reliable global distribution estimation is established, we can leverage it to guide FL OOD tasks in deployment. However, this approach presents two chal- lenges, i.c., CHI: How to estimate the reliable and global probability density among decentralized clients for detection? and CH2: How to enhance intra- and inter-client OOD generalization based on global distribution estimation?
x OUT Data A IN-
O IN Data
Deciioon Boudary between Data
Figure 1: Motivation of FODGD. The distributions of two clients are non-IlI. and we seek to estimate the global distribution among decentralized data.
To fill these gaps, we propose federated collaboration framework named as FODGD, which estimates client distribution in feature space via score matching with maximum mean discrepancy (SM'D) and enhances the client model generalization by Stein augmented generalization (SAG). To solve CH1, inspired by the flexibility of score matching [58] 7]. we originally devise SM to train score model that estimates limited and heterogeneous data distributions for each client, and aggregate score models in server as global estimation. Because the score values are vectors indicating position and changing degree of the log data density [55). SMID brings the potential of discriminating OUT data in low-density areas with large change degree. However, it is unreliable to directly apply vanilla score matching for modeling decentralized data, which suffers from sparsity and multi- modal complexity (72 (55) To obtain a reliable density estimation, SKIDD explores wider space by generating random samples via Langevin dynamic sampling, and constrains the generated samples to be similar to data samples via maximum mean discrepancy (MMD). To mitigate CH2, SAG regularizes feature invariance between data samples and its augmented version, which is measured by score- based discrepancy. Though the existing generalization methods capture the invariance in feature space [44 the vital feature information inevitably lost due to strictly invariant constraints 87 10) This also deteriorates the performance of solving FL OOD generalization. With the benefits of distributional alignment based on Stein indentity [46). SAG in client model captures IN-C data in a similar feature space with IN data, which not only avoids representation collapse but also maintains diversifying information. Thus SAG makes FOOGD generalize to IN-C data from local covariate-shift distribution and unseen client distribution.
The main contributions are: (I) We are the first to study OOD generalization and detection in FL simultaneously, and formulate a evaluation on deploying FL methods in the wild data. (2) We propose FOOGD which estimates reliable global distributions based on arbitrary client probability densities, to guide both OOD generalization and detection. (3) We devise SMCD which not only explores wider probability space for density estimation, but also provides the score function values to detect OUT samples. (4) We utilize SAG to maintain the invariance between IN-C and IN data in feature space, which obtains better generalization without collapsing for FL scenarios. (5) We provide theoretical analyses and conduct extensive experiments to validate the effectiveness of FOOGD.
2 Related Works
2.1 OOD Detection
OOD detection discriminates semantic shift (OUT) data during deployment time 3 20 53 There are two main categories of OOD detection work, i.c., enhancing training-tine time regularization [48]
2

---

21 73 [3], and measuring post-hoe detection function of a well-trained model [20] 74, 37 The
first category focuses on ensuring predictors produce low-confidence predictions for OOD data during training, which is effective but mainly requires access to real OUT data 221 S g 86 By
the way, selecting different auxiliary detection objectives [22 57 unexpectedly varies the overall
performance. The second category utilizes the classification logits 74. energy score [35] 48) and
feature space estimation [37 666 from pre-trained models, to detect the OUT data. This reduces the costly computation burden but rigidly relies on the data distribution captured in the pre-trained model. As one kind of methods in post-hoc way, density-based estimation methods [7 666 [8] can relieve the cost of collecting or synthesizing representative OOD datasets, avoiding biased and ineffective detection 74 35 and bringing the potential of densities composition.
2.2 OOD Generalization
OOD generalization targets extracting invariant feature-labe relationships and maintaining the de- ployment performance of model with covariate-shift data in the open-world world [3] [4] To reach this goal, IRM-based work [2 24 utilizes invariant risk regularization to find invariant representations from different covariate shift data. Besides, there are various work calibrating invariant represen- tations by distribution robust optimization methods [65 [6]. feature alignment methods [15] [4], augmentaed training [10), gradient manipulation methods 24], diffusion modeling 82 and so on.
SCONE 3 takes advantage of unlabeled wild mixture data to enhance generalization and build detectors simultaneously. However, SCONE is not suitable for Fl., since it requires a hyper-parameter of energy margin and the outlier exposure data (83 75). To tackle the meta-task detection and generalization, Chen (9] propose an Energy--Bssed Meta- Metannea (EBML) framework that learns meta-training distribution via two energy-based neural networks. However, it is tough to model two reliable energy models in decentralized models where data and computation resources are constrained.
2.3 Federated Learning with Wild Data
In FL, wild data makes it challening in tackling non-lID modeling, OOD generalization. and OOD detection. Firstly. FL with non-IID data presents significant challenges in balancing global and local model performance [56] 8 39 43
89 42 (Appendix [ Secondly, FL considers two aspects of generalization, i.e., (1) intra-client generality, and (2) inter-client generality. The intra-client generalization keeps the invariant relationship between data samples and class labels 69] 770 63] which is similar to centralized OOD generalization. The inter-client generality work captures invariant representation for heterogeneous client distributions, making the global model adaptive to a newly Lastly, regarding OOD detection FL, it is expected to detect semantic shift data out of the whole class categories set among decentralized data, yet avoid wrongly distinguishing unseen data classes of other clients. FOSTER [83] treats unseen data classes in each client as OUT, and enhances their detection capability via synthesizing virtual data with external classes of other clients. Different from the above methods, we aim to enhance OOD detection and generalization simultaneously by collaborating with different clients. Recently, FedGMM (77) utilizes a federated pnneetaationmaximizzaee estimating Gaussian mixture (Gdel and detects OUT data via computing GMM probability. It can only roughly capture the data distribution with the prior assumption of GMM. Meanwhile, orthogonal paradigm of studies focus on tackling concept shifts in federated process [6] 29).
unseen client S4 60
algorithm to fit data distribution among clients by
However, it overlooks the coexistence of wild data, resulting in suboptimal performance in federated tasks of OOD generalization and detection.
3 Methodology
3.1 Problem Setting
Federated Learning Formulation with Wild Data. We first formulate the wild data in FL de- ployment and provide the optimization goal of FL. Empirically, we assume a dataset decentralizes among K clients, i.c., D = UkEjK) Dr. The data distribution of k-th client is simulated following the real-world wild data, i.e., Dr = DIN + DIN + DOUT The objective of the FL model, which simultaneously tackles OOD generalization and detection, is defined as follows:
argming, , ,, > i (C,(0,, Og: Dz)),
(I)
3

---

llent
SM Detection
Scora Sunxttonn
Server
Client it
SM-D Detection
Cumction
Global
Global
Client
D: Detection
Function
Score M odiel
samples X
samples of
M MMD
logPp(
Extractor
LDS
Soore Model Composition
Global
Feature
Model
KSD
Fy Ingry(e)
OUT
Cliet
C Client
Data A ugmentation
SAG: Generalization
Figure 2: Framework of FOOGD. For each client, we have main task feature extractor, a SMDD module estimates score model (Eq. (8) for detection, and a SAG module regularizes feature extractor for enhancing generalization. The server aggregates models and obtains global distribution.
where La Dz; AN ANCC + EUT and we represents weight ratio for the k-th client. The PIN-C ØUU correspondingly justify the IN generalization, IN-C generaliza
OOD measurements & " tion, and OUT detection each client k, as follows:
t = Baap))app (T (Upedd (fo(z)) + v})
(a)
aN Biz.o))rogee (TI (preed (fo(z)) # v})
(b)
(2)
OLT = B(a.s)--pur (I (g IN})
(c),
where fo(-) is main task model, (o(+) (.) is detector, II is indicator function, and ypeed is predicted label.
Framework Overview. To optimize the FL objective in Eq. (I), propose FOOGD whose framework overview is depicted in Fig. - For K clients with non-IID data, FODGD composes their local distributions and aggregate their model parameters in server. In each client, the data samples as well as its fourier augmented [79] counterparts X, are fed into the same feature extractor of main task fo (+) to obtain their latent features,
fe(x) and 2 fe()), respectively. To avoid overwhelming communication costs brought by score models, score matching with maximum mean discrepancy (SM'D) trains score model so(:) in feature space. This model captures the data distribution by estimating the gradient of log densities (score functions) of latent features z, i.e., se- (z) = Vx log log (z) s V log (D(2) (z) [2]. Then score model serves as the detector for the objective in Eq. (2b), discriminating OUT based on the norm of score function values. Besides, Stein augmented generalization (SAG) enhances the generalization capabilities of the feature extractor fe()). by the distribution regularization defined via score model. Because score model based distribution ensures that data features and their neighboring augmented samples, e.g., z and maintain consistent probability space [46]. The local modeling iterates until performance converges.
In each communication round, since both main task model and score model are parameterized neural networks, it practical to follow conventional weighted average aggregation [56], i.e.,
K
(0.. 0,) = Σ
wk @ 07)
with wi = E [ VλE[K]- These collaborative processes among clients continue until the global
[Dal
model converges, bringing reliable and comprehensive global distribution in the form of global score model. We introduce the details later and illustrate the algorithm of FOOGD in Appendix Algo.
3.2
SM Estimating Score Model for Detection
In this part, we introduce the estimation of FL data distribution and how to utilize it for detection. As shown in Fig. [ a reliable probability density is eagerly necessary for distinguishing IN and OUT data 75 27 Different from existing centralized OUT aware and OUT synthesis methods [ 21) the FL. framework suffers from the accessibility of OUT data [8] In this study, we aim explicitly capture the local IN data distribution of clients, and subsequently compose them to reliable global

---

distribution for discrimination. However, it remains challenging to estimate heterogeneous and non-normalized probability density without prior information during FL modeling.
Dynamic Feature Density Estimation. FOOGD estimates score model via score matching in the feature space 72 32 7 i.e., Pp(z), circumventing the need for prior distribution knowledge or distribution normalization [72]. Moreover, it alleviates the computational burden by modeling the score of latent representations in a smaller, yet more expressive and continuous space, compared to the scores of the original data (7T] Specifically, given the latent features z = [o(æ), we perturb via adding random noise v ~ (0 I) to obtain E = + ov, which follows moise-perturiee data distribution Po (=]=) == A (ž; r²)). And we model it with noise conditional score model [67 so (2, σ) by minimizing the denoising score matching (DSM) loss, i.e.,
min fSM
p (*)) || (ž, V lo =)]]
2
where the score function of V log (2 2) for d-dimensional features, is computed as follows:
1
exp {
=1 20²
}
2
V (2 V
log
(5)
(22m0²)
0
When the noise get to zero, i.e., + O, we have the exact score values ag (2, - (z). However, score model based density estimation will inevitably fail once the distribution contains sparse data samples [67 [55 or multiple modalities (32). as shown in Fig. (a). SM is motivated to broadly explore the generated random features Zgen that samples from the whole distribution space. In detail, SMCD first sample from random distribution, e.g., Normal distribution, as the start latent features, ie., 2d ~ (0, I). Then SMCD utilizes T-step Langevin dynamic sampling (67] (LDS) from density vector fields modeled by the score model, to derive generated latent features Zgen z²:
50(22"1 .)) + Vew', 2
(6)
with indicating the step size and w' ~ NN I) introducing stochasticity in each step. Lastly, the distribution of batch of the generated features Zeen {zgnn J= i.c., Pgen (Zgon), is supposed to approximate the distribution of original features Z {z, ) i.e., Pz(2), with the calibration of
maximum mean discrepancy (MMD(Z, Z, gea)) matching:
MMMD = E2n, [K(ZD, 200)1 2E, ppi Pa [K gg)n)l + EL Es Peac (ki(Zgon Z )
(7)
where k(2, 2') exp( | >
22) with bandwidth h
is Gaussian kernel function 447 49 within a unit ball in
universal Reproducing Kernel Hilbert Space (RKHS). Be-
cause MMD is non-parametric method that accurately measures the distance between two densities in RKHS, it provides reliable estimations and adapts well to complex data modalities [4). This approach mitigates the limita- tions of directly using DSM to estimate distributions by exploring a wider feature space. Unfortunately, as de- picted in Fig. (d), simply using MMD matching does not enhance density estimation, when the target distribution unknown or inaccurate. But it is quite necessary that the latent distribution is inaccurate and heterogencous in FL. To fill this gap, SMCD seeks to harness and integrate the strengths of both density estimation paradigms, via a trade-off coefficient Am
(6) 2,-0
(0 Ar
(4) R.
(40 A 1.0
Figure 3: Motivation of SKOD. Red
points are sampled from target data dis- tribution, and the blue points are gener- ated by LDS in Eq.
OUT (1 Am (DSM AMDD
(8)
In this way, S brings an accurate and flexible implementation for non-normalized data distribution. The implementation procedure of SKID is in Appendix A 2 To illustrate the effectiveness of SWOD we further visualize density estimation of 2-D toy example in Fig. 3 In detail, we model the red target points by tuning series of coefficients, i.c., Am {0, 0.1, 0.5, 1] in Eq. (8) As
5

---

can see, with the mutual impacts between score matching and MMD estimation, SM has more compact density estimation when An 0.1, compared with blankly using score matching (An, = 0) or simply using MMD (Am = 1). As brand new objective of density estimation, SMCD expands the searching range and depth of score modeling, making possible to comprehensively model data density. Moreover, with the calibration of MMD estimation, original data features and the generated samples based on the score model are effectively matched. Hence SM could ensure a more aligned and reliable density estimation for sparse and multi-modal data.
OOD detection in clients. Remind that the score function indicates the gradient of the log density, which are actually vector fields pointing to the highest density area, as shown in the score function visualization of Fig. 2 The IN data should point to the high density and reflect the distance via its vector norm. While the OUT data cannot present this satisfying property and further exposure boldly, since the OUT data is always in low-density area 48] [55] That is, the norm of the score
||s@- (z)ll [=po-(*) decreases in regions of higher density, while increases in lower density.
pe- (2)
indicates the larger the norm of score is, the more likely the data sample is OUT. For negative threshold T < O, we have detection score function:
(<)) = True, when ||s0~ (So- (x))]l -T; otherwise, [aOUT( = False.
3.3
SAG: Enhancing Feature Extractor for Gerneralization
In this section, we will illustrate how to enhance generalization capability of feature extractor in FOOGD. In FL scenarios, solving OOD generalization needs not only to keep the local IN-C data classification correctly, but also to maintain performance consistency among all participating clients. The non-lID issue creates a contradiction between achieving both targets. This is because enhancing IN- accuracy intra-client requires diversification across different classes, whereas inter-client generalization benefits from all IN data being closely clustered, irrespective of class distinctions. Hence, it expected to balance the feature diversification of different classes and the feature consistency of in-distribution, to realize the consistent data-label relationships intra- and inter-client.
Diversifying Feature Invariance Augmentation. FOOGD regularizes invariance among client feature extractors using distribution-awwe aware divergence between the original data æ and its augmented version a == Txx)) by transformation T. To address this, we propose SAG, which utilizes global distribution and optimizes distributional invariance between latent features of the original and augmented data. This approach maintains the distinguishable diversification of features and consistent data-label mapping across clients.
In the feature space, SAG regularizes original data samples to be aligned with augmented ones, i.e.,
aligning = = fe (x) and 2 = fe However, directly computing the norm regularization between and will cause mode collapse (28 in FL, further degrading the estimation of score model (-) based on SM While contrastive methods [62 73 51 50] like FedICON [69]. and L-DAWA [64] ensure diversification and alignment for generalization, they rely on selecting negative samples instead of leveraging global distribution knowledge. Consequently, they fail to maintain consistent
invariance among clients. Instead, SAG alternatively introduces kernelized Stein operator guided by score function, i.e.,
(10) where (z)) is implemented with kernel function k(.,-) mentioned in Eq. 46).. while p(z) and (() are the distributions for a batch of features Z [ and Z = (2,) :- respectively. By utilizing the kernelized Stein operator, SAG encourages the samples of augmented features to align with high probability regions of the original features. Additionally, the second term of (T)) improves feature diversification and prevents data from collapsing directly to the original distribution modes. According to fundamental theory named as Stein identity, i.e., Fey(()) (x)( = 0 for arbitrary distribution q(x) 46), Stein operator brings the potential of measuring two data distributions with the guidance of global distribution estimation. Because score models capture local probability densities and are aggregated into global score model on the server, they inherit distribution information from all participating clients. Specifically, we first illustrate kernelized Stein discrepancy (KSD) [46] 41 45 that measures the distribution discrepancy between original data p(z) and augmented data (2):
Apo(2) = v + v=((z),
(IS((p(z).q())
= 13 (2) se (2' (2')((2 2') + s0) T kZ,)(,))))
(11)
+ s0 ( V E') + trace V (=,,

---

We provide the full induction of KSD between original data and augmented data in Appendix B.2 And KSD(p(=), (()) equals zero if and only if p(z) and q( are the same. By taking the derivative of KSD, we can obtain the updating direction of moving 2 towards which not only keep the invariance of features, but also guarantee diversification avoiding collapse. Therefore, the augmented representation Z has minimal KSD with the original latents Z. This ensures that the final objective of the feature extractor is to minimize the subsequent classification error (between predictions ypred and ground truth Y and achieve invariant alignment:
eNN gIN-C CrossEntropy (Y pred Yg²) + Xo KSD(p(() (Z)).
(12)
Besides, the score model in Eq. (TT communicates among different clients obtain the global distribution, making it possible to be reliable guidance of invariance among clients. This makes SAG a potential generalization approach for modeling feature invariance in the overall FL scenario, even acting warm-start for unseen clients. Therefore, FOOGD is capable of both local IN-C data generalization and consistent performance generalization of clients. The algorithm of SAG can be found in Appendix A Algo.
4 Theoretical Discussion
In this section, we provide the error bound of modeling score model via SM in federated scenarios, and provide the error bound in Theorem 4.1 Besides, the federated training procedure of score model is the same with the main task model. This indicates that our federated learning convergence bound unchanged, following (40) We provide more theoretical details in Appendix B
Theorem 4.1 (Error Bound of Decentralized Score Matching via SM*D). Assume the original
MMD(Z, Z, gen) SC CCror randomly initialized score model (0(z) in Eq. (7). the score model achieves optimum and MMD decreases. By Lemma
B.1
we can obtain the final error bound of global so(.) as:
v Epn(=)]ll
IDC B
[])0(z) V (z)
V~ (og(p(2))111
(13)
a²
where C. is the upper bound of the MMD, B is batch size, and [D] is the data amount.
5
Experiments
5.1 Experimental Setups
Datasets. Following SCONE 3 we choose clear Cifar10, Cifar 100 133 and TinyImageNet 336 as the IN data, and select the corresponding corrupted versions (19, i.e., Cifar10-C, 100ar and TinylmageNet-C. as IN-C data. We evaluate detection with five OUT image datasets: SVHN [59). Texture (UII, iSUN (78), LSUN-C and LSUN-R [81). To simulate the non-IID scenarios, we sample data by label in a Dirichlet distribution parameterized by non-lID degree 223 i.e., a, for K clients. The smaller a simulates the more heterogeneous client data distribution in federated settings. To evaluate FOOGD on unseen client generalization data, we also use PACS 38 dataset for leave-one-out domain generalization. Details of dataset simulation are Appendix D.
Comparison Methods and Evaluations. We study the performance of FOOGD with the state-of -he- art (SOTA) federated learning model and FedAvg-like derivant of SOTA centralized OOD methods, i.e., LogitNorm 76) (FedLN), ATOL 88) (FedATOL). T3A
(3 (FedT3A). We compare FOOGD with three types of baseline models, i.e., (1) Vanilla FL model: FedAvg [56] and FedRoD [8 (2) FL with OOD detection: FOSTER 83 FedLN, and FedATOL, (3) FL with OOD generalization: FedT3A.
FedUR 17). FedTHE 26) FedICON [69 For evaluation, we report the accuracy of IN data (ACC-
IN) and IN-C data (ACC-IN-C) to validate IN generalization and OOD generalization, respectively. We compute the maximum softmax probability (0) (MSP) and report the standard metrics used for OOD detection, i.c., the area under the receiver operating characteristic curve (AUROC), and the false positive rate at threshold corresponding to a true positive rate of 95% (FPR95) [20]
Implementation Details. We choose WideResNet [85 and R8 T8 for TinylmageNet and PACS, and optimize each model local epochs per communication round until converging with SGD optimizer. We conduct all methods at their best
our main task model for Cifar datasets,
7

---

(a) FedATOL
(b) FodAvg+FOOGD
(c) FedTHE
(d) FedRoD+FOOGD
Figure 4: T-SNE visualizations of FedAvg and FedRoD with FOOGD.
and report the average results of three repetitions with different random seeds. We consider client number K 10, participating ratio of 1.0 for performance comparison, and the hyperparameters Am = 0.5, Aa 0.05. We provide the full implementation details in Appendix D.2
Table 1: Main results of federated OOD detection and generalization on Cifar10. We report the ACC of brightness IN-C ACC, the FPR95 and AUROC of LSUN-C as OUT performance.
Non IID
0.1
0.5
Metbod
ACC
ACC IN
FPRSS.
AUROCT
XCC-IN
ACC-IN
FPRS5]
AUROCT
JCC
ACL C
FFR95.
AUROCT
FedArg
58.00
65:44
53.41
58.00
66.59
83.72
43.90
64.15
56.30 51
$5.06
15:24
65.37
Foll
75.34
71:77
56.14
64.14
16:10
$4.10
19.
90.54
$2.30
85.08
13:33
90.87
FalATOI.
55.09
54.44
49.50
96.23
67:55
$5.64
27:87
99.48
59.37
05.75
19.66
95:25
FaITIA
$8.03
61.52
73.12
63.64
$6.59
$2.85
43.70
$4.18
86: 50
$5.00
38:24
83.37
FaIlRR
68.36
66.12
79.48
63.31
56.75
$4.75
40.21
$4.94
$7.57
36.10
34.69
87.66
FolAvgsPOOG0
75309
73.71
35.32
91.21
S3.36
$7.26
17.78
94.53
89.90
55.25
12.82
97.77
FaiRRD
FNITHE
92.74
18.76
25.51
5.2 Experimental Results
Performance Comparison on non-IID data. We categorized our baseline models into groups based on whether they consider personalization. The results for Cifar Cifar100, and ImnylmageNet are shown Tab. [T] Tab. 2 and Tab. ii Appendix E. respectively. For the first group without considering personalization, the existing centralized OOD methods, i.e., LogitNorm (FedLN), ATOL (FedATOL) and T3A (FedT3A), are not directly competitive among different non-IID see- narios. Though FedATOL achieves satisfying results for both generalization and detection tasks on Cifar10 5, it fails neither smaller a and dataset containing more classes (i.e., C00ar100). Mcanwhile, the vanilla FedAvg degrades its performance in OOD generalization for both Cifar10 and Cifar 100 data, and shows no potential of detecting OUT data samples. FedIIR pays more effort to maintain the inter-client generalization via restricting model consistency, making it less effective in non-lID settings. For the second group of personalized FL methods, personalization is quite necessary for both IN data generalization and IN-C generalization, which is similarly illustrated in FedTHE [26] and FedICON [69]. In general. personalized methods are worse in FL detection than non-personalized methods, indicating that there is conflict between detecting OUT data and enhancing prediction in non-IID setting. More surprisingly, we also discover that personalized adaption methods also detect outliers better compared with vanilla FedRoD model. FOSTER has better detection in more heterogenous data distribution, i.e., a = 0.1, compared with its results in 5, but overall performance is supposed enhance in the future. FOOGD flexible FL
Table 2: Main results of federated OOD detection and generalization on Ci0ar100. We report the ACC of brightness IN-C ACC, the FPR95 and AUROC of LSUN-C as OUT performance.
Non IID
0.1
0.5
Metbod
ACC
ACC CY
[FRS51
AUROCY
ACC
ACC-INCC
TPRS5]
AUROCT
ACC
ACCTNN-C
FPRO5.
AUROCT
FalAng
$1.67
47.54
78.35
57.16
58.78
54.53
72.84
70.86
61
56.72
72.68
70.50
FaiLN
53.48
48.15
66:94
74.63
59.30
53.86
68.31
73.41
61.00
56.33
69.18
75.67
FalATOL
43.65
41.06
65.26
81.64
60.52
56.53
70.10
79.27
64.16
63.61
$0.27
60.51
PofTAA
51.67
51.50
18.36
57:23
<9.07
72.56
70.85
55.51
72.77
$1.33
47.58
77/62
61.20
64.66
FedRaD
72.95

---

Table 3: Cifar 10 ablation study on varying a modeled by FedAvg.
Nen DI
0.1.1
0.5
a- 5.0
Motieel
ACC-IN
ACC-IN-CC
FPROS]
AUIROC"
ACC-IN
ACC-IN-C1
FRR95]
ALEROCT
ACC-IN 7
ACC-IN-CT
FFRIS)
AUROCT
AA backbone
68.03
65.43
$1.27
$5.89
36.59
53,72
20.40
95.82
86.50
85.05
15.44
96.96
w/o SHDD
74.70
73.25
41.36
83.85
38.01
57.17
19.90
95.86
83.52
87.79
15.05
97.06
w/o BAG
23.15
70.79
37.59
91.47
$7.32
53.33
18.83
96.13
87.85
85.20
12:75
92.65
Fod0Ag-F0000
75.09
73.71
35.32
91.21
$8.36
87.26
17.78
96.33
$8.90
88.25
12.02
97.77
Table 4: Cifar100 ablation study on varying a modeled by FedAvg.
Motioa
FPRO5)
AUROOC
ACC-IN-C1
ACC-IN-CY
AUROCT
bockhone
31.67
47.54
56.11
58.28
17.26
61.40
55.72
68.04
SNFDD
53.45
43.49
61.52 37:23
62.18
64.03
62.19
53.14
60.39
60.53
57.16
36,40
62.19
55.70
64.96
framework and achieves significant results for wild data tasks, ie., IN generalization, IN-C generalization, and OUT detection, Cifar10, Cifar100, and TinylmageNet. Specifically, FOOGD achieves comparable performance in enhancing both FedAvg and FedRoD, free the FL framework constraints. FOOGD enjoys the benefits of SM achieving distinguishable detection im- provement by Eq. (9) Besides, the regularization of score model with global distribution makes SAG regularize main task feature extractor better than contrastive-based methods, e.g., FedICON, and ebalanced-mettoods e.g., FedTHE and original FedRoD.
We devise the variants of FOOGD i.e., fix backbone, w/o SM and w/o SAG, to study the effectiveness of our three main ideas: (1) obtaining reliable global distribution guidance, (2) estimating score model by SM and (3) enhancing FL. method generalization by SAG, respectively. From Tab. and Tab. 4 simply modeling the knowledge of global distribution. When we remove SM the estimation of data probability is severely impacted, bringing no detection capability. While the generalization performance decreases once we remove SAG. Moreover, compared with fix backbone, both w/o SM'D and w/o SAG have better generalization and detection results, indicating the necessity of regularizing feature extractor with global distribution.
Ablation Studies.
model enhances detection slightly, since it brings
Visualization. To explore the wild data distribution of FL OOD methods, we visualize T-SNE of data representations in Fig. and the detection score distributions Fig. ( Cifarl0 a 5 for FedAvg+FODGD FedRoD+FOOGD and their runner-up methods, FedATOL and FedTHE, respectively. is evident that FOOGD represents IN-C data more tight with IN data, and constructs comparably clear decision boundary between IN data and OUT data. Besides, we also discover that FOOGD will push OUT data away from its IN and IN-C data, which validates the guidance from the global distribution. Additionally, in Fig. 5 FOOGD makes the modes among IN, IN-C, and OUT, more separable than existing methods. This also proves the effectiveness of FDOGD in detection task.
Extensive experiments on other IN-C and OUT data, In this part, we study the performance evalu- ation of FOOGD in additional IN-C and OUT datasets. In Tab. 5 we can find that FOOGD consistently enhances the detection capability for different OUT data, validating the effectiveness of estimating global distribution via SM Meanwhile, we compute the average results of different IN-C accuracy FL models trained on Cifar10 and Cifar100 in Fig. G The FOOGD in each group is short for FedAvg+ FOOGD and FedRoD+ FOOGD, respectively. We provide the details in Appendix Tab. and Tab 14 FOOGD consistently improve the generalization all unseen IN-C data, indicating the effectiveness of enhancing feature extractor via SAG.
Tab. 13
(a) FedATOÍ.
(b) FedAvg+FODGD
(c) FedTHE
(d) FedRDD+FOOGD
Figure 5: Detection score distribution of FL methods Cifar10 (a 5.0).

---

Table 5: Other detection results Cifar10 (a- = 0.1).
ACC-IN-C
8 8
OUT Dats
ISUN
SVHN
LSUNN-
Textnen
Milhoe
FPROS
AURIX
FFRUS.
AUROICT
EPROST
AUROK.
FFROX]
AURICT
Pedling
62:10
76.29
90.02
62.14
62.01
77.02
80.55
66.23
PeiLN
66.41
76.03
70.15
76.82
61.31
7H.34
93.90
71.99
PedATOL
61.81
80.05
85.39
92.17
64.01
79.89
66.35
76:77
PediIR
57.86
77.98
$3.68
04.04
55.44
78.09
91.72
62:32
Fedvyg
Fod fid-PPR
Faam +OOOD
FedRaD FIRTE pormi Feliiit sRoo0n
PedAvg-FooGD
37:55
91.32
44.59
87.63
14:36
90.16
28.00
91.75
PedRoD
43.40
$2.83
40.72
83.55
41.30
$2.92
53.24
81.52
POSTER
48.73
76.29
39.55
83.07
48.09
76.24
54.25
77.62
PedITE
43.72
83.50
39.22
85.95
42.95
$3.46
53.58
82:
Figure 6: The average results for Cifar Cifar100-C 100-C generalization.
PeJICON
49:99
82.95
34.94
X5.56
49.05
83. $3.50
51,57
80.96
36.17
88,89
17.01
94.56
41.46
92.90
19.46
93.39
Client Generalization on PACS Dataset. To validate the effectiveness of FOOGD in domain general- ization tasks, i.e., each client contains one domain data and we train domain generalization model by leave-one-out, following FedlIR [17]. To compare fairly, we pretrain all models from scratch and utilize adaption methods as stated in their main paper. In terms of Tab. 6 FOOGD obtains performance improvements for FedAvg and FedRoD. Compared with existing adaption methods, FOOGD achieves outstanding results even in the toughest task, i.e., leaving Sketch domain out. This also concludes that FOOGD capable of inter-client generalization, via utilizing global distribution knowledge.
Hy perparameeer sensitivity studies and other empirical studies. Due to the space limitation, we leave the other relevant experiments in Ap- pendix E. Summarily, we study four additional evaluations: (1) In Tab. [0]
Table OOD generalization task for PACS.
Method \Domain
Art Painting
Cartoon
Photo
Sketch
Average
FedAvg
97.21
62.58
91.00
35.28
71.52
FedRoD
93.45
88.85
89.34
29.95
75.39
FedT3A
97.13
75.71
93.21
37.40
75.86
We compute different detection met- rics, i.e., MSP, energy score, and ASH,
FediIR
86.86
80.29
88,98
31.38
71.88
FedTHE
96.17
90.72
93.57
29.14
77.40
FedICON
53.36
52.19
51.58
and validate that Eq. (9) is consis- tently powerful in detection. (2) We
FedAvg+ FOOGD
97.46
89.32
91.48
41.40
79.92
FodRoDDFOOGD
97.85
92.31
93.01
50.95
83.53
vary the coefficient of SM'D Am = (0.1,,02, 0.5.0, 1} in Fig. (a)
Fig.
11(b)
and vary the co-
efficient of SAG A {0, 0.01, 0.05, 0.2, 0.8) in Fig. [I(e)
obtain the best modeling in
FOOGD. (3) We vary the number of participating clients in Fig. and found FOOGD can have better
results among different participating clients.
Conclusion and Future Work
In this work, we consider enhancing both detection and generalization capability of FL methods among non-lID settings. To realize it, we try to model global distribution by collaborating clients, and propose FOOGD, which consists of SM'D for estimating score model for detection, and SAG to enhance the invariant representation for generalization. We conduct extensive experiments to validate the effectiveness of FODGD: (1) reliably and flexibly estimating non-normalized decentralized distribution, (2) detecting semantic shift data via the norm of score values, and (3) generalizing adaption of covariate shift data by regularizing feature extractor invariant distribution discrepancy.
In the future, we plan to integrate privacy enhancement techniques, such as differential privacy, into FOOGD. While the score model in FOOGD captures the score function of the data probability in the latent space, which is extremely difficult to be used for reconstructing the original data by attacking. The primary risk exposure for each client arises from the exchange of model parameters, i.e., the feature extractor and score model. Hence, FOOGD has a comparable level of privacy exposure as existing FL methods dealing with non-IID and OOD shifts, acquiring to be addressed comprehensively.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Leading Expert of "Ten Thousands Talent Program" of Zhejiang Province, China (No. 2021R52001), the National Natural Science Foundation of China (No. 621 62172362), and the Fundamental Research Funds for the Central Universities (No. 226202400241).
10

---

References
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In Proceedings of the 37th International Conference on Machine Learning.
volume 119, pages 145-155. PMLR, 2020.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization, arXiv preprint arXiv: 0007000233,... 2019.
[3] Haoyue Bai, Gregory Canal, Xuefeng Du, Jeongyeol Kwon, Robert D Nowak, and Yixuan Li. generalization
Feed two birds with one scone: Exploiting wild data for both and detection. In International Conference on Machine Learning, pages 1454 1471. PMLR, 2023.
[4] Ayush Bharti, Masha Naslidnyk, Oscar Key, Samuel Kaski, and Francois-Xavier Briol. Optimally-weightee estimators of the maximum mean discrepancy for likelihood- inference. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages
2289-2312. PMLR, 2023.
[5] Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for dustof-distriuution detection. In Proceedings of the s International Conference on Machine Learning, volume 235, pages 1629-5659, PMLR,
2024.
[6] Tri Cao, Jiawen Zhu, and Guansong Pang. Anomaly detection under distribution shift. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October
1-6, 2023, pages 6488-6500, IEEE, 2023.
[7] Chen-Hao Chao. Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu-Lin Chang, Chia-Ping Chen, and Chun-Y Yi Lee. Denoising likelihood score matching for con- ditional score-based data generation. In International Conference on Learning Representations,
2021.
[8] Hong-You Chen and Wii-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
[9] Shengzhuang Chen, Long-Kal Kai Huang. Jonathan Richard Schwarz, Yilun Du, and Ying Wei. Secure out-of-distributinn task generalization with energy-based models. In Advances in Neural Information Processing Systems, volume 36, pages 67007-67020, 2023.
[10] Yongqiang Chen, Wei Huang, Kaiwen Zhou, Yatao Bian, Bo Han, and James Cheng. Under- standing and improving feature learning for out-of-distribution generalization. In Advances in Neural Information Processing Systems, volume 36, pages 68221-68276 2023.
[1]] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[12] Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2022.
[13] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't t know by virtual outlier synthesis. In Proceedings of the International Conference on Learning
Representations, 2022.
[14] Haozhe Feng, Zhaoyang You, Minghao Chen, Tianye Zhang, Minfeng Zhu, Fei Wu, Chao Wu, and Wei Chen. Kd3a: Unsupervised multi-source decentralized domain adaptation via knowledge distillation. In Proceedings of the 38th international conference on machine learning, volume 139, pages 3274-3283, 2021.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hama Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversane training of neural networks. Journal of machine learning research, 17(59):1-35, 2016
[16] Soumya Suvra Ghosal and Yixuan Li. Distributionally robust optimization with probabilistic group. In -eveeth AAAI Conference on Artificial Intelligence, pages 11809-11817 2023. [17] Yaming Guo, Kai Guo, Xiaofeng Cao, Tieru Wu, and Yi Chang. Out-of-distribution general- ization of federated learning via implicit invariant relationships. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 1 19905-11933. PMLR, 2023.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[19] Dan Hendrycks and Thomas G Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. arXiv preprint 1781vv1807.01697, 2018.
11

---

[20]
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of -ittri--ittiitttion examples in neural networks. In International Conference on Learning Representations, 2016.
[21] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2018.
[22] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
[23] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identien -deeniettcall data distribution for federated visual classification. arXiv preprint arXiv: 190.:03999000 2019.
[24] Zeyi Huang, Haohan Wang. Eric P Xing. and Dong Huang. Self-challenging improves cross- domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020. Proceedings. Part II 16, pages 124-140. Springer, 2020.
[25] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[26] Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In The
Eleventh International Conference on Learning Representations, 2022.
[27] Wenyu Jiang, Hao Cheng, Mingcai Chen, Chongjun Wang, and Hongxin Wei. Dos: Diverse outlier sampling for out-of-diitribution detection. arXiv preprint arXiv:2306.0P 02031, 2023.
[28] Xue Jiang, Feng Liu, Zhen Fang. Hong Chen, Tongliang Liu, Feng Zheng, and Bo Han. Detecting out-of-distribution data through in-distribution class prior. In Proceedings of the 40th International Conference on Machine Learning, pages 15067-15088. PMLR, 2023.
[29] Ellango Jothimurugesan, Kevin Hsich, Jianyu Wang. Gauri Joshi, and Phillip B Gibbons. Federated learning under distributed concept drift. In International Conference on Artificial Intelligence and Statistics, pages 5834-5853 PMLR, 2023.
[30] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132-5143, PMLR, 2020.
[31] Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In Proceedings of the 39th International Conference on Machine Learning, pages 10848-10865. PMLR, 2022.
[32] Frederic Koehler and Thuy-Duong Vuong. Sampling multimodal distributions with the vanilla score: Benefits of data-based initialization. arXiv preprint arXiv:2310.01700 2023.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
[34] David Krueger, Ethan Caballero, Joem-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapo- lation (rex). In Proceedings of the 38th International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
[35] Mare Lafon, Clément Rambour, and Nicolas Thome. Heat: Hybrid energy based model in the feature space for out-of-distribution detection. In NeurlPS ML Safety Workshop. 2022.
[36] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[37] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting
samples and adversarial attacks. Advances in neural information processing systems, 31, 2018
[38] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conferrncee on computer vision, pages 5542-5550, 2017.
[39] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning
and systems, 2:429-450, 2020.
[40] Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang. and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019,
[41] Yingzhen Li and Richard E Turner. Gradient estimators for implicit models. In International Conference on Learning Representations, 2018
[42] Xinting Liao, Chaochao Chen, Weiming Liu, Pengyang Zhou, Huabin Zhu, Shuheng Shen, Weiqiang Wang, Mengling Hu, Yanchao Tan, and Xiaolin Zheng. Joint local relational augmen- tation and global nash equilibrium for federated learning with non-iid data. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1536-1545, 2023.
[43] Xinting Liao, Weiming Liu, Chaochao Chen, Pengyang Zhou, Fengyuan Yu, Huabin Zhu, Binhui Yao, Tao Wang, Xiaolin Zheng, and Yanchao Tan. Rethinking the representation in
12

---

federated unsupervised learning with non-iid data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22841-22850, 2024.
[44] Quande Liu, Cheng Chen. Jing Qin, Qi Dou, and Pheng-Am Ann Heng. Feddg: Federated domain generalization medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recogni- fion, pages 1013-1023, 2021.
[45] Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of- fit tests. In Proceedings The 33rd International Conference on Machine Learning, pages 276-284, PMLR, 2016.
[46] Qiang Liu and Dilin Wang- Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
[47] Weiming Liu, Jiajie Su, Chaochao Chen, and Xiaolin Zheng. Leveraging distribution alignment via stein path for cross-domain cold-start recommendation. Advances in Neural Information Processing Systems,
11119223-1224,, 2021.
[48] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based
detection. Advances in neural information processing systems, 11-21464-21477 2020.
[49] Weiming Liu, Xiaolin Zheng, Jiajie Su, Longfei Zheng, Chaochao Chen, and Mengling Hu. Contrastive proxy kernel stein path alignment for cross-domain cold-start recommendation.
IEEE Transactions on Knowledge and Data Engineering, 35(11):: (3(11):112166 1230, 2023.
[50] Yue Liu, Xihong Yang. Sihang Zhou, Xinwang Liu, Siwei Wang, Ke Liang. Wenxuan Tu, and Liang Li. Simple contrastive graph clustering. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[51] Yue Liu, Xihong Yang, Sihang Zhou, Xinwang Liu, Zhen Wang, Ke Liang, Wenxuan Tu, Liang Li, Jingcan Duan, and Cancan Chen. Hard sample aware network for contrastive deep graph clustering. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 1922, 2023.
[52] Zhengquan Luo, Yunlong Wang, Zilei Wang. Zhenan Sun, and Tieniu Tan. Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring. In International Conference on Machine Learning, volume 162, pages 14527-14541. PMLR, 2022.
[53] Zheqi Lv, Wenqiao Zhang, Zhengyu Chen, Shengyu Zhang, and Kun Kuang. Intelligent model update strategy for sequential recommendation. In Proceedings of the ACM Web Conference 2024, pages 3117-3128, 2024.
[54] Zheqi Lv, Wenqiao Zhang, Shengyu Zhang, Kun Kuang, Feng Wang, Yongwei Wang, Zhengyu Chen, Tao Shen, Hongxia Yang, Beng Chin Ooi, et al. Duet: A tuning-fres device-cloud collaborative parameters generation framework for efficient device model generalization. In Proceedings of the ACM Web Conference 2023, pages 3077-3085, 2023.
[55] Ahsan Mahmood, Junier Oliva, and Martin Andreas Styner. Multiscale score matching for -istrribiiiiininnnnnnoooo detection. In International Conference on Learning Representations, 2020. [56] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication efficiiet learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[57] Dheeraj Mekala, Adithya Samavedhi, Chengyu Dong, and Jingbo Shang. Selfood: Self- supervised
detection via learning to rank. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10721-10734, 2023.
[58) Chirag Modi, Robert M Gower, Charles Margossian, Yuling Yao, David Blei, and Lawrence K Saul. Variational inference with gaussian score matching. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023.
[59] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[60] A Tuan Nguyen, Philip Torr, and Ser Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. Advances in Neural Information Processing 2022.
Systems, 35:38831-388888888
[61] Kunjal Panchal, Sunav Choudhary, Subrata Mitra, Koyel Mukherjee, Somdeb Sarkhel, Saayan Mitra, and Hui Guan. Flash: Concept drift adaptation in federated learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 26931-26961 PMLR, 23-29 Jul 2023.
[62] Lianyong Qi, Yuwen Liu, Weiming Liu, Shichao Pei, Xiaolong Xu, Xuyun Zhang, Yingjie Wang, and Wanchun Dou. Counterfactual user sequence synthesis augmented with continuous time dynamic preference modeling for sequential poi recommendation. In Proceedings of the
13

---

Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), 2024.
[63] Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In Proceedings of the 39th International Conference on Machine Learning, pages 18250-18280. PMLR. 2022.
[64] Yasar Abbas Ur Rehman, Yan Gao, Pedro Porto Buarque De Gusmão. Mina Alibeigi, Jiajun Shen, and Nicholas D Lane. L-dawa: Layer-wise divergence aware weight aggregation in federated self-supervised visual representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16464-16473, 2023.
[65] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2019.
[66] Vikash Sehwag. Mung Chiang, and Prateck Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2020.
[67] Yang Song and Stefano Ermon. Generative modeling by estimating gradicnts of the data distribution. Advances in neural information processing systems, 32, 2019.
[68] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. out-of-distribution detection with deep nearest neighbors. In Proceedings of the 39th International Conference on Machine Learning, pages 20827-20840. PMLR, 2022.
[69] Yue Tan, Chen Chen, Weiming Zhuang. Xin Dong, Lingjuan Lyu, and Guodong Long. Is heterogeneity notorious? taming heterogeneity to handle test-time shift in federated learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[70] Xueyang Tang, Song Guo, and Jie Zhang. Exploiting personalized invariance for better out-of- distribution generalization in federated learning. arXiv preprint arXiv:2211. 1.12211..1243,... 2022.
[71] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 34:1 1287-11302 2021. [72] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-166
2011.
[73] Fan Wang. Chaochao Chen, Weiming Liu, Tianhao Fan, Xinting Liao, Yanchao Tan, Lianyong
Qi, and Xiaolin Zheng. Ce-refr: Robust counterfactual regression for consensus-enabled treatment effect estimation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, pages 3013-3023, 2024.
[74] Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition. pages 4921-4930, 2022.
[75] Qizhou Wang. Zhen Fang, Yonggang Zhang. Feng Liu, Yixuan Li, and Bo Han. Learning to augment distributions for out-of-distribution detection. In Thirth-seventh Conference on Neural Information Processing Systems. 2023.
[76] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In Proceedings of the 39th International Conference on Machine Learning, pages 2363 1-26644. PMLR, 2022.
[77] Yue Wu, Shuaicheng Zhang, Wenchao Yu, Yanchi Liu, Quanquan Gu, Dawei Zhou, Haifeng Chen, and Wei Cheng. Personalized federated learning under mixture of distributions. In Proceedings of the 40th International Conference on Machine Learning, pages 37860-37879.
PMLR, 2023.
[78] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv
preprint arXiv: 1504. 06455, 2015.
[79] Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian, A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 14383-14399 2021.
[80] Jingkang Yang, Haoqi Wang. Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent -fiit-iitttiittiinn detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8301-8309, 2021.
[81] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.1 03365, 2015.
[82] Runpeng Yu, Songhua Liu, Xingyi Yang, and Xinchao Wang. Distribution shift inversion for out-of-distribution prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 3592-3602, 2023.
[83] Shuyang Yu, Junyuan Hong. Haotao Wang, Zhangyang Wang, and Jiayu Zhou. Turning the curse of heterogeneity in federated learning into blessing for outto--distribution detection. In 2023 International Conference on Learning Representations, 2023.
14

---

[84] Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do we mean by generalization in federated learning? In International Conference on Learning Representations.
2021.
[85] Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks. arXiv preprint
1oXivv6605.07 2016.
[86] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman. Yiran Chen, and Hai Li. Mix- ture outlier exposure: Towards out-of-diitrriuttin detection in fine-grained environments, In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5531-5540, 2023.
[87] Jianyu Zhang. David Lopez-Paz, and Léon Bottou. Rich feature construction for the ciaimizattoo-ggneaalizattooo dilemma. In International Conference Machine Learning, pages 26397-26411. PMLR, 2022.
[88] Haotian Zheng. Qizhou Wang, Zhen Fang, Xiaobo Xia, Feng Liu, Tongliang Liu, and Bo Han. Out-of-distribution detection learning with unreliable out-of-distribution sources. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[89] Zhengyi Zhong, Weidong Bao, Ji Wang. Xiaomin Zhu, and Xiongtao Zhang. Flee: A hierarchical federated learning framework for distributed deep neural network over cloud, edge, and end device. 13(5), Oct. 2022.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction section include the main claims made in the
paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
The paper focuses on federated modeling, and we will focus on enhancing sevaccypreserving ability of FOOGD in our future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have provided the full set of assumptions and a complete (and correct)
proof. Please refer to Section in in the main paper and Section B in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided detailed implementation details in Section 5.1 of the main paper and Section A and D of the Appendix.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided publicly available dataset and experiment details information in Section 5. of the main paper and Section A and D of the Appendix. We commit to releasing the source code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. ctipss.cc/ublic//uui.eu www.iiiciguiisseieeeeeeeeeeeeeee
ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
18

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed implementation details and training settings in Section55.Too the main paper and Section and D of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experimental results computed three times and report the average result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: My research group supports in computer resources.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper fully adheres to the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discussed both potential positive societal impacts and negative societal impacts in Section of the Appendix.
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
20

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The data and code used in this paper have obtained legal permissions.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
21

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22

---

In the supplemental materials, we provide all algorithms in Appendix A the theoretical analysis in Appendix B] additional related work on federated learning with non-IID data in Appendix C experi- mental implementation details in Appendix D the additional experimental details in Appendix E
A Algorithms
The overall algorithm of FOOGD is in Algo. In line 1:10, the server collaborates with clients to optimize the feature extractor model for representation and the score model for density estimation. The clients execute training local models separately in line 11:19. In each client, SM estimates data density based on the latent representation of feature extractor, and SAG computes the kernelized stein model to regularize the optimization of feature extractor. We update score
discrepancy based model with S in Algo. 2 and the training procedure of feature extractor is detailed in Algo.
Algorithm Training procedure of FOOGD
Input: Batch size B, communication rounds T, number of clients K, local steps E, dataset D
[RE[R]DK
Output: feature extractor and score model parameters, i.e., 0z and 0
Server executes():
2: Initialize (0), ) with random distribution
3: for t= 0, 1..,, T do
4: for k = 2, ..., K in parallel do
5: Send {0 03} to client k
6: {0)A A < Client executes(k;, (6), 0
7: end for
8: Update parameters of (0), 07) by Eq.
9: end for
10: return (²'
11: Client executes(k; (0%,0%)
12: Assign global model to local model (0), }
107, 0,)
13: for each local epoch 1, 2, .... F. do
14: for batch of samples (X1.B. YI.B) E D3 do
Execute f D X1.B, Y1.B) Algorithm
Execute 0 SAG X I.B, YI.n) in Algorithm
17: end for
18: end for
19: return 0 to server
Algorithm 2 Algorithm of SMOD
Input: Batch size B, batch of samples (X1.8, Y1.B) Dr. fixed feature extractor 0,, and initialized score model 0,
Output: score model parameters, i.c., 0,
1: Feature Extraction Z + (X.i..))
2: Sample B random data points 2 with 2! ~ ~0.,
3: Take Langevin dynamic sampling started at T to obtain generated samples Zyen by Eq. 6
4: Perturb noise data features to obtain Ż (Z
5: Compute denoising score matching by Eq. 4
6: Regularize maximum mean discrepancy between generated Zgen and Z by Eq. 77)
7: Optimize score model 0, with objective ÆOUT by Eq. 8
8: return 0,
23

---

Algorithm 3 Algorithm of SAG
Input: Batch size B3 batch of samples (X1:B, XI.B) Dx, fixed score model O,, and initialized feature extractor f
Output: feature extractor parameters, i.e., f
1:
Augment samples X 1.B i()
2: Feature extraction 11:B - I. and Z t (x( (X::)
3: Compute kernelized Stein divergence by Eq. (T))
4: Compute cross entropy loss between prediction Y pred Classificr Z) and ground truth y
5: Optimize feature extractor 6 with objective eIN IN by Eq. 12
6: return y
B Theoretical Analysis
B.1 Error Bound of S
Lemma B.1 (Error Bound of Decentralized Score Matching). The error bound of global score model aggregated from local scores models is
TTu v o² Px [|] logpp(z))
V log)o(=)) V= 1o
(14)
Proof. For the global score model aggregated from local score models that estimate IN data probabil- ity densities, holds:
K Wkk 80, (z)
V logpo(z) so
k=1
K
V log Po. (2)
(15)
k=
wa log pe, (z).
Then we formulate the score matching for global distribution as
V (z
V log (x))l
K
wk V log (z) V log logP)(>))1
K
wk V- Po. (2)
we V 1z)P1(z)11
(16)
wk V log (x) V [=)]
1=1
K
wk V2 log (z) V log pD (2)
1=1
where Σ k= Wh 1, and the last term is held by Jensen inequation.

---

In term of Vincent (72). the DSM for each local model s0, (z) is bounded as follows,
(Ds
Bp(2,5) |1se, (z) V logp (2 =)]
B)(x,z)
[2)11 2so, (z) _ V logp (2 2) + V logp (ž x)]l²
Bp(x))
||s00 (=)]l²
2)x()
100 (z)
V logp (ž
x)]
e' v
(17)
σ²
- (Os) + p(z.5)
80 (z)
v
E V
a2
v
(E
B ||| V log (1))
σ²
where o ~ NO I) and
1
{-11 2
2
Va logp (# z) = Vs
log
(v2T03) a exp
20²
σ²
0
(18)
Therefore, when 0 0" 0% we have JESM (0) JESM (0x) (0x)=
Ak E [K], the global score
matching finally satisfies:
[s(()) V logpp(z)]1
V log V [=g])
Wk V log (z) V= lo))
(19)
k=1
E log
which holds due to Σx= w/
K
Theorem B.2 (Error Bound of Decentralized Score Matching via SKDD), Assume the original MMD(Z, Zgen < for randomly initialized score model sø(z) in Eq. 7 the score model achieves optimum and MMD decreases. By Lemma B.1 we can obtain the final error bound of global so(.) as:
T r²
DCc
D
[100(2) - V2 log (og))
<
(20)
B
where C' is the upper bound of the MMD, B is batch size, and [D] is the data amount.
B.2 The overall induction of Kernelized Stein Diserepancy in SAG
Assume feature distributions p(z) and ((2) are two bounded distributions satisfying lim |=|=+o0 p(z))(z) z) = 0 and lim q (2))(22) = 0. And we denote the gradient of log density
in z as V log ((2) = vz)($)
((*)
Lemma B.3 (Stein identity). If the o(-) in Stein operator A,,(2) = Vz log (2) + VE0(2)
introduced in Eq. (TO) is Stein class, then we have fundamental property called Stein identity as below:
Eeng [0(2) V log (2) Vz0(2) 0.
(21)
25

---

Proof.
Egnu [0(2)
(o da
(2))(() + d2
(3)V (
(22)
()())
(3)
O.
Definition B.4 (Stein Discrepancy). Stein identity induces Stein discrepancy for two distributions p(z) and (3):
SD(p(z). sup dEF
Ez [Ap)(2)] Eginn (Ap)(E')),
(23)
where db.) is the stein class function satisfying boundary conditions, and F is the function space. Lemma B.5. If F is a unit ball in reproducing kernel Hilbert space (RKHS) with positive definite kernel function k(.,-) E F, we obtain the Kernelized Stein Discrepancy for p(z) and ((2) as below:
KSD(P(z).g()) Eg (s) so(3") 2') s0( V k(2 2') + so V2k(2,
+ (race(V2 V2
(24)
Proof. Firstly, considering the expectation of g(z) on the Stein operator with score of p(z), we can expand via introducing Stein identity:
Egng [Apd(2)] Bggig [Ap)(2)] Eggg [An)(2)]
= Egnng (Ap - -2)
(25)
EEgg (b(2) (V log p(2) V log (()))
Then, with the property of RKHS, we have k(2,2') = (0(3), 0 (E'')) , 5p(3) and (3) are short for V p()) and V log (()), respectively, the Stein discrepancy can be rewritten as
S(p(z), (())
= EEing 440(2)) E Eginng [Apd(E']]
Eg
(Vz logop((> ()) V (z)) 1(2,2') logp(2') V log ((2))
Eg
(s)(3) - ,, (3)) k(2 (s) - -1,8))))))))))) s,(3))
Eg
(sp(2) - s, (8)) (k(2, (((2)))),,('') + Ve
- 1(2, ((2,)')s,EE)) v2:
= Eg,g'ng
(sp(3) - (2)) T (2(2,2')s,(3) + V k(,
(26)
where the last second equation is also held by Stein identity.
Next, we define the
=(, + (22
(27)
introducing another Stein identity holds, i.e.,
Eg,zing [2) ((2, + V 22 Eg,3' (A,u(2, (2,22)) 0.
(28)
26

---

Finally, taking v(2,2) back to Eq. (26) and substitute s0 Vglog/00 po(),, we can obtain the final KSD without the requirement computing score values of g(2).
KSD(p(=).q(3)
-Eg-y [Apo(2)] Egging [Apo(E']
Eg, (Vglogp(2)
V3 V=logg(2)) k(2, (V logp(E) V log
Rg [s0(3) T (0()))((22))) (3) V k=, =0 +
81))
(29)
B.3 Bound of Client Model Divergence
In this part, we first introduce mild and general assumptions [40], and induct the model updating di- vergence bound for each client. Because FOOGD aggregates client models similar its original model, i.e. FedAvg and FedRoD, its generalization bound is unchanged compared with the generalization bound proposed in 44). Please kindly refer the original paper.
Assumption B.6. Let (O) be the expected model objective for client k. and assume Fi,... FK are all L-smooth, i.e., for all Og. Fi (Ox) < Fx,(2)) + (0x- Ox)) (Ox( + / 0 0x112.
Assumption B.7. Let F,, Fv are all p-strongly convex: for all Oz. (O.) 2 Fg (O.) ) (0x- 0x) (v) 0 0x
Assumption B.8. Let & be sampled from the k-th client's 's local data uniformly at random. The variance of stochastic gradients in each client bounded: E V (0 { VF, (0I)l S o Assumption B.9. The expected squared norm of stochastic gradients is uniformly bounded, i.e.,
E F (0 2 < 2 for all 1.... K and t .,....77 1
Next, we introduce the lemma related to the bound of client model divergence.
Lemma B.10 (Bound of Client Model Divergence). With assumption B.9 dtt non-increasing and < 21/++ (learning rate of 1-th round and E-th epoch) for all 2 0, there exists to S 1, such that
t to S E - and 0 0 for all k E [K]. It follows that
K
E
Σ
wk 0
< (E EE
(30)
Proof. Let E be the maximal local epoch. For any round t >> 0, communication rounds from to to t
exist t to < 1. and the global model 0 and each local model 0 are same round ty-
K
E
Wk 00
01
wy (1) 000) (0' - 0
(31a)
Wk| 10 10
(316)
E
Wkk
m: Fv +
(31c)
to
<
wh (t - to)
Σ
nho IV ( E
(31d)
7=to
< (n F2- 122
(31c)
where the Eq. (31b) holds since E(O 0 0³ 200 and E E|X|] and Eq. (31d)
derives from Jensen inequality.
0
27

---

C Related Work
C.1 Federated Learning with Non-IID Data
Federated Learning (FL) with non-lII data presents significant challenges in balancing global and local model performance. One prominent method, FedAvg [56], uses simple averaging but struggles with client heterogeneity, often degrading individual client models. To improve global performance, methods like FedProx
39 introduce regularization to keep local updates close to the global model, and SCAFFOLD 30 uses control variates to reduce variance from client heterogeneity. For local personalization, meta-learning and transfer learning techniques such as DFL 52 focus on enhancing individual client models to adapt better to local data. Lastly, methods like FedRoD [8] attempt to achieve joint global and local performance by decomposing models, aiming to balance both objectives, though extreme non-IID settings still pose challenges. However, these FL. methods modeling non-lll) data take no actions to OOD data, causing them less advantageous.
D Experimental Implementation Details
D.1 Experimental Setups
Datasets Following SCONE B) we choose clear TinyImageNet (36], Cifar10 and Cifar100 [33] and as the IN data. For OOD generalization, we select the corresponding synthetic covariate- shift dataset as IN-C data, by leveraging 15 common corruptions for all datasets, and 4 additional corruptions for Cifarl0-C and CO0CC [19]. To evaluate FOOGD unseen client data, also perform experiments on PACS (38 for leave-one-out domain generalization. For OOD detection, we evaluate five OUT image datasets: SVHN [59], Texture (11, iSUN (78), LSUN-C and LSUN-R [8].
Heterogeneous client data The original train and test datasets are split to all clients to simulate the practical non-IID scenario (23). Specifically, we sample proportion of instances of class to client k using a Dirichlet distribution, i.e., Pj.k ~ Dir(a), where a denotes the non-IID degree of each class among the clients. A smaller a indicates more heterogeneous data distribution. For the PACS dataset, 3 clients are set where each client holds data from one distinct domain, and the remaining unseen domain data is used for testing.
OOD evaluation setups We construct three types of test sets to assess the model's classification, domain generalization, and nat-of-distribution detection ability. Test set from IN dataset is used to evaluate how well the model adapts to local training distribution, i.e., model's classification ability. To simulate the non-IID distribution in real-world scenarios, we partition the IN-C dataset with the same heterogeneous distribution as the IN dataset. This setup evaluates the model's generalization ability on the IN-C dataset to determine whether existing FL methods can keep the data-label relationship in the presence of covariate shift in data features. All covariate-shift types in the IN-C dataset are test individually. For testing OOD detection ability, all clients use the same OOD test set for fair evaluation. After collecting performance data from all clients, we calculate a weighted average of the performance based on the volume of data each client holds.
D.2 Implemetnation Details
We choose WideResNet [85] as our main task model for Cifar datasets, and ResNet18 [18] for TinyImageNet and PACS, and optimize each model 5 local epochs per communication round until converging with SGD optimizer. We conduct all methods at their best and report the average results of three repetitions with different random seeds. We consider client number K 10, participating ratio of 1.0 for performance comparison, and the hyperparameters Am = 0.5. A = 0.05.
Below the detailed settings and hyperparameters for all federated baseline models.
1. FedAvg [56] is the classic federated learning method in which clients perform multiple epochs of SGD on their local data. The learning rate set to 0.1, with momentum of 0.9 and weight decay of Se-4.
FedITR [77 tries to implicitly learn invariant relationships through inter-client gradient alignment.
We set the ema parameter 0.95 and penalty term 1c-3.
28

---

Table 7: Main results of federated OOD detection and generalization on TinyImageNet.
Method
ACC-IN
ACC-IN-CI
FPR95]
AUROCT
FedAvg
29.51
15.18
69.22
80.17
FedLN
38.23
15.64
61.40
82.31
FedATOL
94.91
FedTAA
29.46
00.50
69.14
80.08
FodlIR
78.84
FedA
47.87
31.16
22.17
98.24
FedRoD
57.78
30.51
68.55
73.82
FOSTER
56.91
28.
67.17
73.22
FerTHH
58.23
30.24
63.63
76.83
TedICON
FFdRo
63.27
37.26
47.26
89.31
FedRoD is the personalized federated learning method that adopts two classifiers to achieve both generic and personalized performance.
4. FOSTER 883 learns a class-conditional generator to synthesize virtual external-class OOD samples to enhance the detection ability. The weight for the outlier exposure term set to 0.1.
FedTHE [26] also contains personalized classifier. This method adopts an test time adaptation strategy that interpolates the personalized head and global classifier to enforce feature space alignment. We set = 1.1 and B 0.3 as suggested in the original paper.
FedICON [69 performs different contrastive learning during training and test phrase handle test-time shift problem. Each client finetunes their classifier with learning rate 0.01.
also compare with centralized OOD generalization and detection methods, adapting them to FedAvg-like approach for the federated learning scenario.
1. LogitNorm 76 applies straightforward modification to the cross-entropy loss, imposing constant norm on the logits to improve detection capabilities. т is tuned to set as 0.04.
ATOL generates OOD data to devise an auxillary OOD detection task to facilitate real OOD detection. We set the dimension of the latent to be 100, the mean and variance of the Gaussian distribution generating OOD data be 5.0 and 0.1.
T3A 25 adjusts trained linear classifier using a pseudo-prototype. The filter number is set be 100 for experiments Cifar10, Cifar100 and TinylmageNet, 50 for experiments on PACS.
E Extensive Experiment Results
To summary, we study four additional evaluations: (1) To compare the performance in domain generalization, also provide the leave-one-out study on PACS (38) in Tab. II where FODGD also obtains better results. (2) In Tab. TO We compute different detection metrics, i.e.. MSP, energy score, and ASH, and validate that Eq. (9) is consistently powerful in detection. (3) We vary the
coefficient of SM Am = 0.21, 0.5, 0.8, in Fig. (1 Fig. [1(b) and vary the coefficient of
SAG A = {0, 0.01, 0.05, 0.1, 0.2, 0.5, 0.8) Fig, [I(e) obtain the best modeling FOOGD. (4)
We vary the number of participating clients in Fig. 10 (Oand found FOOGD can have better results among different participating clients.
E.1 Evaluation on TinyImageN ImageNet Dataset.
Additionally, we present the results of TinylmageNet in Tab. 7 and report our main results with variances Tab. and Tab. 16 vividly states that FOOGD can also generalize the task of more classes and more heterogeneous data distribution.
E.2 Ablation Studies
We devise the variants of FOOGD, i.e., fix backbone, w/o SMDD, and w/o SAG, to study effectiveness of our three main ideas: (1) obtaining reliable global distribution as guidance, (2) estimating score model by SM&D and (3) enhancing method generalization by SAG, respectively.s From Tab. ?? and its full version in Appendix E.2 Tab. 8 and Tab. simply modeling score model fails in both OOD

---

Table 8: Cifar 10 ablation study on varying a modeled by FedAvg.
Nen DI
0.1.1
0.5
a- 5.0
Motieel
ACC-IN
ACC-IN-CC
FPROS]
AUIROC"
ACC-IN 1
ACC-IN-C1
FRR95]
ALEROCT
ACC-IN 7
ACC-IN-CT
FFRIS)
AUROCT
AA backbone
68.03
65.43
$1.27
$5.89
36.59
53,72
20.40
95.82
86.50
85.05
15.44
96.96
w/o SHDD
74.70
73.25
41.36
83.88
38.01
57.17
19.90
95.86
83.52
87.79
15.05
97.06
w/o BAG
23.15
70.79
37.59
91.47
$7.32
53.33
18.83
96.13
87.85
85.20
12:75
91.65
FodAAg-F0000
75.09
73.71
35.32
91.21
$8.36
87.26
17.78
96.33
$8.90
88.25
12.02
97.77
Table 9: Cifar 100 ablation study on varying a modeled by FedAvg.
Nen II
0.1
--05
5.0
Methed
ACC-IN
ACC-IN-CC
[PROS)
AUIRCC
ACC-IN 1
ACC-EN-C1
FFR95]
AURROCT
ACC-IN 7
ACC-IN-CT
FPROS)
AUROCT
RN backbone
31.67
47.54
56.31
82.94
58.28
54,52
68.90
77.26
61.40
55.72
65.04
77.05
WAs SNPDD
$3.45
51.58
43.49
89.26
61.52
59.91
62. 18
84.72
64.03
62.19
64.18
K3.16
whs SAG
53.
48.35
37:23
91.13
60.39
53.72
60.53
85.17
62.12
57.16
50.58
$2.54
F
53.84
51.69
36.40
91.41
62.19
60.25
55.70
86.42
64.96
64.18
57.70
$4.03
generalization and detection tasks, since feature extractor not adjusted with the global distribution. When remove S, the estimation of data probability is severely impacted, bringing detection capability. On the contrary, the generalization performance decreases once we remove SAG. Moreover, compared with fix backbone, both w/o SM and w/o SAG have better generalization and detection results, indicating that it is necessary to introduce the global distribution.
E.3 Toy Example for validating S
To illustrate the effectiveness of D, we further visualize density estimation of 2-D toy example in Fig. 3 In detail, we model the red points sampled from target distribution, by tuning a series of coefficients, i.e., Am {0, 0.05, 0.1, 0.2, 0.4, 0.5, 0.8, 1} in Eq. (8). To start with, the blue generated data contract loosely close to the red target data. Then the distribution divergence gets smaller, making generated distribution overlap with targeted distribution. While, as the effect of MMD) increases, the distribution alignment dramatically worsens, even causing the blue generated data to collapse into the expectation of target distribution. As we can see, the mutual impacts between score matching and MMD estimation, S has more compact density estimation when Am 0.1, compared with blankly using score matching (Am = 0) or simply using MMD (Am = 1). In the brand new objective of density estimation, SMCD expand the searching range and depth of score modeling, making it possible to comprehensively model data density. Moreover, with the calibration of MMD estimation, original data representation and the generated latents based the are effectively matched, bringing more realistic and correct estimation. Hence SM'D D could ensure more aligned and reliable density estimation for sparse and multi-modal data.
model
E.4 Defection Score Methods Comparison
To study the effectiveness of our choice, i.e., I-OUT(-) defined by the norm of score model in Eq. ( we compare with existing benchmarks, MSP 20) Energy score [], and ASH [12]. As listed in Tab. 10 MSP is the runner-up method to detection, and it is flexible to detect in all baseline methods. However, T.) more competitive and reliable, since utilizes the global distribution as guidance.
E.5 Extensive Visualization Results
To explore the wild data distribution of FL OOD methods, we visualize T-SNE of data representations in Fig. 8 and the detection score distributions in Fig. 9 on Cifar10 5 for FedAvg, FedRol),
Table 10: Metric comparison FedRoD+FOOGD Cifar10.
Non-IID a
5.0
Method
FPR95
AUROC1
FPR95
AUROCT
FPR95.
AUROC1
MSP
47.96
80.95
37.02
86.49
36.13
86.64
Energy
61.90
85.55
49.73
90.93
54.10
91.12
ASH
51.06
89.55
42.36
92.11
38.77
93.14
+F00GD
32,99
91.76
25.51
94.19
18.91
96.25
30

---

14
00
(0) Am
Am
0.05
(c) 1m = 0.1
(d) Am 0.2
(e) 2m 0.4
(f) mm 0.5
(g) Am 0.8
(h) 2m 1.0
Figure 7: Motivation of SM The red points are sampled from target distribution, while the blue points are generated via Langevin dynamic sampling from random noise.
OUT
OUT
(a) FedATOL
(b) FedTHE
FedAvg
out
OUT
(d) FoOGvg+FOOGD
(e) FedRoD
(1) FedRoD+FOOGD
Figure 8: T-SNE visualizations of FedAvg and FedRoD with FOOGD.
FedAvg+FOOGD, FedRoD+FOOGD and their runner-up methods, FedATOL and FedTHE, respectively, It is evident that FOOGD represents IN-C data more tight with IN data, and constructs comparably clear decision boundary between IN data and OUT data. Besides, we also discover that FOOGD will push OUT data away from its IN and IN-C data, which validates the guidance from the global distribution. Additionally, Fig. 9 FOOGD makes the modes among IN, IN-C, and OUT, more separable than existing methods. This also proves the effectiveness of FOOGD in detection task.
E.6 Client Generalization on PACS Dataset
To validate the effectiveness of FOOGD in domain generalization tasks, i.e., each client contains one domain data and we train domain generalization model by leave-one-out, following FedIIR [17]. To obtain fair comparison, we pretrain all models from scratch and utilize adaption methods as stated in their main paper, instead of using public ImageNet pre-trained model. In terms of Tab. [1] FOOGD obtains performance improvements for FedAvg and FedRoD. Compared with existing adaption methods, FOOGD achieves outstanding results in the toughest task, i.c., leaving Sketch domain
31

---

-2
INC
0.-
RC
OUT
10
04-
DenoR)
Duro/t
0.3
0.3
0.0
FedATOL
(b) FedAvg+FOOGD
P6C
1.2
0.8
0.4
0.0
(c) FedTHE
FedRoD+FOOGD
Figure 9: Detection score distribution of FedAvg and FedRoD with FOOGD Cifar10 (a 5).
Table 11: OOD generalization task for PACS.
Method \Domain
Art Painting
Cartoon
Photo
Sketch
Average
FedAvg
97.21
62.58
91.00
35.28
71.52
FedRoD
93.45
88.85
89.34
29.95
75.39
FedT3A
97.13
75.71
93.21
37.40
75.86
FedII
86.86
80.29
88.98
31.38
71.88
FedTHE
96.17
90.72
93.57
29.14
77.40
FedICON
50.42
53.36
52.19
50.87
51.71
FvdAvg+FOOGD
97.46
89.32
91.48
41.40
79.92
FFOROD+FOOGD
97.85
92.31
93.01
50.95
83.53
out. This also concludes that FOOGD is capable of inter-client generalization, since FOOGD has utilized global distribution knowledge.
E.7 Extensive Experiments on Other IN-C and OUT data
In this part, we study the performance evaluation of FOOGD in additional IN-C and OUT datasets. In Tab. [2] we can find that FODGD consistently enhances the detection capability for different OUT data, validating for the effectiveness of estimating global distribution via SM Meanwhile, we compute the average results of different IN-C data on Fig. [2] and provide the details in Tab. [3]] and Tab. [1] FOOGD consistently improve generalization of enhancing feature extractor via SAG.
unseen IN-C data, indicating the effectiveness
E.8 The Study of Hyper-parameter Sensitivity
We vary the coefficient of SM'D An (0.11,02, in Fig. [I(a) coefficient of SAG X {0. 0.01, 0.05, 0.1, 0.2, 0.5. 0.8) in Fig. [1(c) to obtain the best modeling in FOOGD. To study the effect of different client numbers, we vary the number of participating clients K = (5, 10, 20. 50} in Fig. and find FOOGD can have better results among different participating clients.
11(b) and vary the
32

---

ACC-IN-C
ACC-IN
88
86
84
Fed-ATOL FedAvg+
Fed-ATOL
84
FedAvg+FOOGD
82
10
20
50
5
10
20
50
number of clients K
number of clients K
(a) IN ACC
(b)
30
AUROC
FPR95
Fed-ATOL FedAvg+FOOGD
94
Fed-ATIOL
20
92
FOOGG
50
number of clients K
number of clients K
(c) OUT FPR95
OUT AUROC
Figure 10: Effect of participating clients numbers K.
91.30
37.2
AUROC
FPR95
17.0
Accuracy
IN
36.8
91.15
Am
2
Ag
(a) Amn
(b) Am
(c) X
Figure 11: Effect of Am and Xa-
Table 12: Other detection results Cifar10 (a 0.1).
OUT Data
ISUN
SVHN
LSUN-R
Texture
Method
FPR95
AUROC
FPRY5
AUROC
FPRUS]
AUROC:
FPRUS,
AUROC:
RodArg
62.10
76.29
80.02
62.14
62.01
77.02
80.53
66.23
FedLN
66.41
76.03
70.95
76.82
61.31
78.34
93.90
71.99
FoJATOL
61.01
80.05
85,39
82.17
64.01
79.89
66.33
78.77
FedlIR
57.86
77.98
83.68
64.04
58.44
78.69
91.72
62.32
FedArg
FedRoD
43.40
82.83
40.72
83.55
41.80
82.92
53.24
$1.52
FOSTER
39.55
48.09
54.23
77.62
FeITHE
43.72
83.50
39.22
85.95
42.95
83.46
53.58
82.19
Red[CON
49.98
82.95
34.94
85.56
49.05
83.30
51.57
80.96
33

---

60
75
ACC-IN-C
ACC-IN-C
40
50
FedAvg
FeiLN
FedATOL
FedT3A FellIR
+FOGG
FedRoD FOSTER
FedTHE FedICON
+FOOGD
FodAvg
Fed
FedATOL
FedT3A
FedlIR
#ROOGD
FedRoD
FOSTER
FeiTHE FedICON
#FOOGD
(a) Cifarl0 a 0.1
(b) Cifar100 a = 0.1
Figure 12: Average generalization results. +FOOGD is short for FedAvg+FOOGD and FedRoD+FO0GD,
respectively.
34

---

FedRoD+ D
+FOOGD
92.74
86.12
75.44
81.98
86.38
76.84
86.62
80.45
88.08
83.38
89.80 $9.80
86.27
81.55
83.12
60.88
87.90
79.16
91.81
81.83
84.23
FedICON
89.18
86.35
70.16
81.22
82.32
87.09
83.04
76.43
85.41
86.63
86.37
83.48
76.82
80.34
86.05
85.23
85.08
88.64 64
80.43
83.17
FedTHE
89.61
83.85
75.42
80.10
83.43
71.85
82.31
78.49
84.74
84.35
87.50
84.67
78.37
80.31
82.07
87.66
81.32
88.73
79.72
82.34
FOSTER
83.35
75.03
81.76
83.05
72.82
82.76
76.52
85.71
84.44
86.68
84.82
76.80
78.94
82.41
85.63
81.88
87.06
78.63
81.95
88.
FedRoD
89,90
81 81.48
68.49
77.86
81.11
71.87
81 81.20
75.25
85.65
82.13
86.61
83.37
73.75
76.62
79.27
85.55
77.97
88.49 19.49
76.60
80.17
FedAvg+FOOGD
73.71
60.96
45.80
51.05
11.90
49.14
63.84
52.33
64.37
58. 58.66
66.55
59.18
53.92
58.31
52.97
65.31
53.26
71.98
57.72
59.00
Table 13: IN-C generalization for FL methods trained Cifar10(a = 0.1).
FedIIR
66.12 12
54.85
44.53
44.23
55.52
41.35
55.91
48.45 45
59.10
52.72
60.46
53.21
49.15
53.09
46.57
60.97
47.51
63.32
53.20
53.17
Fed-T3A
61.14
49.52
40.41
45.51
51.52
36.93
52.16
42.79
53.34
52.60
57.64
52.45
45.27
48.82
49.48
55.25
48.52
58.41
48.38
50.01
Fed- Fed-ATO ATOL
54.44
48.17
28.97
41.52
45.64
38.90
41.25
40.36
46.20
46.36
47.94
45.35
35.20
39.57
42.33
51.54
43.23
54.39
40.03
43.76
FedLN
71.77
60.82
42.33
52.65
60.55
45.02
58.22
50.52
62.00
61.08
68.61
61.29
50.25
54.55
54.88
67.33
55.64
71.76
54.30
58.08
FedAvg
65.73
53.89
43.13
41.30
54.80
41.25
56.21
49.32
56.88
52.37
61.56
52.12
48.66
52.73
45.15
62.18
46.86
63.62
52.25
52.63
IN-C Type
Brightness
Frosted glass blur
Motion blur
Contrast
Impulse noise W
Pixelate
Defouce blur
Elastic transform
Gaussian Noise
Shot noise
Zoom blur
Spatter
Gaussian blur
Saturate
Speckle noise
average
Fog
Snow
Frost
Jpeg
35

---

F
75.70
64.23
38.83
60.09
61.34
51.06
43.30
12.88
64.20
65.51
65.66
46.47
53.00
60.09
58.32
66.93
52.63
58.67
60.
63
FedICON
67.79
56.16
37.12
53.58
54.79
43.62
51.08
39.21
52.17
60.08
62.57
60.13
37,00
43.64
55.38
62.92
54.41
58.24
44.28
52.32
FedTHE
69.09
57.69
36.78
53.13
55.80
44.98
53.15
38.44
51.60
60.75
63.55
61.34
35.51
42.33
55.92
64.11
54.56
59.87
43.39
52.74
FOSTER
67.50
56.01
39.94
53.53
55.40
42.91
52.36
40.68
54.97
60.76
63.10
61.11
38.43
44.96
56.06
63.32
55.41
58.40
45.67
53.19
FedRoD
69.26
56.26
34.86
54.24
54.61
42.32
50.08
38.66
53.73
61.02
62.51
61.36
33.41
40.32
56.77
63.20
55.11
59.39
41.75
$2.05
FedAvg+FOO00 +FOGGD
51.69
40.98
27.44
39.68
40.64
30.98
38.54 54
26.24
42.52
46.23
45.81
47.47
28.28
32.81
41.62
49.59
40.63
44.87
32,86
39.42
Table 14: IN-C generalization for FL methods trained on Cifar100(a = 0.1).
FediIR
47.88
36.80
19.67
33.34
35.69
26.94 66.94
33.33
21.84
33.31
39.92
41.90
39.36
21.79
27.03
34.75
42.04
35.39
38.92
27.47
33.55
FedT3A
51.50
41.45
27.48
37.65
39.41
29.88
37.40
23.65
42.10
44.18
46.63
44.72
22.50
29.20
39.34
47.32
38.98
42.19
29.31
37.63
FedATOL.
41.08
33.87
18.40
30.23
32.39
26.96
29.50
21.58
32.51
34.67
33.64
33.89
19.67
24.01
30.63
36.39
30.63
35.61
24.47
30.01
FedLN
48.15
37.11
27.32
35.09
27.10
35.38
24.26
36.11
41.05
43.36
41.49
24.83
30.28
36.51
43.90
36.29
39.43
30.53
35.62
38
FedAvg
46.85
36.15
20.96
32.95
35.09
26.39
32.53
22.99
34.41
39.17 17
41.17
38.65
21.21
26.37
33.82
42.41
34.18
38.59
26.43
33.17
IN-C Type
Brightness
Fog
Frosted glass blur
Motion blur
Snow
Contrast
Frost
Impulse noise W
Pixelate
Defouce blur
Jpeg
Elastic transform
Gaussian Noise
Shot noise
Zoom blur
Spatter
Gaussian blur
Saturate
Speckle noise
Average
36

---

Table 15: Main results of federated OOD detection and generalization on Cifar10. We report the ACC of brightness as IN-C ACC, the FPR95 and AUROC of LSUN-C OUT performance.
0.29
0.558
0.29
87 77.66 +:047
2:44
1:
+
0.20
AUROCY
85.37
90.87
25.25
85.37
97.77
86.65
86. 86.19
86.79
96.25
38.24 0.55
1133:22:1 2.38
2.62
38 24 0.55
+ 1.07
12.02 0.34
1: 1.14
35.35 = 1.94
0.96
PPROS!
33.3311
16.13
33.33
19.06
34,69
12.02
36.42
18.91
ACC-IN-C
0.4.9
1 1.43
1.32
+ 1.46
0.95
0.12
4: 1.19
1: 1.02
0.57
88 0.37
85 08
83 08.08
88 28
85 01
86 10
88 25
85 96
86 18
86
85
87.20 1 1.26
- 0,68
0.33
: 0.66
87.69 1 0,88
8783 + 1.38
88.14 + 0.24
+ 1.13
0.88
ACC-INT
06.5010..
86. 46.50+0.3
87.77
89.44
84 0.23
0.52 -
= 0.23
+ 0449
018
+ 1.11
= 1.9
0.78
AUROCT
64.64
93.48
84 94
96.55
83.91
30
94.1
84
85
0.83
1 1.14
1.32
0.83
= 0.64
- 0.02
+ 1.51
2.43
1.58
1.46
FPRO5.
0.5
43.70
39
27.87
43,70
40.91
17.78
42.03
40.
56
56
25.51
ACC-IN-CT
83.72 1.74
1 1.82
82 2 0.44
192
85.82 # 1.10
87.68 = 0:41
0.51
84 20.20:2
85 65.64
84 74.75
87 87.26
87.70
75.35
90.16
66
1.13
- 0.89
0.91
86 59 - 1.13
75 : 0.98
88.34 043
+ 1.85
= 0.93
+ 1.07
0.78
ACC-IN
86, 59
86 10
87.55
86 92
89.14
75.83
90 46
89
0.89
1 0.37
2.74
+ 1.38
+ 1.38
0.78
+ 0.93
82.04 + 1.15
1 0.26
AUROCT
58.05
84.14
86.22
63.64
63.31
91.21
77.43
11.28
91.76
1.57
1 091
1.59
1.57
+ 0.99
1.02
1.27
2.79
130
FPRU5,
83.41
56.14
19.50
78. 12
79 .8
35.32
47.97
47.4 77.40
48.22
32 29.99
0.1
49
58.
ACC-IN-CI
13441111 1.18
71.7711:00 0.67
1.72
1.
06.12:00.4
0.74
0.93
88.70 : 0.82
89,71 0.91
3E 0.46
01710000
65.44
71.77
54.44
61.52
66.12
73,71
89.18
92.74
89
68.03 +: 1.17
75.24 1: 0.44
55.93 1.87
08.03 1.17
68 26 + 0.66
25.09 1 0.79
90 22 : 0.88
91 91.05 + 0.66
93.51 I 0.65
ACC-INT
91 1:
89.06
FedAwg+FODGE
Non- IID
Method
FedAvg
FeilN
FedATOL
FedT3A
FediR
FedRoD
FOSTER
FoETHE
FedICON
37

---

We report the ACC of brightness as IN-C ACC, the FPR95 and AUROC of
70.59 0:199
1.75
+:0.9
+
0.15
1.31
1.72
+
#:03
AUROCT
75.5 73.87
60.51
70.59
69.07
84.03
62.76
65.4
69.01
70.84
+ 0.34
09.18 0.46
1.61
0.34
72.57+0.33 0
$7.7010.8 0,8
74,86 = 0.98
76. 76.94 + 1,02
0.64
0,64
#
1.19
FPR99L
72.68
0818111
71.43
72.68
80.27
72.68
72.57
70,91
3.0
ACC-NN-C
56.72 2.17
0.82
0.42
0:96
18 0.31
1.21
+116
0.28
0.72
15.00122
57.62
56.33
63
55.51
57.65
64
55,80
49.28
57.03
57.62
62 62.56
ACC-IN 1
70.86 = 1.52 61.40 61.40 0.12
0.12
61.00
14.16.0.88 0.81
0.12
61.70 0.76
0,51
61.34
0.78
33.80
0.31 0.31
+ 0122
62.11 0.74
61:40
64.16
64.95
61.34
61.03
62.11
73.41 0,33
79.25 27 1 061
10.86 = 1.2
65.88 65 046
024
= 066
69 39 1.64
0.39
AUROCT
86 42
69.48
48
71.03
72.84 = 0,81
1.24
0.81
72.8 # 1.56
+ 1.10
0,78
0.86
1.13
+ 1.84
1 1.02
0.67
0.5 FPRY5,
68. 68.31
70.
77.62
55,70
70.13
23.26
72.95
69
99
ACC-IN-C
58 28 # 0.48 54.62 +: 0.67
56.63 1 0.1
55-42 1.63
0.31
61.28 0.88
61.19 0.92
61.3 0.5
0.5
Table 16: Main results of federated OOD detection and generalization on Cifar 100. LSUN-C as OUT performance.
53.86
55.72
39 99.9
ACC-IN 1
0.72
0.61
0.48
38.06 #
61.82 0.20
88 0.61
62.45 0.55
+ 0.68
65 86 0.81
50 39
60.62 62
58 28
58.00
66,22
30
66
AUROCT
67.16 + 1.17
0,33
67.16 + 1.17
0.53
91.41 0.36
+ 1.82
+
0,34
7 0.55
81.64
63.99
73.02
75.44
86.07
78.35 + 1.64
0.96
+ 1.64
+ 153
01.25 + 1.05
64.73+0.33 0.79
61.36
039
048
HPR95,
65.26
78.35
36.40
66.34
64.73
61.36
66
ACC-IN-C
47.54 0.48
1
0.60
11500=0.00
0.49
1.19
0.12
0024:044
0.41
0.56 0.56
67.79
0031 31
75.70
0.26
48.15
41.08
$1.50
47.88
$.69
69.26
67.50
69.09
67.79
75.70
51.67 + 1.37
43.65 1 0.54
51.67 + 1.37
51.63 + 061
53.84 0.83
23.13 + 0.85
73.85 + 0.48
72.22 : 0.72
ACC-NN
52.48 +
22.54
77 88
f POOOO
FedRoD&FODDD FOODD
Non IID
Method
FedAvg
FodLN
TedATOL
F
FodIR
FedkeD
FOSTER
FedTHE
FedICON
38


Reviews:
[{"review_id": "dez1waGZ50", "reviewer": "Reviewer_RpTd", "summary": "This paper focuses on the federated learning setup that non-IID, semantic-shift, and covariate-shift take place in the same time. And the authors propose a framework of FOOGD with a unnormalized distribution estimation method, i.e., SM3D, to release the distribution assumption and constraints in heterogeneous modeling. Both federated OOD generalization and detection can be soundly solved via SM3D and SAG modules. Compared with existing methods, FOOGD is validated via extensive experiments for effectiveness.\n\nAfter rebuttal, I raise the Rating from 7 to 8.", "questions": "1.\tHow to enhance the generalization for unseen client distribution?", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe presentation is straightforward and clear, with detailed and instructive figures. The formula derivation is thorough and essential.\n2.\tThe paper addresses a significant challenge in federated learning, focusing on learning from large-scale non-IID data with out-of-distribution (OOD) shifts.\n3.\tThe authors conduct extensive and robust experiments, effectively demonstrating the effectiveness of the proposed FOOGD method.\n4.\tDetailed theoretical foundations are laid out for both the estimation model and the consistency of convergence optimization.\n5.\tUtilizing score modeling for federated learning is quite novel and insightful for heterogeneously distributed data.", "weaknesses": "1.\tHow to understand IsOUT() function with negative threshold in Equation 9?\n2.\tScore model brings additional computation cost. \n3.\tThe correspondence between federated loss of $\\mathcal{L}_k$ and the practical loss $\\mathcal{L}_s$ and $\\mathcal{L}_f$ are not explicitly explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for appreciating the contributions of our work. For the weakness concerned by the reviewers, we provide the discussion as below.\n\nW1: How to understand IsOUT() function with negative threshold in Eq. (9)?\n \n A1:IsOUT() function is the OOD detection function in our work. When the norm of score function of the data sample is larger than a given threshold, the probability of OUT is higher. The threshold $\\tau$ can be both positive values and negative values.\n If the threshold $\\tau>0$, the bigger the norm than $\\tau$ indicates more potential to be OUT data samples.\n\n$\\operatorname{IsOUT}(\\boldsymbol{x})=$ True, when $\\left\\|s_{\\boldsymbol{\\theta}^*}\\left(f_{\\boldsymbol{\\theta}^*}(\\boldsymbol{x})\\right)\\right\\|>\\tau ; \\quad$ otherwise, IsOUT $(\\boldsymbol{x})=$ False.\n\n On the contrary, if the threshold $\\tau<0$, the bigger the norm than $-\\tau$ indicates more potential to be OUT data:\n\n$\\operatorname{IsOUT}(\\boldsymbol{x})=$ True, when $\\left\\|s_{\\boldsymbol{\\theta}^*}\\left(f_{\\boldsymbol{\\theta}^*}(\\boldsymbol{x})\\right)\\right\\|>-\\tau ; \\quad$ otherwise, IsOUT $(\\boldsymbol{x})=$ False.\n\nBoth formulations of OUT detection functions are the same, we choose the later one in order to have approximately consistent visualization with existing detection scores, e.g., MSP, in our experiments. \nThat is, in Fig. 5 of main paper, the detection score distributions are consistently in the order, i.e., OUT in the left, IN-C in the middle, and IN in the right. \t\t\n\n\n W2: Score model brings additional computation cost.\n\nA2: **The computation cost of score model is slightly increasing but controllable.**\n Specifically, we estimate the score model in latent representation space, which reduces the computation cost of score function of original image data[1]. Besides, in implementation, we choose small 3-layer MLP models to capture the distribution in 512-dimentional latent space.\n In Tab. a, we provide the computing cost of different modules in FOOGD, e.g., forward of main task, score estimation of $SM^3D$. The most computation burden is due to main task inference, i.e., 88.97%-90.00% of the total computation time.\n Though computation is added for modeling distribution, FOOGD enhances its robustness for OOD shifts data. We believe the advantages of FOOGD outperforms its short-comings.\n\n  **Table a. Average computation cost of batch samples for training score model.**\n|            \t  \t| Main task | $SM^3D$ | Total |\n|--------------------|-------------------|-----|-----|\n| Cifar10 $\\alpha=0.1$ (s)  | 0.0252\t   | 0.0031  | 0.0283 |\n| cifar100 $\\alpha=0.1$ (s) | 0.0281    \t| 0.0031   | 0.0312   |\n ||\n\n\n[1] Vahdat A, Kreis K, Kautz J. Score-based generative modeling in latent space[J]. Advances in neural information processing systems, 2021, 34: 11287-11302.\n \n \n W3: The correspondence between federated loss of $\\mathcal{L}_k$ and the practical loss $\\mathcal{L}_s$ and $\\mathcal{L}_f$ are not explicitly explained.\n \n A3: The loss $\\mathcal{L}_k$ is the general optimization objective in client $k$, while $\\mathcal{L}_s$ enhance the score estimation for detection which corresponds with the implementation of $\\ell_k^{OUT}$, besides, $\\mathcal{L}_f$ enhance feature extractor for generalization, which further corresponds to $\\ell_k^{IN}$ and $\\ell_k^{IN-C}$.\n In summary, \n$\\mathcal{L}_k= \\ell_k^{IN}+ \\ell_k^{IN-C}+ \\ell_k^{OUT}= \\mathcal{L}_s+\\mathcal{L}_f$.\nWe will clarify this relationship in our later version.\n \nQ1: How to enhance the generalization for unseen client distribution?\n\nA: In FOOGD, we maintain a score model that estimates the overall global distribution of in-distribution data. Therefore, for any new client, we can warm-start its local score model with a global one, which is directly applicable. In Table 4 of our empirical study main paper, we evaluate FOOGD on PACS by leaving one domain untrained, achieving competitive performance. This validates the effectiveness of unseen clients.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nvlQcK9Ubu", "reviewer": "Reviewer_MUVw", "summary": "This paper addresses various OOD (Out-of-Distribution) shifts that may occur in federated settings by proposing a unified framework to simultaneously tackle OOD generalization and detection issues. Specifically, this paper introduces FOOGD, which estimates arbitrary client probability densities to create a reliable global distribution that guides both OOD generalization and detection. Besides, it presents $SM^3D$ and SAG to respectively enhance OOD detection and generalization capabilities. Finally, extensive experiments demonstrate the effectiveness of the proposed methods", "questions": "See weaknesses.", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a novel unified framework to simultaneously address OOD (Out-of-Distribution) generalization and detection problems in federated settings.\n\n2. The method has theoretical guarantees and empirical analysis to ensure its effectiveness.\n\n3. The structure of this work is logical, and the writing is well-organized.", "weaknesses": "1. This paper proposes the FOOGD, $SM^3D$, and SAG methods to address OOD generalization and detection problems. It is essential to clearly define the relationships among these three methods in the abstract and introduction to enhance the readability of the entire paper.\n\n2. This paper uses 'semantic-shift' to describe the issue of unknown class bias in OOD detection, following the traditional machine learning concept[1]. However, in federated learning, this problem is generally referred to as 'concept shift'[2][3]. It is recommended to revise this terminology to align better with the conventions of the federated learning community.\n\n3. In the related work section (Section 2.3), it is recommended to include a discussion on 'concept shift' in federated learning and cite relevant literature.\n\n4. In the methodology section, given the involvement of multiple innovative components, it is recommended to analyze their complexity.\n\n\nReferences:\n\n[1] Haoyue Bai et al. Feed two birds with one scone: Exploiting wild data for both out-of-distribution generalization and detection. ICML, 2023.\n\n[2] Panchal K et al. Flash: concept drift adaptation in federated learning. ICML, 2023.\n\n[3] Jothimurugesan E et al. Federated learning under distributed concept drift. ICAI, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive review! For your concerns, we explain them in the following.\n\nW1:  It is essential to clearly define the relationships among FOOGD, $SM^3D$, and SAG, in the abstract and introduction to enhance the readability of the entire paper.\n \n A1: FOOGD is the overall federated learning framework for adapting wild data.\n In FOOGD, each client or server maintains a feature extractor and a score model.\n And $SM^3D$ is devised for training a score model that estimates distribution and detects OUT data.\n While SAG is proposed to enhance feature extractor in the main task, e.g., classification, as well as generalization based on score estimation.\n We will clarify it in the abstract and introduction of our later version.\n \n W2: However, in federated learning, this problem is generally referred to as 'concept shift'[2][3]. It is recommended to revise this terminology to align better with the conventions of the federated learning community.\n \nA2: **FOOGD is proposed for adapting non-IID data and OOD shifts data, which is orthogonal with FL methods solving 'concept shift'.** OOD shifts are commonly existing in federated scenarios, and various FL methods are proposed for OOD detection, e.g., FOSTER, and OOD generalization, e.g., FeDIIR and FedTHE.\n **It is possible for FOOGD to resolve the 'concept shift' issue, similarly following the effectiveness in OOD generalization.**\n Specifically, FOOGD adaptively estimates the arbitrary data distribution, and aggregates the global distribution on the server. Then FOOGD utilizes the newly updated score model as distribution guidance, for concept shift data.\n Since FOOGD utilizes global distribution and outperforms on the tasks of new unseen clients with PACS data, it can adapt to concept shift data as well. We are willing to elaborate the relationship between FL with OOD data and 'concept shift' data in our later version.\n\n\n \n W3: In the related work section (Section 2.3), it is recommended to include a discussion on 'concept shift' in federated learning and cite relevant literature.\n\n A3: We are willing to extend our related work discussion on 'concept shift' in federated learning and cite relevant literature, e.g., Flash and FedDrift.\n \n \n \n W4: In the methodology section, given the involvement of multiple innovative components, it is recommended to analyze their complexity.\n\n\n A4: For newly added modules, the complexity of $SM^3D$ is $O((T+1)BD + B^2 D)$, and the complexity of SAG is $O(B^2 D)$, where $B$ is the batch size of samples, $T$ is the steps of sampling, and $D$ is the feature dimension .\n \n**$SM^3D$ is responsible for estimating the score model for density estimation and OOD detection. The main procedure of $SM^3D$ consists of score matching (SM), Langevin Dynamic sampling (LDS), and Maximum Mean Discrepancy (MMD) computation.**\n \n- **Score Matching:** The complexity of computing score matching is $O(B D)$.\n- **Langevin Dynamic Sampling (LDS):** This involves $T$ steps of sampling, each with a complexity of $O(B D)$. Therefore, the overall complexity for LDS is $O(T B D)$.\n- **Maximum Mean Discrepancy (MMD):** Computing the MMD involves pairwise comparisons of samples, leading to a complexity of $O(B^2 D)$.\n \nCombining these, the complexity of $SM^3D$ is $O((T+1)BD +  B^2 D)$.\n \n**SAG enhances the generalization capability of the feature extractor by aligning distributions of original and augmented data measured by Kernelized Stein Discrepancy (KSD).**\n \n- **Kernelized Stein Discrepancy (KSD):** This involves computing gradients and kernel functions between pairs of features. The complexity is $O(B^2 D)$, where $B$ is the batch size of samples, and $D$ is the feature dimension.\n \nThus, the complexity of SAG is $O(B^2 D)$.\n \n We also provide the empirical reports in Tab. a-b. It is obvious that the most computation cost stems from main task inference. Though the newly modules bring additional cost, the overall burden is controllable and effective in OOD data.\n \n  **Table a. Average computation cost of batch samples for training score model.**\n|            \t  \t| Main task | $SM^3D$ | Total |\n|--------------------|-------------------|-----|-----|\n| Cifar10 $\\alpha=0.1$ (s)  | 0.0252\t   | 0.0031  | 0.0283 |\n| cifar100 $\\alpha=0.1$ (s) | 0.0281    \t| 0.0031   | 0.0312   |\n ||\n\n \n  **Table b. Average computation cost of batch samples for training feature extractor.**\n|        \t  \t \t| Main task | SAG | Total |\n|--------------------|-------------------|-----|-----|\n| Cifar10 $\\alpha=0.1$ (s)  | 0.0503      | 0.00212  | 0.0524 |\n| cifar100 $\\alpha=0.1$ (s) | 0.0561\t\t| 0.00236   | 0.0585   |\n ||", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7TgMXEytrk", "reviewer": "Reviewer_cFix", "summary": "This paper introduces FOOGD, a federated collaboration framework designed to achieve both out-of-distribution (OOD) generalization and detection. FOOGD estimates the probability density of each client to obtain a reliable global distribution and incorporates the SM3D model and SAG module to enhance the detection capability of the score model and the generalization ability of the feature extractor.", "questions": "1.\tCould the experimental results section provide a more detailed analysis of OOD to better explain the superiority of the FOOGD method in this aspect?", "limitations": "See the above weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tFOOGD is the first framework in federated learning to simultaneously address OOD generalization and detection, demonstrating significant innovation.\n2.\tThe paper is written clearly and coherently, and it is generally well-organized.\n3.\tThe authors have made an effort to highlight their contributions, providing a reasonably good overview of the key points and innovations presented in the paper.", "weaknesses": "1.\tThe experimental section discusses the results and performance comparisons. Although this section provides performance metrics for multiple datasets, the description of the chosen statistical analysis methods (such as whether hypothesis testing or confidence interval calculation was performed) is insufficient, affecting the reliability of the result interpretation.\n2.\tEvaluation metrics like ACC-IN and ACC-IN-C are not explained.\n3.\tThe related work section only introduces some comparison methods and lacks more related work, such as FedAvg. Besides, some references are incomplete, such as ref. 46.\n4.\tNo open-source code has been provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate it a lot for your insightful reviews.\n\nW1: Although this section provides performance metrics for multiple datasets, the description of the chosen statistical analysis methods (such as whether hypothesis testing or confidence interval calculation was performed) is insufficient, affecting the reliability of the result interpretation.\n\nA1: We follow FedTHE to choose accuracy for OOD generalization, and SCONE to choose AUROC and FPR95 for OOD detection, for fair comparison.  We will extend the description of the chosen statistical analysis methods in our future work.\n\n \n W2: Evaluation metrics like ACC-IN and ACC-IN-C are not explained.\n\n A2: ACC-IN and ACC-IN-C are commonly used accuracy for the two test sets, i.e., in-distribution data (Non-IID data) and covariate-shift data (OOD Generalization), respectively. We follow FedTHE to choose ACC-IN and ACC-IN-C. We will explain it in detail in our future version.\n\n \n W3: The related work section only introduces some comparison methods and lacks more related work, such as FedAvg. Besides, some references are incomplete, such as ref. 46.\n\nA3: We provide the illustration of FedAvg in line 671 of supplemental pages, and ref. 46 is the SVHN dataset for detecting OUT samples. We will elaborate the related work during reversion.\n\n \nW4: No open-source code has been provided.\n\nA4: We will release our project in the later version, and we have released the test model trained on Cifar10 $\\alpha=0.5$ for FedAvg-FOOGD, in our supplements, for validating the effectiveness of FOOGD.\n\n Q1: Could the experimental results section provide a more detailed analysis of OOD to better explain the superiority of the FOOGD method in this aspect?\n\nA5: **Yes, we can provide more detailed analysis and corresponding evaluation intention in our future vesion.** \nFOOGD demonstrates its superiority in OOD shifts data by three main aspects:\n(1) In Tab. 1-2 and Tab. 4-5: Superior performance of non-IID data classification, OOD detection and generalization for setup of different degrees of non-IID, and various datasets (Cifar10, Cifar100, TinyImageNet, PACS).\n(2) In Fig. 4:  FOOGD has more separable clustering among IN, IN-C, and OUT data, in feature space, and FOOGD pushes OUT data away from its IN and IN-C data, which validates the guidance from the global distribution.\n(3) In Fig.6 (Tab. 6) and Fig.7: Extensive OOD detection  and generalization tasks consistently achieve performance enhancement, affirming the reliable performance generalization of FPPGD in handling OOD data.  \nWe will improve the the details of relavant empirical studies, in our future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QGZ4YcHSF", "reviewer": "Reviewer_dEK5", "summary": "This paper introduces a FL framework FOOGD aimed at addressing the simultaneous challenges of the out-of-distribution (OOD) generalization and OOD detection in decentralized environments. FOOGD estimates the data distribution of different clients through SM3D and introduces SAG to ensure the consistency of features.", "questions": "(1) The dual approach of handling both OOD generalization and detection introduces significant computational complexity and overhead. However, FOOGD is not clearly stated, and there are no experiments to quantitatively analyze it. The iterative process of updating and aggregating both main task models and score models can still be costly in terms of bandwidth and synchronization requirements among clients.\n(2) The need to transmit detailed model parameters and score function values between clients and the server can lead to high communication costs. In addition, the score function values may cause privacy leakage issues, thereby attacking the score model.\n(3) While the FOOGD framework aims to generalize across different client distributions, the inherent diversity and possible sparsity of client data can cause the model to overly adapt to specific local patterns, reducing its effectiveness in capturing broader, global trends.\n(4) The effectiveness of FOOGD heavily relies on the accuracy of the local density estimation provided by the SM3D component. However, in scenarios where data distributions are highly complex or sparse, it may lead to less reliable detection of OOD samples and potential false positives or negatives.", "limitations": "The authors indicated the paper's focus (on privacy issues) as an limitation but not elaborated in paper itself.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "FOOGD is designed to handle both OOD generalization and OOD detection tasks and integrates mechanisms to manage covariate shifts and semantic shifts within the FL setup, which is helpful for deployment environments. The use of SM3D allows for accurate estimation of the probability density of client data distributions and is used to identify out-of-distribution samples. In addition, SAG helps maintain feature space invariance between in-distribution and augmented data samples. Finally, this paper demonstrates the effectiveness of FOOGD through experiments.", "weaknesses": "There is  no training-log provided in supp materials. There are also questions regarding the foundation methods as listed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your professional and detailed review! \n\n W: The training log is not provided.\n \nA: To validate the effectiveness and implementation, we release a well-trained FedAVG-FOOGD model on Cifar10 $\\alpha=0.5$ in supplemental materials. And we are willing to release our project as well as the training logs in our later version.\n\n Q1: The computational and communication analysis of introducing score models.\n \nA1: **FOOGD introduces fairly controllable communication cost and computational cost, but brings competitive and flexible adaptation capability in OOD shifts data.**\n**(1) computation cost: the complexity of $SM^3D$ is $O((T+1)BD + B^2D)$, and the complexity of SAG is $O(B^2 D)$, where $B$ is the batch size of samples,$T$ is sampling steps, and $D$ is feature dimension.**\n$SM^3D$ Module consists of Score Matching ($O(BD)$) Langevin Dynamic Sampling ($O(TBD)$), and Maximum Mean Discrepancy computing ($O(B^2D)$), bringing overall computation as $O((T+1)BD + B^2D)$. \nSAG Module needs to compute Kernelized Stein Discrepancy, requiring $O(B^2D)$.\n\nWe also provide the empirical reports in Tab. a-b. While both $SM^3D$ and SAG introduce extra computation burden, the most proportional of computation is due to main task inference, e.g., about 89% for training $SM^3D$ and 95% for SAG.Both SM3D and SAG add extra computation, but the main task remains the most significant part.\n\n **Table a. Average computation (s) of batch samples for training score model.**\n|| Main task|$SM^3D$|Total|\n|-|-|-|-|\n|Cifar10|0.0252| 0.0031|0.0283|\n|Cifar100|0.0281| 0.0031|0.0312|\n\n \n  **Table b. Average computation (s) of batch samples for training feature extractor.**\n|  | Main task | SAG | Total |\n|-|-|-|-|\n|Cifar10|0.0503|0.00212|0.0524|\n|Cifar100|0.0561|0.00236|0.0585|\n\n **(2) communication cost: we transmit maintask model (2,243,546 parameters) and score model (132,992 parameters). Score model brings additional but controllable communication burden, increasing 5.93% overburden.** We can observe that the most significant communication burden is derived from main task rather than additional modules. \n\nQ2: Introduce score model bring communication cost and privacy leackage.\nA2:**Score model introduce slightly controllable communication burden, and no additional privacy risk.(1) Communication Burden:** FOOGD only transmits score model parameters between clients and the server, adding just 5.93% extra overhead. Score function values are not transmitted; the score models carry only distribution knowledge. The communication cost details are provided in A1.**(2) Privacy:** FOOGD poses no additional privacy risk compared to existing FL methods. The score model captures data probability in latent distribution, revealing only the direction and distance towards the in-distribution, making it hard to reverse-engineer original data. The only risk exposure of each client is transmiting model parameters, i.e., feature extractor model and score model. But this kind of privacy exposure can be mitigated via enhanced privacy-preserving techniques, e.g., differential privacy. We will explore this in our future study.\n\n Q3: The inherent diversity and possible sparsity of client data can cause the model to overly adapt to specific local patterns, reducing its effectiveness in capturing broader, global trends.\nA3: **FOOGD avoids over-adapting to local patterns and effectively captures global distributions, even with diverse and sparse client data.** \n(1) **Local Modeling:** SM3D estimates data distributions broadly, addressing sparsity, and uses MMD for accurate alignment. SAG ensures invariant alignment, enhancing generalization.\n(2) **Global Modeling:** Aggregated local score models form a global score model, ensuring consistent global distribution in SAG, as guaranteed by theorem 4.1.\n(3) **Empirical Evaluation:** For α=0.1, FOOGD outperforms baselines on all datasets, e.g., Cifar10, TinyImageNet and PACS, demonstrating robust performance with diverse and sparse data.\nThese points highlight FOOGD's superior handling of diverse and sparse data in federated learning.\n\n Q4:  When data distributions are highly complex or sparse, it may lead to less reliable detection of OOD samples and potential false positives or negatives.\n A4: **$SM^3D$ accurately captures local data density for complex and sparse data, providing reliable OOD detection.**\n **Firstly, we explain the mechanism of $SM^3D$.**\n  As introduced in Sec. 3.2, $SM^3D$ achieves better distribution modeling for complex and sparse data, since it benefits from both denoising score matching (DSM) and maximum mean discrepancy (MMD). $SM^3D$ takes the approximate distribution prior from DSM, and utilizes MMD to mitigate the impact of complex and sparse data pattern by minimizing the distribution between generative data samples from wider distribution space, and original data samples.\nThis is validated in lines 184-201. The visualizations show that with a balanced trade-off $\\lambda_m=0.1$), $SM^3D$ achieves more compact density estimation, providing comprehensive data density estimation.\n**Secondly, we also provide various empirical studies that validate the effectiveness of $SM^3D$.**\n**(1) $SM^3D$ directly enhances federated OOD detection even in sparse and complex data.** For instance, in the Cifar10 experiments with α = 0.1 Tab. 1-2, FOOGD with $SM^3D$ outperforms other baseline methods. Besides,  $SM^3D$ creates more separable modes among IN, IN-C, and OUT data, which is good for detection. \n**(2) Removing $SM^3D$ causes significant performance drop.** Removing $SM^3D$, the AUROC drops to 88.88%, indicating the necessity of $SM^3D$ for optimal performance.\n**(3) $SM^3D$ implicitly improves federated OOD generalization, even for unseen and complex data.** With $SM^3D$, FOOGD has  an average accuracy of 79.92% on PACS. **These experiments collectively demonstrate the effectiveness of $SM^3D$ in federated OOD detection, and generalization.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper focuses on simultaneously dealing with OOD detection and OOD generalization in FL scenarios for the first time. To solve this novel challenge, the proposed method, FOOGD, that estimates the probability density of each client to derive the global data distribution using SM3D, and employs SAG to train a feature extractor that captures invariant yet diverse knowledge, enabling both local covariate shift generalization and inter-client generalization. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.
The reviewers initially suggested the following strong points and concerns.

Strong points:
- FOOGD is the first framework in federated learning to simultaneously address both OOD generalization and detection, effectively tackling a significant challenge in the field.

- The use of score modeling in federated learning is both novel and insightful, particularly for managing heterogeneously distributed data.

- The theoretical and empirical analyses effectively demonstrate the effectiveness of the proposed method.

Concerns:

- The computational costs of the proposed method seem to be high (as noted by all reviewers).

- Certain sections lack sufficient explanations or detailed content.

- There are privacy concerns regarding the transmission of detailed model parameters and score function values between clients and the server.

- The effectiveness of the proposed method appears to heavily depend on the accuracy of the score modeling, raising concerns in scenarios where data distributions are highly complex or sparse.

The authors have adequately addressed the reviewers' concerns, leading all reviewers to raise their initial scores. The authors are advised to incorporate these changes into the final version of the paper.
Decision: Accept (poster)