FLR: Label-Mixture Regularization for Federated Learning with Noisy Labels
Authors: Taehyeon Kim
Donggyu Kim
Se-Young Yun
Venue: Accepted by TMLR ()

Abstract: Label noise in federated learning (FL) has garnered increasing attention due to the decentralized nature of FL, where data is collected from multiple clients with potentially different levels of label noise. This study introduces two pivotal contributions to this domain. First, we anatomize the memorization phenomenon in FL into server-side and client-side components, marking the first investigation into how these distinct forms of memorization impact learning. Second, to mitigate the memorization in FL, we present the Federated Label-mixture Regularization (FLR) strategy, a straightforward yet effective approach that employs regularization through pseudo labels generated by merging local and global model predictions. This method not only improves the accuracy of the global model in both i.i.d. and non-i.i.d. settings but also effectively counters the memorization of noisy labels. We empirically find that FLR aligns with and advances existing FL and noisy label mitigation methods over multiple datasets under various levels of data heterogeneity and label noise.

Published in Transactions on Machine Learning Research (11/2024)
FLR: Label-Mixture Regularization for Federated Learning with Noisy Labels
Taehyeon Kim
kimteekyenn100@@mal com
KAIST AI
Donggyu Kim*
custstar99779@mmail.com com
MediPicel
Se- Young Yun KAIST AI
yunseyoung@kaiis..ccc
Reviewed on OpenReview: https: // openreview. net/ formm? id= 28A3HDgS0E
Abstract
Label noise in federated learning (FL) has garnered increasing attention due to the de- centralized nature of FL, where data is collected from multiple clients with potentially different levels of label noise. This study introduces two pivotal contributions to this domain. First, anatomize the memorization phenomenon in FL into server-side and client-side components, marking the first investigation into how these distinct forms of memorization impact learning. Second, to mitigate the memorization in FL, we present the Federated Label-mixture Regularization (FLR) strategy, a straightforward yet effective approach that employs regularization through pscudo labels generated by merging local and global model predictions. This method not. only improves the accuracy of the global model both i.i.d. and non-i.i.d. settings but also effectively counters the memorization of noisy labels. We empirically find that FLR aligns with and advanccs existing FL and noisy label mitigation methods over multiple datasets under various levels of data heterogeneily and label noise.
1 Introduction
The advent of large-scale datasets has propelled deep neural networks to remarkable achievements in fields such computer vision, information retrieval, and language processing (Yao al., 2021). However, these dalasets often contain sensilive personal information, making conventional centralized learning impractical for applications such as person identification, financial services, and healthcare systems. Federated learning (FL) addressos privacy issues by allowing clients (c.g., edge devices) to collaborate with a central server (c.g., a service manager) without sharing their local data. Instead, clients update their local models using their private data, and the central server aggregates these updates to improve the global model. This procedure is repeated until convergence (FedAvg) (McMahan al., 2017). FL enables on-device learning and avoids systematic privacy risks at the data level, making it particularly suitable for the computing devices such phones, tablets, and autonomous driving systems (Kim & Yun, 2022; Kim et al., 2023).
In FL, multiple clients may have different levels of label noise (Xu et al., 2022) due to various factors, such as annotator skill, client bias, malfunctininn data collectors (Kim et al., 2018), or even malicious tampering with labels (Chen et al., 2020). For instance, in the healthcare industry, manually labeling patient records each hospital is susceptible to corruption due the complexity of medical terminology and the potential for annotator bias. Additionally, data entry mistakes or misunderstandings by medical professionals may introduce inconsistencies errors into the original data (Xu et al., 2019; Karimi et al., 2020). Training with noisy labels can negatively impact over-parancterized neural networks, as they are prone to memorizing
Work done while KAIST

---

Published in Transactions on Machine Learning Research (11/2024)
Forward pulh
Noisy label supervision
Regulatization
Pscudo label updatic
Global
Client k
Model in global-mode
Running Average Prediction
Label Mixture
Freeze
Model
Update
Local data
Neisy Label
Running Average Prodiction
Clein dat
Noiyy lateled dab
Figure 1: Overview of our proposed regularization, Federated Label-mixture Regularization (FLR).
mislabeled instances (a.k.a., memorization). Label noise, combined with FL's heterogeneous nature, exacerbate this phenomenon. Traditional methods for addressing label noise in centralized learning (CL) are not directly applicable in the FL setting due to privacy constraints, small client datasets, and difficulties in handling data heterogeneity (Chen et al., 2020; Xu et al., 2022; Ji et al., 2024; Tsouvalas et al., 2024).
In this work, we first identifies and addresses the dual nature of memorization in FL local memorization at the client level and global memorization the server level (Scction 4). We explore how local memorization, driven by client-specific data and biases, poses significant. risk of overfitting to noisy labels. Conversely, global memorization highlights the challenges faced by the central model in distilling accurate information from diverse, and potentially noisy, client updates. Our analysis reveals the intricate interplay between these two forms of memorization and their cumulative impact on the learning process in FL.
To combat these challenges, we introduce a simple yet efficient novel regularization for dealing with noisy labels in FL, termed as Federated Label-mixture Regularization (FLR; Figure 1), which supervises the noisy instances with pseudo labels generated through a combination of global server's running average predictions and local rumning average predictions. Our method is grounded in the recognition that memorization in FL manifests distinctively at server and client levels. In both i.i.d. and non-ii.d. sccnarios, the server's and client's 's temporal ensembling vectors each mitigale memorization issues. However, neither alone fully addresses both local and global memorization. This highlights the strength of our FLR method, which combines these vectors to effectively handle various noise conditions and data heterogencity. Through experimental validation, we ascertain that leveraging a mixture of both vectors culminates various noise conditions and data heterogeneity. Building upon this, we further develop several techniques by balancing the scale between global running average predictions and local running average predictions. In light of the above, our key contributions are follows:
robust label mixture adept at handling
We conduct exploratory experiments analyze memorization patterns FL, focusing on both client and server sides, to understand how label noise affects learning (Section 4).
We demonstrate that FLR effectively prevents local and global memorization and outperforms other methods on benchmark datasets with various levels of data hetcrogencity and label noise (Section 4, Section 5).
2 Related work
2.1 Learning with noisy label
Label noise in supervised learning can significantly reduce the generalization capability of deep neural networks (DNNs), leading to the development of varions methods for robust training (Song et al., 2022). Recent progress in this field has included designing robust architectures (Yao et al., 2018; Momeny al., 2021; Li al., 2021), loss functions, and regularization techniques (Pereyra et al., 2017: Zhang al., 2018; Liu et al., 2020; Wei et al., 2022; Kim et al., 2021b), well as methods for loss adjustment, (Reed et al., 2015; Ma et al., 2018;
2

---

Published in Transactions on Machine Learning Research (11/2024)
Zheng et al., 2020) and sample selection (Han et al, 2018; Li et al., 2020a; Kim et al., 2021a). However, most of them have been proven to work only under CL settings.
One of the major challenges in dealing with label noise is the DNN's ability to memorize unreliable data, inchuding random data or noisy labels, a.k.a., memorization phenomenon. This memorization phenomenon is observed to occur during the training phase, where DNNs first learn correctly labeled data and then gradually memorize the wrong labels (Zhang et al., 2017; 2021; Arpit et al., 2017). Bai et al. (2024) demonstrate be memorized in the early stage. To prevent the memorization
that such mislabeled examples even phenomenon, some approaches have focused on stopping training before memorization becomes severe (Song et al., 2019; Li et al., 2020b; Bai et al., 2021), and others have utilized regularization in the carly-learning phase (Liu et al., 2020; Xia et al., 2020). However, their effectiveness in the FL setting has yet to be fully explored.
2.2 Federated learning with noisy label
Federated learning with noisy labels (FNT.) is a scenario where the clients have label noise in their local datasets. This problem is particularly prevalent in FL, where data is decentralized and collected from multiple clients with varying levels of label noise. Training a model with noisy labels can be detrimental as peeepaarametei neural networks are prone to fitting the training dataset, leading to a memorization problem. Existing FNL methods address this issuc by using techniques such as sample selection, robust aggregation, label correction, and robust loss functions. For instance, FedCorr (Xu et al., 2022) corrects label noise in multiple stages, such as preprocessing, fine-tuning, and regular training, and FedRN (Kim al., 2022) robustly aggregates local models by weighting them based on the estimated label noise level. FedNoRo (Wu et al., 2023) integrates noise detection via Gaussian models and robust updates through knowledge distillation and distance-aware aggregation. However, some studies have limitations, such as relying on supervision (Chen et al., 2020; Tuor et al., 2021) or not analyzing results in non-i.i.d. settings and situations with heterogeneous noise levels (Jiang et al., 2022; Kim ct al., 2022; Ji et al., 2024; Zhou & Wang, 2024).
3 Preliminaries
The objective of FL is to solve the optimization problem for a distributed collection of heterogeneous data: min f(w) : Enes Pe Fiww)) where S is the set of total clients, with each client & has ny local training data samples, and n = Lies ng. The local objective of client k is to minimize (ww EayDD, (T I, Uk: w)], where & is the loss function parameterized by w on the local data (02, from local data distribution Da. Here, we consider setting having N clients and dataset with C classes. Fach client k has the training set (( yk, ) where Fs, is the i th input and yı, E (0, is the corresponding one-hot label vector; yk: = 1 iff æs. belongs to class c. The main issue is that it is unknown whether such labels are correctly annotated (i.c., clean) or not (i.c., noisy).
Data heterogeneity In order to consider the data heterogeneity in FL,, we assume the true data distribution Da prior to introducing label noise. In i.idd. setting, all clients have the same size local training set and an equal number of data samples per class. In contrast, the local data distributions in non-i.i.d. settings are more complex, with varying local training set sizes and imbalanced data samples per class. Following the settings in Xu et al. (2022), we create non-i.i.d scenario. We first sample Bernoulli random variable The ~ Bernoullipp) for each client & and class e, where Phc = 1 represents that client k has class e and PRe = otherwise. We then distribute the data samples of class e among the clients with & = using Latent Dirichlet Allocation (LDA), assigning the partition of the class e samples, where a is a vector of length Σm the with positive elements D To ensure that no clients are left. without data, we allocate at. least one sample of class e to each client with The = 1 regardless of qe-
Noise level We synthetically design scenarios with different levels of label noise among clients, following Xu et al. (2022). We first introduce the paramcter P. which represents the ratio of clients with label noise in their datasets. We then assign the noise rate of each noisy client based on 7, the lower bound of each client's noise rate. We sample the noise rate r% of noisy client k uniformly at random from the range (((T, 1).
3

---

Published in Transactions on Machine Learning Research (11/2024)
Correct
Wrong
Metnorieod
Cate
heay Data
Claa Label
bolay hart
tootio Sample.
Rouad
Round
Round
Round
Server memorization
(b) Client side memorization
Figure 2: (a) Server side memorization with LoE, and (b) client side memorization with Lee CIFAR-10 of the iid. setting, with symmetric noise of (p. (0.8,0.0 0.0). In (b), the fraction values are calculated by averaging the values contributed solely by participating noisy clients in each round.
Finally, we randomly select Tank data samples and reassign their labels to create noise. It worth noting that, while the original paper by FedCorr treated p and ra as probabilities (through Bernoulli sampling), our implementation treats them fixed ratios.
Noise type We examine two forms of label noise: symmetric and asymmetric noise (Patrini et al., 2017). For symmetric noise, we randomly and uniformly choose one class among the C' classes and reassign yk: as an i..d. random one-hot vector. As for asymmetric noise, it mimics human-like errors that happen for similar classes (e.g., cal ++ dog, truck + automobile). We reassign yku based on the class of Tki, the initial value of yks. for asymmetric noise.
4 Method
4.1 In the view of memorization in FL
In this subsection, we delve into the phenomenon of memorization FL. Despite numerous studies CL exploring memorization, the topic remains notably unexplored in the context of FL. To provide clear understanding, distinguish between two types of memorization by referring to the definition of memorization in the field of learning with noisy labels in CL (Liu et al., 2020):
Global memorization (server side) transpires after federated aggregation when the accuracy of the training dataset calculated using the server model.
Local memorization (client side) takes place immodiately after a local model update and refors to the accuracy computed on the training dataset using a personalized model that has been updated locally before federated aggregation occurs.
Figure 2 showcases both local and global memorization. In this context, estimate memorization instances where the model's predictions coincide with incorrect labels. We observe that the accuracy curves of correct samples in both clean and noisy label plots increase during the early stages. However, in the noisy label plot, the accuracy of correct. samples declines after reaching a certain point, while the level of memorization rises. In contrast, the clean label curve remains stable (Figure 2 (a)). In both cases, the count of proong-peedclad samples consistently decreases. Alongside examining the server-side, we also probe into the client-side memorization. Owing to the scarce amount of data available to each client, memorization becomes more prominent, regardless of whether the data is clean noisy (Figure 2 (b)). Specifically, the proportion of samples with incorrect labels memorized by the local model increases rapidly as the round progresses, eventually reaching 100%. This signifies that the local model has memorized all the noisy labels present in the client's local dataset.
Based on this observation, we define memorization from both local and global perspectives, capturing severity and prevalence, respectively. Catastrophic memorization occurs at the client level when model nearly
4

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1: FLR
INPUT Neural network M(-), server model parameter eerverr, randomly initialized parameter to. client k's model parameter 0k, global round Т, local training epoch E, learning rate 7) client k's dataset D ((,))) y1,)) balancing hyperparameters for FLR a, 8,7
O server t 80
// Initialization
/* Phase 1: Warmup FedAvg with standard CE loss */
2: for t + ......T--.. do
3: St + -AAMPECCIINTS
4: for each client & E St parallel do
0x 0 servvrr
for e 0. O..... E do
Pky: Pkig Mo, (a 1.... Mo (arking /
0g 0k 11 V no E 2o=1 yk: log Rog))!) (c)))
(c)
9: end for
// Client-Update with standard CE loss
10: end for
|D*| Oserver ΣXESt Exrest [DK 0x
11:
// Aggregation
12:
end for
/* Phase 2: FedAvg with FLR loss")
13: for t 0, 0....., T do
14: St + SAMPLECLIENTS
15:
for each client & St in parallel do
16:
0g O Beerver
17:
Pki
Phone
server Mo (Kki), Mo (Ikmm)
/1 Oserver is frozen
18:
for e 0, O..... .......
do
19:
Pki: Pkm: Mo (#ka) (on)).,,,M,..
20:
for 1,..., Thk do
21:
8k. t A8k, (1 3)pk. server
// Global running average mean // Local running average mean
22:
mi: t mm, + (1 - Pkk
23:
tk: 0sk;
-a)mk;;
// Mixture
24:
end for
25:
0g t 0% n (- 1 Ene Ee Uk. log (e) +
Eeh log (1 - (Pk. . .tt))) Client-Update with
26:
end for
27: end for
D
28: Seevver Evest
Lirest DD 0%
// Aggregation
29:
end for
memorizes its noisy training data, leading to suboptimal generalization and overfitting to incorrect labels. Conversely, common memorization refers to shared memorization across clients from the global perspective, emerging when numerous client models exhibit similar patterns of memorizing noisy labels, indicating a more pervasive issue affecting the overall FL process.
4.2 Global & local moving average label mixture
To combat these memorization issues, we introduce regularization term that penalizes the local prediction based the difference between the model's output and synthetic pseudo labels:
Re (pk. tt,))
C)))(() (0) +
log
T2k
(1)
C
(A) = Tik
ykv
(c)
log
Pk.
(c)

---

Published in Transactions on Machine Learning Research (11/2024)
Correct
Wrong
Menorized
Clean Label Data
Nakey Label
Label Data
Notay
Voccin of (f5rrge
Saund
Reaidd
92ued
Abund
(a) Server side memorization
(b) Client side memorization
Figure 3: (a) Server-side memorization with LFLR, and (b) client-side memorization with LELR under the same setting used in Figure 2.
Sk. 38h, + (1 P) p
server
Global
ma: mi, + (1 p)pk;
Local
tk t ask, + a)m
Mixture
where 0 denotes the parameters of the neural network, Pk, is the local prediction, 4, is the pseudo label vector for a client k's ith input, $k, is the global running average prediction, Pk is the global model's prediction, ma is a local running average prediction, and e the index of the vector, which satisfies with I c c C. The logarithm component serves as a regularization term that aligns p in the same direction tR.- Synthetic pseudo labels l,, are generated by merging local running average prediction with the server's running average prediction, called Federated Label-mixture Regularization (FLR). For initial prediction of a client k's i th input, Sk, and my are set to be equal to Pk, peruer and Pk. respectively. Since the server receives randomly augmented data during each local iteration, peevee undergoes slight changes. The number of updates to tx depends on the number of clients participating in each round. If a client does not participate, the most recent version 4, is stored locally, and this value is used for updates when the client is selected from the server. Algorithm 1 illustrates the overall procedure within the general FL framework.
The parameters a, 8, and y in Eq. (1) play critical roles in balancing the influence of global and local information. Here, controls the weighting between the global and local running averages, influencing the model's bias towards more generalized or personalized predictions. A higher a value emphasizes the global context, reducing overfitting to noisy local data but possibly underfitting local nuances. Conversely, & and y manage the decay rates of the global and local running averages, respectively. Specifically, 3 adjusts how quickly the global information is integrated into the model updates, while 7 dictales the retention of local data characteristics in the learning process. These hyperparameters are pivotal in optimizing the trade-off between memorization and generalization, ensuring that the model remains robust to label noise and data heterogeneity across different clients. Their values should be chosen based on empirical evaluations and the specific characteristics of the dataset, as thoroughly discussed in Subsection 4.3, which examines their impacts through a series of ablation studies.
Tackling confirmation bias The phenomenon of models overfitting to noisy labels, known as memorization, can be viewed as a form of confirmation bias, where incorrect pseudo-labeia predicted by the network are erroneously reinforced (Tarvainen & Valpola, 2017; Arazo al., 2020). FLR uses pseudo labels t, derived from blend of local and global running average predictions- to temper this bias. The local predictions capture client-specifie noise patterns, while the global predictions reflect the broader data distribution, together creating a comprehensive and resilient lcarning signal. The balancing parameter a plays pivotal role, meticulously adjusting the blend of local and global perspectives to ensure optimal noise resistance and overall performance. While local running average prediction alone can be effective in centralized settings, it may exacerbate local overfitting more swiftly FI. contexts due to data heterogeneity and data shortage. Our strategy of blending local and global predictions can thus be particularly advantagcous in such situations. More details in Subsection 4.3.

---

Published in Transactions on Machine Learning Research (11/2024)
Drrets
(a) Sym, CE
(b) Sym, FLR
(c) Asym, CE
(d) Asym, FLR
Figure 4: t-SNE mapping view of the penultimate layer's feature outputs. Colors represent the noisy label y and the ground truth label y".
Table 1:
Server side memorization and client side memorization at last round on CIFAR-10 of non-i.i.d. setting
(adir 1.0) with symmetric noise (p. r) = (1.0, 0.0) and iidd. setting with symmetric noise (p, = (0.8, 0.0). Vanilla denotes no use of the regularization in Eq. (1)-
Global memooiication (scrver-side]
Local memoriaatiin (client-side)
Heterogencity
Pseudo Label
Clean label
Noisy label
Clean label
Noisy label
Test Ace.
Correct
Wrong
Correct
Wrong
Memorized
Correct
Wrong
Correct
Wrong
Memorized
Vanilla
69.10
30.90
32 96
31.61
32.43
95.38
4.62
12 12/10
7.10
80.50
66.54
non-i.id.
Local Ouly
80.11
19.89
62.48
26.32
11.00
95.61
4.39
10.19
10.01
19.77
81.12
Global Ouly
11.31
20.72
82.08
Mixture (FLR)
88.01
11.96
70.36
21.31
8.13
97.61
2.36
71.58
13.03
13.39
83.04
Vanilla
90.92
9.08
16.57
11.16
72.26
99.96
0.04
0.00
0.00
100.00
67. 87
Local Ouly
94.88
5.12
80.34
5.46
14.19
99.55
0.45
21.22
0.88
77.90
83.88
Lid.
Only
96.38
77.93
7.50
99.27
0.73
39.17
85.82
Mixture (FLR)
95.70
4.30
79.30
9.35
11.35
98.24
1,76
81.96
2.07
15:97
87.07
Dissecting gradients The regularization term's logarithm in Eq. (1) works to balance out the exponential nature of the softmax function that is present in the probability of Pk. Similar to Liu et al. (2020), the gradient of our method is calculated as follows:
gradient of Los
gradicut the regularization
n
VLFLR(() = V.aa
Pk: yk
Agh.
)
(2)
(c)
(c)
Pk:
where gk.
(pkss
lk
Pks
1=1
where V.a., is the Jacobian matrix of the neural network W having parameters 6 for client k's i th input with respect to the 0. Suppose the uct class is c*. In the early training phase, the c" tt entry of 8k, is dominant. Setting a 2 0.5 results in a negative c* th entry gk. since the server memorizes slower than clients, preventing harmful memorization by noisy clients. Morcover, in the case of clean labels, the gradient of Log term Pk; yk. decreases quickly due to small dataset size, potentially leading to dominance by wrong labels in the gradient. The term Jk; mitigates this by maintaining the magnitude of coefficients on clean label examples. For mislabeled examples, the gradient of Lev term Pk:
(r) - y is positive, and adding the
negative term g( reduces the coefficients on these examples, minimizing their gradient impact. Beyond a convergence point, the server might not curtail local memorization due to overfitting. The term my; can be added this stage to counteract local memorization with local running average prodictions.
Warmup FedAvg (phasel) Our analysis of Figure 2 (a) reveals that the initial rounds, there is no significant server-level memorization. This observation justifies the application of warm-up phase in our
7

---

Published in Transactions on Machine Learning Research (11/2024)
methodology, aligning with the standard practices in previous LNL and FNL research (Jiang et al., 2022; Xu et al., 2022; Kim et al., 2022; Liu et al., 2020; Li et al., 2020a).
Experimental results Our empirical observations indicate that FLR effectively reduces both local and global memorization when compared to models trained using the general Fed method, which relies solely on cross-entropy loss. As demonstrated in Figure 3 (a), the learning curve is smoother for FLR, and the amount of memorization is considerably lower. In addition, the model trained with FLR accurately identifies the ground truth labels of noisy samples as the rounds progress, while maintaining the memorized ratio of incorrect labels below 10% (Figure 3 (b)). We employ warm-up training for regularization with server prediction during the first 50 epochs and switch to local running average prediction regularization thereafter. This transition causes the inflection point in the graph to appear shortly after warm-up completion in both Figure 3 (a) and (b). Although the accuracy for clean data experiences a slight decrease, the memorization of noisy data is significantly reduced.
Analysis according to data heterogeneity Table 1 presents the server-side and client-side memorization for both i.i.d. and non-i.idd settings when using local and global moving averages. In i.i.d. sottings, employing local moving averages alone effectively mitigates global memorization but not local memorization. In contrast, in non-i.i.d. scenarios, using the global moving average has a more significant impact on reducing memorization. This can be attributed to the fact that when clients have similar data distributions, local information is sufficient to prevent memorization in the global model. However, in non-i.i.d. situations, relying solely on local representation is inadequate for preventing global model memorization, making the use of the global moving average more advantageous. In both cases, utilizing a mixture of local and global moving average predictions as synthetic pseudo labels produced the best results, as shown in the table's ' Mixture rows. Although the degree of data heterogeneity may affect the outcome, a balanced mixture of global and local predictions scems to be more effective than adopting biased pseudo labels from either side exclusively.
Qualitative analysis Figure 4 provides t-SNE mappings of the penultimate layer's outputs, where each dot is visualized according to the predictions by the model trained with standard CE loss (Figure 4 (a) and (c)) and the model trained with FLR (Figure 4 (b) and (d)). The color corresponds to the ground truth labels. This visual analysis reveals that FLR significantly enhances cluster fidelity to reflect true class information of the samples. Figure 4 (a) and (c) depict the clustering under the CE loss, where color discrepancies indicate noisy label misclassifications. In contrast, Figure (b) and (d) illustrate substantial improvements under FLR; clusters are well-separated according to their true classes. Black dashed boxes highlight the visible transformations from the CE to the FLR model, illustrating the robustness of FLR in correcting misclassifiatiioss. For instance, the transition from scattered and inconsistently colored clusters in Figure 4 (a) to more cohesive and correctly labeled groups in Figure 4 (b) not only evidences a decrease in label noise but also reflects substantial qualitative enhancements in model performance. The t-SNE plots in Figure 4 provide an intuition regarding how FLR successfully addresses label noise.
4.3 Implementation
Figure 1 provides an illustration of how our method affects model updates. It is widely recognized that using a running average (i.e., temporal ensembling (Laine & Aila, 2017)) is more effective than relying on static model output. As a result, the baseline version of our method follows Eq. (1). We empirically investigate the effects of the components a, B,y (Figure 5). Insights for FLR's effective label noise handling in FL scttings include:
Influence of a and 7: We observe that higher values of a consistently yield better results, indicating a robust influence of global model predictions on local updates (Figure 5 (a)). An a setting of 0.9 acts as a potent regularization mechanism, akin to knowledge distillation, guiding local models with the refined, effective knowledge from the global model, thus bolstering system robustness as illustrated in Figure 2. At an a value of 1, the model fully embraces global predictions, cpitomizing a comprehensive knowledge distillation strategy. This provides profound insights into the extent of global model influence on local training dynamics. Meanwhile, a 7 setting between 0.5 and 0.7
8

---

Published in Transactions on Machine Learning Research (11/2024)
0.0
0.2
0.5
0.7
Accuracy (in alpha/gamma)
11.11
Ganima
0.5
11:43
10.31
12:48
64
9.50
Accuracy
8031
8014
01:93
31.27
48
01
42
38
36
Alpha
160
180
200
220
240
260
280
Epoch
(a) Heatmap w.t.t a,
(b) Learning curve w.r.t.
Figure 5: (a) Heatmaps with respect to o, at (p,T) (0.6, 0.5) on CIFAR-10, (b) learning curves according to the changes of B at (p,T) (1.0, 0.5) on CIFAR-10.
fine-tunes the update frequency, striking balance between historical accuracy and mitigation of local noise. This measured approach parallels careful learning rale adjust promoting gradual adaptation of local models to global insights and preventing destabilizing shifts.
Preventing global memorization with 8: In the presence of extreme noise, global memorization tends to occur in the latter half of the training process, It is apparent that higher 3 settings result in improved performance in highly noisy environments. This trend highlights the efficacy of increasing to mitigate global memorization risks by incorporating the moving average predictions from the global model (Figure (b)*).
Implications of a, B, We briefly discuss the implications of our methods:
Balances between local and global memorization. A lower leads the model to focus more local memorization, capluring specilic patterns unique lo each client, but risks overfitting. A higher shifts the emphasis towards global memorization, encouraging learning of universal patterns across the dataset.
8: Addresses global memorization in the server model, especially after several epochs, by employing temporal ensembling. This helps the model to maintain an overarching perspective on the learned patterns.
Aims to control local memorization. While the server model counters local memorization initially, y becomes essential when local memorization re-emerges at a certain convergence point. However, applying too early can interfere with the learning trajectory, thus strategically implemented after reaching certain level of learning.
To effectively tackle memorization, approach dynamically adjusts the influence of parameters my and 8k. throughout training. Initially, m&, is more influential to support early learning stages, but as training progresses, 8k, takes precedence for temporal ensembling to address memorization challenges. This dynamic is managed through a sophisticated scheduler:
i R
(()) -
so if R.
a(r) =ax R'
8(r)
[ else
else
"The selection of values from the 150th epoch in Figure (b) informed by empirical evidence showing shift in local memorization patterns around this stage decision hased proven efficacy in enhancing model performance. This strategic timing enables precise monitoring and reporting on the method's impact during critical learning phases, where interventions are crucial.
benchmarks. This period marks the initiation of method's application,

---

Published in Transactions on Machine Learning Research (11/2024)
Aggregate
Worst
85
Accuracies
75
65
SLR
FedProx Methods
FedLSR
FedRN
FedCor
Figure Test accuracies for each method, evaluated on CIFARION dataset (Wei et al., 2021). ELR. is implemented with FedAvg. 'Aggregate' and 'Worst' denote different noise type of CIFAR 10N dataset.
Fed/vg
FedPPox
FedDyn
20
90
90-
25
85
Accurii
80
60
CE
CE FLR
70
FLR
FLR
Asym Asym 0.8)
10
Aoym to 0.3.38
Asym 0.0)
(.4.0.9)
Figure 7: Collaborations with server-aware FL methods (FedProx (Li al., 2020c) and FedDyn (Durmus et al., 2021)) CIFAR-10 with iidd setting.
Here, denotes the total communication rounds, the current round, and Rn the warm-up period for B. We set 9,, typically to (2.0, 0.9, 0.7, 0.5) with Rn at 50, with further details provided in the Appendix.
The linear scheduler for delicately manages the balance between local and global learning. Initially, it enhances local learning to compensate for the global model's lower accuracy but gradually shifts to support global learning as the model's noise handling improves. 8 and start at zero to accommodate initial learning stages and gradually increase to harmonize the integration of local and global predictions as training advances.
For hyperparameter selection crucial to the effective deployment of our methodology, we recommend tailoring these parameters based on specific data characteristics and learning objectives. Comprehensive guidance on selecting and optimizing these parameters, along with practical application tips, detailed in the Appendix.
5 Experiments
5.1
Experimental setup
We evaluate our methods on standard benchmark datasets, CIFAR-10 and CIFAR-100 (Krizhevsky al., 2009), and two real-world datasets, CIFAR1ON (Wei et al., 2021) and Clothing (Xiao et al., 2015), with varying numbers of clients. We conduct experiments with 100 clients for CIFAR-10 and CIFARION, 50 clients for CIFAR-100, and 500 clients for Clothing We consider both i.idd. (CTFAR-10/100) and non-i.i.d. settings (CIFAR- 10/100, CIFARION, Clothing1M); non-iidness is parameterized by p and apir- As CIFAR-10/ 100 do not inherently contain label noise, we introduce synthetic label noise, where the level of noise parameterized by and т. CIFARION and Clothing naturally contain label noise, with data samples randomly distributed among clients. p, apir P: and 7 are claborated upon in Section To contrast between CL and FL. we also compare methods with conventional LNL methods in centralized settings,
10

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Average and standard deviation of the best test accuracy of trials for each method, evaluated on ClothingIM dataset (Xiao al., 2015) in federated scenarios with uniformly random distributed settings. FLR* denotes the accuracy of the FLR with FedCorr framework.
FedAvg
GCE
Co-teaching
ELR
FedProx
RoFL
FedLSR
FedRN
FedCorr
FLR
FLR
61.5712.85
67.67.10.10
67 69.3810.15
68.54.0.366
89.3110.11
69.7610.119
70.1110.14
Co-teaching
DivideMix
FedProx
FedCorr
FLR+
aoir 1
Cbr = 10
100
80
70
Clean
Sym(0.0,0.6)
Asym(0.4.0.6)
Clean
Sym(0.0,0.6)
(s,0(0.4,0.6)
Figure 8: Average of the best test accuracy of 5 trials for each method, on CIFAR-10 with non-i.i.d setting by varying the concentration parameter apiir under different noise levels. The x-axis indicates the noise level by changing the (r, p).
implemented following Xu et al. (2022). In the centralized scenario, we employ a dataset that has been corrupted using the exact same scheme in the federated scenario. Further implementation specifics are available in the Appendix.
5.2 Results on real world noisy datasets
Our experiments demonstrate that FLR surpasses traditional LNL methods utilized in FL. Notably, FLR's effectiveness is further amplified when combined with FodCorr, indicating robustness and adaptability across diverse FL settings. Figure 6 showcases FLR's significant advantage other label regularization techniques.
CIFAR1ON
Clothing1M FLR consistently outperforms other LNL methods. Although the accuracy differences among the methods are modest, FLR's performance stands out. This consistency in performance is particularly noteworthy considering the use of pre-trained model for the initial global model in these experiments. The results, detailed in Table 2, underscore FLR's capability to improve model accuracy in FL settings, even under conditions with pretrained models.
5.3 Results on CIFAR datasets
5.3.1
I.I.D. setting
Our comprehensive evaluation on CIFAR-100 under an i.i.d. setting (Table 3) showcases FLR's remarkable performance superiority across diverse label noise settings, including both symmetric and asymmetric noise scenarios. We observe that well-established CL methods like DivideMix (Li et al., 2020a), while robust in CL, do not seamlessly translate their effectiveness into the federated context. This finding resonates with prior literature on FL (FedCorr (Xu et al., 2022), FedRN (Kim et al., 2022)). Notably, Generalized Cross Entropy (GCE) (Zhang & Sabuncu, 2018) exhibits strong performance, particularly in high symmetric noise scenarios, yet FLR consistently outperforms it in most cases. Combining FLR with FedCorr (i.e., FLR*) yields synergistic effect, leading to the highest test accuracics across almost all noise settings, except for the noise-free case. This combination demonstrates FLR's versatility and potential for enhancing FL frameworks under diverse noise conditions.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table Average and standard deviation of the best test accuracy of 5 trials for each method, on CIF with i..d. setting and different noise levels.
Symmetric
Asymmetric
Setring
Method
0.0
T
0.4
p
p 0.8
p= 0.6
p 0.8
p 0.6
Centralized
CE
50.62:1142
46.30+1.88
43.0221108
32.92:11.20 1.26
55.7410.12
GCE
61.69:0.777
59.06:+0.92
57.25:10.58
18.91=1.32
62.18:+0.51
Co-teaching
12.68:1.22
18.45
36.38:0.55
27.18=00.81
11.90:11.33
DivideMix
69.17:0..3
6..76:1.224
64.91:10.11
54.9441.02
69.95-10.00
ELR
61.26+1.55
56.58+2.09
53.09+1.21
39.71=1.10
69.57+0.16
Federated
FedAvg
42.991 29
36.6811..2
34.53::1.12
25.48=0.80
51.79:1.32
GCE
48.51=1.37
43.1811.54
1..55
1..96=1.14
54.60:10.48
Co-teaching
44.74+1.22
38.81+1.75
36.55+00.83
27.61=0.97
53.30+0.82
DivideMix
55.10+1.20
52.23+0.61
18.88+1.34
43.57=0.81
55.50+0.29
ELR
51.02+1.58
13.1742..0
10.58+1..71
28.50=1.03
56.98-00.67
FedProx
40.61=1.47
34.36::1.95
32.71=0.99
24.04=0.88
49.11=:0668
RoFL
13.44=1.36
36.61#2.12
38.61#1.133
26.53=1007
51.83+00.55
FedLSR
60.51+1.21
15.6241.88
11.51+1.00
30.35=0.84
0.99
FedRN
43.10:1.900
36.58:22.34
38.46:11.88
24.32+1.000
53.0110.400
FedCorr
53.2341181
16.25:11.9
14.13
31.43=1544
59.6110.56
FLR
59.13#1.61
54.18+1.87
51.12#1.46
40.15=1.34
65.21+0.56
FLR
67.85+0.91
64.12
64.6440.38
54.10+1.01
69.52
Collaborations with server-aware FL methods We also delve into how FLR collaborates with server- aware FL methods. Our findings, presented in Figure 7, demonstrate that integrating FLR with these methods significantly boosts accuracy in various noisy client scenarios. This enhancement observed consistently across different. ratios of noisy clients, highlighting FLR's capability to complement server-aware strategies. This synorgy particularly crucial in foderated environments where client data is diverse and often unreliable, showcasing FLR's versatility and effectiveness in improving outcomes in complex FL settings.
5.3.2 Non-I.I.D. setting
Concentration parameter in Dirichlet distribution apir Figure 8 shows the results by changing the data heterogeneity ODir at p 1.0. Overall, it can be seen that the performance is lower when a 1 (local data heterogeneity 1) than when &Diir = 10 (local data heterogeneity J). It is confirmed that FLR consistently shows better performance at most noise levels and there seems more clear difference, especially when data heterogeneity becomes more severe.
6 Conclusion
This paper presents Federated Label-Mixture Regularization (FLR), a simple but effective method designed to tackle label noise in FI. environments. Our work is grounded two key contributions: a detailed exploration of memorization dynamics in FL, distinguishing between server-side and client-side memorization, and the development of FLR. to address these specific challenges. By creating novel pseudo labels, which are combination of global running average predictions and local running average predictions, FLR effectively mitigates these memorization issues. Through comprehensive experiments, FLR demonstrates marked improvement in global model accuracy across both i.id. and non-i.i.d. data settings. While it shows promising results, we acknowledge that FLR may not be optimally suited for all types label noise. Future research could delve into exploring complementary regularization techniques address various noise structures. The potential impact of FLR on real-world applications, especially in sectors like healthcare and financial services where data privacy is paramount, is significant.
12

---

Published in Transactions on Machine Learning Research (11/2024)
Acknowledgments
This work was partly supported by Institute for Information & communications Technology Promotion (ITTP) grant funded by the Korea government (MSIT) NN..RS-019--T110000 Artificial Intelligence Graduate School Program (KAIST), 10%, and No. 2022-0-00871, Development of Al Autonomy and Knowledge Enhancement for Al Agent Collaboration, 90%].
References
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias deep semi-supervised learning. In 2020 International Joint Conference on Newral Networks (IJCNN), pp. 1 IEEE, 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, el al. A closer look at memorization in deep networks. In International conference machine learning, pp. 242. PMLR, 2017.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34:24392 24403, 2021.
Yingbin Bai, Zhongyi Han, Erkun Yang, Jun Yu, Bo Han, Dadong Wang, and Tongliang Liu. Subclass- dominant label noise: A counterexample for the success of early stopping. Advances in Neural Information Processing Systems, 36, 2024.
Mare Bartholet, Tachyeon Kim, Ami Beuret, Yo-Young Yun, and Joachim M. Buhmann. Hypernctworkddrivnn model fusion for federated domain generalization, 2024. URL https: http:////ww..../////////000// wrgg/bas//402..09/74
Yiqiang Chen, Xiaodong Yang, Xin Qin, Han Yu, Biao Chen, and Zhiqi Shen. Focus: Dealing with label
quality disparity in federated learning. arXie preprint 2rXiv:2001. 11359, 2020.
Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. Federated learning based on dynamic regularization. In International Conference on Learning Representa- tions, 2021.
Shuhao Fu, Chulin Xie, Bo Li, and Qifeng Chen. Attack-resistant federated learning with besidual-based reweighting, arXiv preprint ar arxiv:1912. 11464, 2019.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in neural information processing systems, 17, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1125 1130. PMLR, 2017.
Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537, 2018.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arxiv preprint arXiv:1503. 02531, 2(7), 2015.
Xinyuan Ji, Zhaowei Zhu, Wei Xi, Olga Gadyatskaya, Zilong Song, Yong Cai, and Yang Liu. Fedfixer: Mitigating heterogeneous label noise in federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12830 12838, 2024.
Xuefeng Jiang, Shong Sun, Yuwei Wang, and Min Liu. Towards federated learning against noisy labels via local reuuaarggulriiiaiinn.. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 862 873, 2022.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Analysis, 65:101759, 2020.
Joongsoo Kim, Sihwan Kim, Namyeong Kwon, Hyohyeong Kang, Yongduk Kim, and Changhuhn Lee. Deep learning based automatic defect classification in through-silicon via process: Fa: Factory automation. 2018 29th Annual SEMI Advanced Semiconduclor Manufacluring Conference (ASMC), pp. 35 39. IEEE, 2018.
SangMook Kim, Wonyoung Shin, Soohyuk Jang, Hwanjun Song, and Se-Young Yun. Fedrn: Exploiting k-reliable neighbors towards robust federated learning. arXiv preprint (xXi::2205003100 2022.
Tachyeon Kim and Se- Young Yun. Supernet training for federated image classification under system heterogeneily. aiviv preprinl arXiv:2206.00 01366, 2022.
Tachyeon Kim, Jongwoo Ko, JinHwan Choi, Se Young Yun, et al. Fine samples for learning with noisy labels. Advances in Neural Information Processing Systems, 04:24137-24149, 2021a.
Taehyeon Kim, Jaehoon Oh, NakYil Kim, Sangwook Cho, and Se- Se- Young Yun. Comparing kullback-leible divergence and mean squared error loss in knowledge distillation. arXiv preprint ar 11x::1105.000 05.08919, 2021b.
Tachyeon Kim, Eric Lin, Jumu Lee, Christian Lau, and Vaikkunth Mugunthan. Navigating data heterogencity in federated learning: semi-supervised federated object detection. In Thirty-sewendh Conference on Neural Information Processing Systems, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning, In International Conference on Learning Representations, 2017.
Jingling Li, Mozhi Zhang, Keyulu Xu, John Dickerson, and Jimmy Ba. How does neural network's architecture impact its robustness to noisy labels? Advonces in Neural Information Processing Systems, 34: 9788-9803, 2021.
Junman Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020a.
Mingchen Lí, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. In International conference on artificial intelligence and slatistics, pp. 4313 4324. PMLR, 2020b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia. Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429-455 2020c.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez- Granda. Early- regular- ization prevents memorization of noisy labels. Advances in neural information processing systems, 33: 20331 20342, 2020.
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijewickrema, and James Bailey. Dimensionalliy-drivee learning with noisy labels. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3355 3364. PMLR, 10-15 Jul 2018.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp. 1273 1282. PMLR, 2017.
Mohammad Momeny, Ali Mohammad Latif, Mchdi Agha Sarram, Razich Sheikhpour, and Yu Dong Zhang. A noise robust convolutional neural network for image classification. Results in Engineering, 10:1 10:100225, 2021.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Jungwuk Park, Dong-Jun Han, Minseok Choi, and Jackyun Moon. Sageflow: Robust federated learning against both stragglers and adversaries. Advances in Neural Information Processing Systems, 14:840-851, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1952-,99,,111 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions, 2017.
Scott E Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In ICLR (Workshop), 2015.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. How does early stopping help generalization against label noise? arXiv preprint arXiv:1911.08059, 2019.
Hwanjum Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve superviee deep learning results. Advances in neural information processing systems, 30, 2017.
Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European Symposium Research in Computer Security, pp. 480-501. Springer, 2020.
Vasileios Tsouvalas, Aaqib Saced, Tanir Ozcelebi, and Nirvana Meratnia. Labeling chaos to learning harmony: Federated learning with noisy labels. ACM Transactions on Intelligent Systems and Technology, 15(2): 1 1-26, 2024.
Tiffany Tuor, Shiqiang Wang, Bong Jun Ko, Changchang Liu, and Kin K Loung. Overcoming noisy and irrelevant data in federated learning. In 2020 25th International Conference on Poltern Recognition (ICPR), pp. 5020-5027. IEEE, 2021.
Ching Pui Wan and Qifeng Chen. Robust federated learning with attack-adaptive aggregation. arXiv preprint arXiv:2102.05257, 2021.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: study using real-world human annotations. arXic preprint arXiv:2110.11 12088, 2021.
Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu. To smooth not? When label smoothing meets noisy labels. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23589 23614. PMLR, 23 Jul 2022.
Nannan WU, Li Yu, Xuefeng Jiang, Owang-Ting Cheng, and Zengqiang Yan. Fednoro: Towards noise- robust federated learning by addressing class imbalance and label noise heterogeneity. ar preprint arXiv:2305.0522 2023.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust larly-learning Hindering the memorization of noisy labels. In International conference on learning representations, 2020.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2015.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Jingyi Xu, Zihan Chen, Tony QS Quek, and Kai Fong Ernest Chong. Fedcorr: Multi-stage federated learning for label noise correction. In Proceedings of the TEEE/CVS Conference on Compuler Vision and Pattern Recognition, pp. 10184 10193, 2022.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. T. dmi: A novel information-theorriic loss function for training deep nets robust to label noise. Advances in neural information processing systems, 32, 2019.
Jiangchao Yao, Jiajie Wang, Ivor W Tsang, Ya Zhang, Jun Sun, Chengqi Zhang, and Rui Zhang. Deep learning from noisy image labels with quality embedding. IEEE Transactions on Image Processing, 28(4): 1909-1922, 2018.
Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under a structural causal model. Advances in Neural Information Processing Systems, 34:4409 4420, 2021.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commuuiicttion of the ACM, 64(3):107 115, 2021.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11447-11457. PMLR, 13-18 Jul 2020.
Xiaochen Zhou and Xudong Wang. Federated label-noise learning with local diversity product regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17141-17149, 2024.
16

---

Published in Transactions on Machine Learning Research (11/2024)
A Overview of Appendix
We present additional details, results, and experiments that are not included in the main paper.
B Ethics Statement
As we advance the field of Federated Learning (FL) with our novel Federated Label-mixture Regularization (FLR) method, it is crucial to consider its ethical implications. We address potential concerns in the areas of privacy, fairness, environmental impact, and potential misuse.
Privacy and data security FLR, like other FL methods, is designed to maintain the privacy of clients" data by performing computations locally and only sharing model updates. This approach inherently mitigates privacy concerns arising from the transmission of raw data. However, potential risks persist with the possibility of model inversion attacks other adversarial actions. Future work must continue to focus strengthening the robustness of FT. methods against such threats.
Fairness FLR could potentially exacerbate or mitigate existing fairness issues in FL depending on how it is deployed. On one hand, FLR used predominantly with data from certain demographics, it may introduce bias into model predictions. On the other hand, FLR's ability to handle noisy labels may allow for the inclusion of previously excluded data, promoting a more diverse and fair model.
Environmental impact FLR, like other FL methods, reduces the need for centralized data storage and computation, potentially lowering the carbon footprint associated with these processes. However, the energy consumption of local computation and communication for model updates must be carefully managed to ensure environmental sustainability.
Potential misuses While FLR is designed to improve the robustness of FL methods against noisy labels, it could potentially be misused to intentionally introduce bias or misinformation into models. For instance, malicious actors could leverage our method to facilitate their objectives by manipulating the noise in labels. Hence, it crucial develop safeguards against such misuse.
In conclusion, while FLR contributes positively to the FL field by offering a solution handle noisy labels, it is essential to apply it thoughtfully, considering its potential ethical implications. We encourage ongoing dialogue and scrutiny to ensure that its deployment aligns with principles of privacy, fairness, and social responsibility.
C Limitations
While our FLR method presents several noteworthy advancements in handling noisy labels in FL, it is crucial to acknowledge the limitations that accompany it:
1. Performance with highly imbalanced data: Although our FLR method demonstrates robustness in diverse data heterogeneity scenarios, its efficacy in situations with severe data imbalance across clients is an area that requires further investigation. Data imbalance here refers to the uneven distribution of data classes across different clients. For example, one client might have a majority of samples from a particular class while another client might have a very few samples of that class. This severe skewness in data distribution can lead biased learning, where the model performs well on classes with more samples but poorly on classes with fewer samples. Although FLR has mechanisms to mitigate the effects of data heterogeneity, it is not explicitly designed to handle extreme data imbalance.
Computational overhead: While the FLR method is effective improving the model's robustness against noisy labels, it also introduces additional computational overhead. This is primarily due to the calculation of local and global moving averages, which are central to the FLR method. The
17

---

Published in Transactions on Machine Learning Research (11/2024)
moving average calculations require additional computation and memory resources, which might not. be suitable for resource-constrainedl devices often participating in FI.. This could potentially limit the scalability and applicability of our method in real-world FL scenarios, particularly where computational resources are a constraint. Therefore, optimizing the computational efficiency of FLR without compromising its effectiveness against noisy labels is an important direction for future research.
3. Noise type sensitivity: FLR has been specifically developed to tackle the challenge of label noise in FL. This label noise can occur due to various reasons, such as human crrors during data labeling or miscommunicatioo during data transfer. However, in real-world scenarios, there are other lypes of noise that can equally affect the learning process. For instance, feature noise refers to inaccuracies or inconsistencies in the input data, which might be caused by faulty sensors or measurement errors. Model noise, on the other hand, is related to the discrepancies between the true data generating process and the model assumptions. Since our FLR method primarily focuses on label noise, it might. not exhibit the same level of efficiency when dealing with feature model noise. Future iterations of FLR could explore these areas to provide more comprehensive solution to noise handling in FL.
D Future directions
Despite the aforementioned limitations, the FLR method opens up several promising avenues for future roscarch:
1. Handling other noise types: Future work can look into extending the FLR method to handle other types of noise efficiently. This would make it a more comprehensive solution for real-world federated scenarios where different types of noise coexist.
Investigating and enhancing the performance of FLR with highly imbalanced data distribution would be valuable future direction. Techniques like adaptive resampling or cost-sensitive learning could be integrated with method to tackle this challenge.
Adaptation for imbalanced data:
3. Applications on natural language processing: Despite the emergent ability of Large Language Models (LLM), they also struggle with the privacy issues. Employing LLM training in federated scenario may meet noisy labels and texts with high probability, and thus our work can be a good solution in this area.
4. Optimization of computational efficiency: Future research could also focus on optimizing the computational efficiency of the FLR method. Reducing the computational overhead without compromising the robustness against noisy labels would make our method more practical for real-world FL scenarios.
Robustness against adversarial attacks: As the FL domain evolves, adversarial attacks pose an increasing threat to model robustness. Future work could explore how to bolster the FLR method (and FL methods, in general) to ensure robustness against adversarial attacks.
Domain generalization: Another major challenge of data heterogeneity in real-world FL applica- tion is the divergence of domain data, also known as domain shift (Bartholet al., 2024). Future work should also consider the robustness to domain shift adding the noisy labeled data.
By addressing these limitations and exploring these future directions, we continuously refine and evolve the FLR method to better serve the ever-growing demands of FL.
D.1 Adversarial clients in FL
Mcanwhile, there exist some studies (Fu et al., 2019; Tolpegin et al., 2020; Wan & Chen, 2021; Park et al., 2021) to combat adversarial attacks on federated systems. Among them, data poisoning attack might seem
18

---

Published in Transactions on Machine Learning Research (11/2024)
similar to FNL; few malicious clients aim to break the global model by sending local models trained with mislabeled data. However, FNL is distinct from it since every client can have some degree of label noises in FNL. From this perspective, we do not consider such works in our study.
E
Experimental settings
The common seltings are used in this paper for all baseline experiments on each datasets (Table 4).
Table 4: Datasets and the common settings. f indicates the pretrained architecture from ImageNet-Ie
Dataset
CIFAR-10 (CIFAR-ION) CIFAR-100
Clothing
# of train
50,000
50,000
1,000,000
# of test
10,000
10,000
10,526
# of classes
10
100
14
# of clients
100
50
500
Participation ratio
0.1
0.1
0.02
Model
ResNet-18
ResNet-34
ReaNet-50f
Learning
0.03
0.01
0.003
E.1 Heterogeneous data distribution settings on CIFAR-10
Figure 9 shows how partition the CIFAR-10 datasel among the client's local datasets.
E.2
Heterogeneous data distribution settings on CIFAR-100
Figure shows how we partition the CTFAR-100 dataset among the client's 's local datasets.
E.3
Noise transition matrix
Figure 11a and Figure 11b illustrate the practical transition matricos from our implementation, which show how labels are reassigned based on their ground truth labels.
E.4 Pseudocode for FLR implementation
Algorithm provides detailed procedure for implementing the Federated Label-mixture Regularization (FLR) method, strategy designed to tackle noisy labels within FL environment.
The algorithm is structured into two distinct phases. The first phase serves as a warm-up round, applying the standard Federated Averaging (FedAvg) method utilizing standard Cross Coos--Etrrpy (CE) loss. This phase forms baseline global model trained on noisy labels. The main steps are as follows:
Initialization:
The server model parameters, denoted as Oservers are initialized with randomly chosen parameters, dg.
A loop for each epoch initiates, where subset of clients S are selected for training. For every chosen client, they start with model whose parameters match those of the server model (0, ).ereer). The model parameters are updated through local gradient descent steps on their individual datasets, applying the standard CE loss.
Training iterations:
Aggregation: Lastly, the server gathers the client models into a global model by computing a weighted average. The weight for cach client model is proportional to their datasct size.
In the second phase, the FedAvg procedure is augmented with the proposed FLR loss. The steps are akin phase one, but with the following enhancements:
19

---

Published in Transactions on Machine Learning Research (11/2024)
Deer
Frog
Ship Truck
Automoblle
Dog
p-1.0, a-10
p-100, a-1
100
100
60
Clent
Client
400
500
600
600
800
1000
Class Distribution
Class Distribution
p=0.7, a=10
p=0.3, a=10
130
100
Client
0-
100
200
300
400
500
600
700
800
200
400
600
800
1000
1200
Class Distribution
Class Distribution
Figure 9: Heterogencous data distribution scttings CIFAR-10.
For each mini-batch in a client's dataset, both global and local rumning average predictions are computed. The global running average predictions (sk,) incorporate the server model's predictions, whereas the local running average predictions (mz,) utilize the predictions of the client's model.
Running average predictions:
Mixed labels: mixture label, tx,, is formed by combining both global and local running average predictions. This facilitates effective correction of noisy labels.
The client models are updated via gradient descent, applying the proposed FLR loss. The FLR loss consists of the standard CE loss and regularization term, which is based the distance between the current model's prediction and the mixture label.
Client update with FLR:
The delineated algorithm offers comprehensive blueprint for implementing FL in the presence of noisy labels, effectively handling data heterogeneity and varying label noise across different clients.
In addressing potential concerns regarding the computational demands posed by our FLR method, it is important to note that these are effectively managed through localized compulations at the client level coupled with strategic synchronization. These strategies are integral to FL frameworks and are designed to manage the computational load efficiently. By employing localized computations and selective synchronization, we ensure that the additional complexities introduced by FLR do not overburden the system. This approach not only imizes resource utilization but also confirms the practicality and feasibility of implementing FLR
20

---

Published in Transactions on Machine Learning Research (11/2024)
p=0.7, a=10
p=0.3, a=10
30
30
Client
200
400
600
800
1000
1200
200
400
600
800
1000
1200
1400
Class Distribution
Class Distribution
Figure 10: Data distributions of non-i..d. settings on CIFAR-100.
Symmeene, reb.C.pmd. pmó.6
Symnetric, t-0.0 ped
Symmetticc 1-000 p-0
Symmente 1m1.0, pedd pedal
Asymeitticc
4symmetr pw2.8
Agymmeer pe0.6
Aderinctii,
(a) Noise Iransition matrix for CIFAR-10.
(b) Noise transition malrix for CIFAR-100.
Figure 11: Noise transition matrices.
within the operational constraints typical of federated environments. By integrating these strategics, we maintain the method's efficiency, making a viable solution for FL scenarios characterized by noisy labels.
21

---

Published in Transactions on Machine Learning Research (11/2024)
E.5
Implementation details
Our implementation details for the methods as follows:
FedProx (Li et al., 2020c) requires coefficient for its proximal term. We chose μ 0.001 for our experiments.
FedCorr (Xu et al., 2022) has lot. of hyperparameters. Among them, the number of rounds in finetuning and usual FL stage set to be (300, 300) for CIFAR-10, (300, 400) for CIFAR-100, and (50, 50) for ClothingIM. We followed the values of original paper for the rest of hyperparametors.
GCE (Zhang & Sabuncu, 2018) requires two hyperparameters, 9 and We selected q 0.7, K 0.1 for CIFAR-10 and ClothingIM, and g 0.1, 0.3 for CIFAR-100 in experiments.
Co-teaching (Han al., 2018) uses two models. In our implementation of Co-teaching for FL, we maintain two global models wi and w2- When each round starts, both of them are given each client as the local models Wik and u22, and each client trains them with Co-teaching process. After the local updates done, the global models are updated by global aggregation respectively; {WIR} - W1 and {w2k) W2-
DivideMix (Li et al., 2020a) for FL is implemented in the same way as in the Co-teaching case. Regarding hyperparameters, we set 0.5 for the sharpening temperature, and 25 for the unsupervised loss coefficient.
Early Learning Regularization (ELR) (Liu et al., 2020) requires two coefficients À and | for its regularization term. After many trials of grid search, we found that À 4.0 with 3 = 0.5 for symmetric noise and 0.9 for asymmetric noise works better than any other hyperparameter values that we tested, in most of our experimental settings. We used X = 4.0 and 8 = 0.5 also for ClothingIM.
FLR require several coefficients X, a; 8, and 7, and has the option of using linear scheduling for a. (,)) is fixed to (0.9, 0.7, 0.5), and other settings are specified in Table 5.
FLR+ used am = (beta distribution parameter used for Mixup augmentation) when coupled with FedCorr framework, and aM for DivideMix.
For learning stratogy, we implement the training follows:
0
if 7()) else
00 if R.
a(r) aax B(r)
7())
B
7 else
where Rw denotes the epoch for warmup training, which is set to 50. For the methods developed in CL settings such as Mixup (Zhang et al., 2018), GCE (Zhang Sabuncu, 2018), Co-teaching (Han et al., 2018), DivideMix (Li et al., 2020a), and ELR (Liu al., 2020), we just follow the implementation of their original papers.
Building on the equation 1, other regularizations are formulated according to the uses of a, 8,
Early Learning Regularization (ELR) Liu et al. (2020): This method is a specific case of FLR where
= 0, 0 in federated settings. The significant advantage of ELR. is that it does not require additional storage for $k.-
Entropy Regularization (ER): scenarios where = = 0, the regularization aligns closely with the entropy minimization term Grandvalet & Bengio (2004).
22

---

Published in Transactions on Machine Learning Research (11/2024)
Table Impelmentation details of FLR for our experiments.
Dataset
Case
Symmetric noise
2.0
CIFAR-10 (CIFAR-1ON)
Asymmetric noise
3.0
Symmetric noise, w/ FedCorr
3.0
Asymmetric noise, w/ FedCorr
3.0
Symmetrie noise
2.5
CIFAR-100
Asymmetric noise
5.0
w/ Mixup
2.0
w FedCorr
5.0
2.0
Clothing M
w/ FedCorr
3.0
Server-aware Learning Regularization (SLR): This another special case of FLR, specifically when where local predictions are directly penalized by global predictions. Although similar to general distillation, SLR differs in its of the logarithmic function rather than the Kullback- Leibler function (Hinton et al., 2015). In FL, other methods like FedProx (Li et al., 2020c) and FedDyn (Durmus et al., 2021) improve generalization against data heterogeneity by reducing weight divergence between the local and global model. However, our approach is distinct in that directly mitigates divergence between predictions, rather than model parameters.
E.6 Comparison with other label regularization methods
Our adaptive regularization, especially ER, slightly overlaps with calibration methods, which have been extensively studied in CL. We compare ER with some confidence-related previous methods:
Entropy Minimization (EM) (Grandvalet & Bengio, 2004) requires one coefficient for its regularization term. We set the coefficient to be 4.
Label Smoothing (LS) (Pereyra et al., 2017) and Negalive Label Smoothing (NLS) Wei et al. (2022) requires one smoothing parameter. We set its value to be 0.1 for LS, and -0.05 for NLS.
Temperature Scaling (TS) (Guo et al., 2017) requires sharpening temperature. We set the value to be 4.0.
Entropy Regularization (ER): In scenarios where a = = 0, the regularization aligns closely with the entropy minimization term (Grandvalet Bengio, 2004).
Table shows the results of ER comparing with Entropy Minimization (EM) (Grandvalet & Bengio, 2004), Label Smoothing (LS) (Pereyra et al., 2017), Negative Label Smoothing (NLS) (Wei et al., 2022), and Temper- ature Scaling (TS) (Guo et al., 2017). It is observed that even simple entropy penalization for each instance with either EM or ER can make the model to be robust towards noisy labels.
F Gradients of the adaptive regularization
Lemma. (Gradient of the regularization). The gradient of the loss is defined in equation Similar to Liu al. (2020), denote the federated adaptive regularization by
R(0) log (1 (pk; tx.)))
(3)
23

---

Published in Transactions on Machine Learning Research (11/2024)
Table Comparison with other label regularization methods, on CIFAR-10 with iidd setting.
Symmetric
Asymmetric
Method
T 0.0
T 0.4
= 0.6
p p= 0.8
p= 0.6
p = 0.8
CE
74.58=1.00
67. 67.76+1.14
85.90 25
13.03333335
EM
86.14=0.44
82.90+11011
88.25=+0.00
86.17=0.60
LS
73.95=0.79
67.42+00.7
84.04=0.12
80.23=0.58
NLS
79.55+0087
71.36+00.64
68.86+00.62
66.87+0.87
TS
82.28-0.65
76.38
87.45=0.334
$4.97:0.63
ER
86.45+0.445
83.12=0.88
88.73-0.24
86.35+0.42
70
50
50
40
40
sir
sir
mixup
20
50
100
150
200
250
300
100
150
200
250
300
(a) LLD.
(b) Non L.I.D
Figure 12: (a) CIFAR-10 of the i.d. setting, with symmetric noise of (p. r) (1.0, 0.0), (b) CIFAR-10 of the non-i.idd setling, with symmetric noise of (p, T) 1..00.0).
The gradient of R is
VR(0) = V (Pk: txt))
(4)
(px. tx.)
It can be expressed with the probability estimate in terms of the softmax function and the deep-learning map- (f)))=(()e)) where the bolded numerator softmax probability vector. Substituting
ping (2),), Pk; = Σ. exp((N=()c)' and solving this, we get
VR(O) vNa(0)
(pk: 2 th: (pk. tx.) .P..) (2)
(Pi. (x,)
DE (t(rr) (()))( (r)
Pk.
(5)
(O)
1 (pk. -t.))
(C) D
(t()) 10) )p!
Pk.
G
Learning curves across different data heterogeneity
Building on the equation 1, other regularizations are formulated according to the uses a,, 8,
ELR (Liu al., 2020): This method is a specific case of FLR where a 0, > in federated settings.
The significant. advantage of ELR is that it. does not require additional storage for Sk;-
24

---

Published in Transactions on Machine Learning Research (11/2024)
SLR: This is another special case of FLR, specifically when 1, where local predictions are directly penalized by global predictions. Although similar to general distillation, SLR differs in its use of the logarithmic function rather than the Kullback-Leiberr function (Hinton et al., 2015). In FL, other methods like FedProx (Li et al., 2020c) and FodDyn (Durmus et al., 2021) improve generalization against data heterogeneity by reducing weight. divergence between the local and global model. However, our approach is distinct that directly mitigates divergence between predictions, rather than model parameters.
In this section, we present the learning curves over 300 epochs for four different strategies: ELR, SLR, Mixup, and FLR. The curves are plotted under both the i..d. and non-ii.d. settings.
Our observations indicate that both FLR and SLR generally perform well, with FLR consistently cxhibiting superior performance (Figure 12). This underlines the effectiveness our proposed FLR method combating the effects of noisy labels in the FL environment.
On the other side, the number of epochs increases, ELR and Mixup's performance tends to deteriorate. This is particularly evident in their generalization capabilitiess which decrease due to memorization. This memorization effect is particularly evident when these methods are exposed to prolonged training, which is a common situation a real-world scenario.
25


Reviews:
[{"review_id": "XT4ddITzw4", "reviewer": "Reviewer_mncB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper addresses a highly relevant and challenging problem in the realm of federated learning.\n- FLR is innovative in its approach to integrating both local and global model predictions to counteract label noise.\n- Extensive empirical validation is provided, showing the effectiveness of FLR across multiple datasets and scenarios.\n- The methodology is well-explained and the experiments are thorough, providing a clear demonstration of the benefits of the proposed method.\n\nWeaknesses:\n- In Algorithm 1, the parameter E appears to represent not the number of local training epochs, but rather the number of communication rounds between the client and the server. Furthermore, it is unclear whether E for FLR is twice as long as that for FedAvg, or if they are equivalent.\n- In Method, 1) To calculate the running average prediction, do clients use local data to train predictions for both the global and local models, subsequently averaging these predictions across all clients? If so, a new issue emerges: requiring clients to share their predictions introduces a data security concern. 2) At an alpha value of 1, the model fully embraces global predictions. However, the paper does not appear to discuss the experimental results in Figure 5 for this configuration. 3)The excessive number of parameters requiring tuning presents a clear disadvantage. Tuning multiple parameters not only elevates the model's complexity and risks overfitting, but also amplifies the experimental complexity and time investment. In addition, too many parameters to be optimized may limit the generalisability of the method.\n- In Experiments, 1) Why DivideMix conducts only three experiments, while other methods are five. 2) Compared to Table 2, Table 3 lacks FedLSR and FedRN for comparison. What is the reason for this omission?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new method, Federated Label-mixture Regularization (FLR), aimed at addressing the challenges posed by label noise in federated learning environments. The authors dissect the memorization phenomenon typically encountered in federated learning into server-side and client-side components, offering a novel perspective on how these distinct forms of memorization affect the learning process. The FLR strategy employs pseudo labels generated by merging local and global model predictions, which not only enhances the accuracy of the global model across different data distribution scenarios but also effectively counters the memorization of noisy labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Clarification of Parameter E:\n- Please provide a clear definition of the parameter E within Algorithm 1. Specify whether it represents the number of local training epochs or communication rounds between the client and the server. Additionally, clarify whether E for FLR is double that of FedAvg or equivalent.\n\nRunning Average Prediction:\n- Detail the procedure for calculating the running average prediction. Specifically, confirm if clients use local data to train predictions for both the global and local models and then average these predictions. \n\nDiscussion on Alpha Value Settings:\n- It is essential to include a discussion on the results obtained when the alpha value is set to 1, as mentioned in Method. The absence of this discussion in the results presented in Figure 5 leaves a critical gap in understanding the model's behavior under these conditions.\n\nExperiment Replication and Omissions:\n- Explain why DivideMix conducts only three experiments compared to the five conducted for other methods. \n- Address the omission of FedLSR and FedRN in Table 3, which are present in Table 2. Please provide a rationale for their exclusion or include these methods in revised experimental results to ensure a comprehensive comparative analysis.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lAN6cDlZfW", "reviewer": "Reviewer_B7Dt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Studying learning with nosiy labels in FL is reasonable and well motivated.\n2. The writing and overall presentation are clear and easy to follow.\n3. The exploration of memorization dynamics is interesting and well-motivated, particularly for algorithm design, such as the warmup phase with CE loss.\n4. The proposed FLR method is empirically effective across several datasets, outperforming multiple baseline methods.\n\n---\n\n## Weaknesses\n\n1. In Section 4.2 (Qualitative Analysis), the authors mention significant changes in color coding in Figure 4. However, the authors do not explain what the color coding represents, e.g., what colors represent noisy labels and ground truth labels?\n\n2. As the regularization method requires hyperparameter selection, the authors should provide details on how to select the hyperparameters. Additionally, what suggestions do the authors have for practitioners in selecting appropriate hyperparameters?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work empirically explores the memorization dynamics in Federated Learning (FL) from both client-side and server-side perspectives. It uncovers the phenomenon that memorization is prominent in both clean and noisy data settings. Motivated by these findings, the authors propose Federated Label-Mixture Regularization (FLR), which includes a warmup phase using FedAvg with cross-entropy (CE) loss followed by FedAvg with FLR loss. They conduct experiments on four datasets to verify the effectiveness of their method and provide detailed ablation studies and quantitative analysis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PUWOYVz08Q", "reviewer": "Reviewer_iN5g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nOne of the main strengths of this paper is the question it addresses, which is an interesting problem. The framework considered is fairly general, the experiments are numerous, and the proposed method seems also appropriate in situations where the data are not i.i.d. Finally, the concept of local and global memorization is intriguing and seems relevant in the context of federated learning.\n\n## Weaknesses\n\nAlthough Section 4.1 on the analysis of memorization in FL is fairly comprehensive, I find the rest of the presentation lacking in clarity. In particular, Section 4.2, which presents the method, lacks rigor and organization. More importantly, the proposed method is not clearly motivated, unlike its centralized analogue (Liu et al. 2020), e.g. regarding the interest of making convex combinations of the parameters in Eq. (1).\n\nAnother weakness concerns certain practical aspects of the method. FLR requires 4 hyperparameters to be set: $\\alpha,\\beta,\\gamma,\\lambda$ (+ warm-up phase duration).  Although there's a study of their impact in 4.3, the way in which these hyperparameters can be set in practice seems crucial to me and is currently missing. Another aspect concerns the algorithmic complexity and calculation times of the method. If I have understood correctly, at each iteration all possible predictions must be made, which seems computationally important to me, and it should be discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors address the problem of federated learning in the presence of potentially corrupted data. More specifically, they address the case of supervised classification where labels may be noisy and not correspond to the true classes. Their contributions are twofold. Firstly, they propose an empirical analysis aimed at understanding the memorization phenomena that occur in federated learning, a phenomenon that is fairly well understood in centralized learning, but not in federated learning. Their analysis reveals the presence of two levels of memorization, a local one at client level, and a global one at server level. Next, the authors introduce their method, Federated Label-mixture Regularization (FLR), which can be seen as an extension of the centralized method proposed by Liu et al. 2020, to the federated framework, taking both types of memorization into account. The quality of the proposed method is empirically demonstrated in a rather large number of experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In line with my previous comments, hereafter are other changes that I think would increase the paper's quality.\n\n- The importance of the hypermarameters and their role should be explained shortly after Eq. (1).\n- The proof of Eq. (2) should be made rigorous, in the manner of Liu et al. (2020, Lemma 2).\n- Typos in Algorithm 1 should be fixed: $u$ is sometimes replaced by $k$, $t$ by $e$, and conversely. \n- Hyperparameters $\\alpha,\\beta,\\gamma$ and $\\lambda$ should appear clearly as inputs of the algorithm. Moreover, the calculus of the prediction parameters $p_{k_i}$ should appear in the algorithm.\n- The experimental setup could be better described (including how to chose the hyperparameters)", "broader_impact_concerns": "No ethical implications other than those that appear in machine learning in general.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewer discussion after the author response focused primarily on hyperparameter selection.  The main points are present in the reviewer-author discussion.  On the balance, the submission does address hyperparameters in Sections 4.2 and 4.3 and Appendix E.5, and in my opinion therefore satisfies the review criteria for TMLR.
Decision: Accept (poster)