Federated Representation Learning via Maximal Coding Rate Reduction
Authors: Juan Cervino
Navid Naderializadeh
Alejandro Ribeiro
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Representation Learning
Information Theory

Abstract: We propose a federated methodology to learn low-dimensional representations from a dataset that is distributed among several clients. In particular, we move away from the commonly-used cross-entropy loss in federated learning, and seek to learn shared low-dimensional representations of the data in a decentralized manner via the principle of maximal coding rate reduction (MCR2). Our proposed method, which we refer to as FLOW, utilizes MCR2 as the objective of choice, hence resulting in representations that are both between-class discriminative and within-class compressible. We theoretically show that our distributed algorithm achieves a first-order stationary point. Moreover, we demonstrate, via numerical experiments, the utility of the learned low-dimensional representations.

Under review as a conference paper at ICLR 2023
FEDERATED REPRESENTATION LEARNING VIA MAXIMAL CODING RATE REDUCTION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose federated methodology to learn low-dimensional representations from a dataset that is distributed among several clients. In particular, we move away from the commonly-used cross-entropy loss in federated learning, and seek to learn shared low-dimensional representations of the data in a decentralized manner via the principle of maximal coding rate reduction (MCR²). Our pro- posed method, which we refer to as FLOW, utilizes MCR as the objective of choice, hence resulting in representations that are both between-elass discrimi- native and within-class compressible. We theoretically show that our distributed algorithm achieves a first-order stationary point. Moreover, we demonstrate, via numerical experiments, the utility of the learned low-dimensional representations.
INTRODUCTION
Federated Learning (FL) has become the tool of choice when seeking to learn from distributed data. As opposed centralized setting where data concentrated in single node, FL. allows datasets to be distributed among a set of clients. This subtle difference plays an important role in practice, where data collection has moved the edge (c.g., cellphones, cameras, sensors, etc.), and centraliz- ing all the available data might not be possible due to privacy constraints and hardware limitations. Moreover, under the FL paradigm, clients are required to train on their local datasets, which unlike the centralized setting, successfully exploits the existence of available computing resources at the edge (i.e., at each client).
The key challenges in FL. include dealing with (i) data imbalances between clients, (i) unreliable connections between the server and the clients, (iii) a large number of clients participating in the communication, and (iv) objective mismatch between clients. A vast amount of successful work has been done to deal with challenges (i), (ii), and (ii). However, the oficn-overlooked challenge of objective mismatch plays fundamental role in any distributed problem. For an client to participate in a collaborative training process (as opposed to training on its own private dataset), there must be motivation: each client should see itself improved by taking part in the collaboration. Recent work has shown that even in the case of convex losses, FL converges to a stationary point from a mismatched optimization problem. This implies that there cases where certain clients own the majority of the data (or even of certain classes), and see their individual performance curtailed by the collaborative approach.
When optimizing the average of the losses over the clients, the solution to the optimization problem generally differs from the solution of the individual per-client optimization problems. Objective mismatch becomes a particularly difficult problem in FL given the privacy limitations, which pre- vents the central server from curtailing this undesirable effect. Moreover, given that in standard FL, the central server possesses data, and that no proxies of data structures should be shared, cen- order to resolve the objective mismatch issue, several
tralized solution cannot implemented. approaches have been proposed. However, most such approaches rely obtaining more trustwor- thy gradients in the clients, at the expense either more communications rounds, or more expensive communications.
In this work, we propose an alternative representation learning based approach to resolve objective mismatch, where low-dimensional representations of the data are learned in distributed manner. We specifically bridge two seemingly disconnected fields, namely federated representation learning and rate distortion theory. We leverage the rate distortion theory to propose principled way of

---

Under review as a conference paper at ICLR 2023
optimizing the coding rate of the data between the clients, which does not require sharing data between clients, and can be implemented in the standard FL setting, i.e., by sharing the weights of the underlying backbone (i.e., feature extractor) parameterizations. Our approach is collaborative in that all clients are individually rewarded by participating in the common optimization objective, and follows the FL paradigm, in which only gradients of the objective function with respect the backbone parameters (or equivalently, the backbone parameters themselves) are shared between the clients and the central server.
Related Work. Several studies have been conducted in the context of FL to show the problem of objective mismatch, by proposing modifications in the FL algorithm (Yang al., 2219)) adding constraints to the optimization problem (Shen et al. [0021) or even including extra rounds of com- munication (Mitra al., 202T). As opposed to these methods, we propose to tackle the problem by introducing common loss that is in all clients' self-interest to minimize. Another line of re- search seeks to learn personalized FL. solutions by partitioning the set of learnable parameters into two parts, a common part, called the backbone, and personalized part, called the head, to be used for individual downstream tasks. Often referred to as personalized FL, this area of research is in- terested in learning models utilizing a common backbone that is collaboratively learned among all clients, while personalizing the head to each individual agent's 's task or data distribution Liingg et al. 2020): Collins et al. (2021); al. (202T): Chen & Chao (021): [Silva et al. (0022) Collins al. 2022: Chen et al. (002). We, on the other hand, are interested in learning representations in a principled and interpretable way, as opposed to converging to solution without any guarantees on its behavior. In the context of information theory, rate distortion theory has been used to provide theoretical (Altug al., 2013 Unal & Wagner 2017: Mahmood & WWagner] 2022) and empirical
(Ma et al., [007]] Wagner & 2021) results on the tradeoff between the compression rate of random variable and its reconstruction error. However, most such solutions are centralized.
Contributions. We summarize our key contributions as follows:
1. We introduce a theoretically-groundee federated representation learning objective, referred to as the maximal coding rate reduction (MCR²), that seeks to minimize the number of bits needed to compress random representations up to a bounded reconstruction error.
2. We demonstrate that obtaining low-dimensional representations using our proposed method, which we refer to as FLOW, entails an objective that is naturally collaborative, i.e., all clients have motivation to participate in the learning process.
2 BACKGROUND
2.1 FEDERATED LEARNING
Consider a federated learning (FL) setup with a central server and N clients. For any positive integer M, let [M] denote the set {1...., .M} containing the positive integers up to (and including) M. Each client n E [N] is assumed to host a local dataset of labeled samples, denoted by D,
{(=). yz [P where x. E RD and y," E [K]. Vi E [[P,.]] Vn E [N]. Focusing on a set of parameters θ we assume that the nh client intends to minimize a local objective, denoted by fa (D,;; 0), given its local dataset D,, In many cases, such as the cross-entropy loss (CE), this local objective can be decomposed as an empirical average of the per-sample losses, i.e.,
(n 0) = [Da Σ ((ba(x").v))
(1)
where he : -R + [K] is parameterized model that maps each input sample x its predicted label ho(x), and L: [K] x [K] + IR denotes a per-sample loss function.
The global objective in the FL setup is to find single set of parameters 0" that minimizes the average of the per-client objectives, i.c..
8" arg min lEe
Σ (D.::0).
(2)
2

---

Under review as a conference paper at ICLR 2023
It is assumed that the clients in a FL setup cannot share their local datasets with each other. This implies that the optimization problem in needs to solved in a distributed manner. To that end, assume that each client E [N] maintains local set of parameters 8 E θ over series of time steps E (7). Each client performs r number of local updates using stochastic gradient descent sent central server every 7 time steps, that the server
(SGD), and then the local parameters averages clients' parameters and broadcasts the resulting aggregated parameters to to the clients to replace their local models. More precisely, denoting the learning rate by 1), and letting Vo represent the stochastic gradient with respect the model parameters, the sequential parameter updates are given by
Vola(Dn 0?) mod J + 0,
0
(3)
This forms the basis of the FedAvg algorithm (McMahan et al. 2017).
2.1.1 PERSONALIZED FEDERATED LEARNING
Leveraging the representation learning paradigm (Bengio et al. [013 Oord et al. 2018 Chen et al, 2020), the parameterized model he RD + [ can be decomposed into components, namely 1) a backbone has TR. - Rdd, parameterized by a set of parameters @ E Ob, that maps each input sample I E RD to a low-dimensional representation z = h*(x) E R4, where we assume that < D. and ii) head h& I + [K]. parameterized by a set parameters zi E V, that maps the representation z E id to the predicted class (z)(z) = (hs = ha (z) E [K]. This implies that the set of end-to-end model parameters is given by 0 (Ø, v)), with the corresponding parameter space being decomposed as è x V.
Such decomposition procedure, while the training process for the head can be personalized and local for each client. In particular, for the Th client, assume that the local objective [n, 0) can be decomposed into objective on the backbone parameters, denoted by (Dn Φ). and separate objective on the head parameters, denoted by (D, ni v). where,
then be used to train shared backbone for all the clients using the FL
Dr, = , M2 [(h 21) P.l
(4)
i.e., the dataset D, with each input sample x, being replaced by its low-dimensional representation =!" = (=.)). Then, the global backbone objective would be variation of (2), where the end-to-end objectives are replaced by their backbone counterparts, i.e.,
arg min
N
12
Similarly to 3 in order to derive the optimal backbone parameters o' using SGD, the backbone parameters each client n E [N] can be sequentially updated as
1)
Vonn ni 27) if mod T + 0
(6)
1 07
O.W.
Once the optimal backbone parameters o" are derived, each client [N] can freeze its backbone and train its personalized head parameters the based on local dataset Dnnar, i.e.,
, arg min Sn (n (Dnna=;<)). NeW
2.2 CATE-DISTORTION THEORY AND MAXIMAL CODING RATE REDUCTION
Among the many ways to define the backbone objective fo (D: Φ to learn now-dimensional repre- sentations for given dataset D (see, e.g., ... (Chen et al.) 2020 Grill et al., 2020; Wang & Isola 2020 [bbontar et al. 2021 Bardes et al. 2021). Lezama et al. (2018)) the maximal coding rate reduction (or, MCR in short) has been recently proposed by Yu et al. (2020)
ceareticalll-groundde way
3

---

Under review as a conference paper at ICLR 2023
of training low-dimensional representations based on the rate-distortion theory (Cover & Thomas 2006).
Consider i..d. sequence [EM of M random variables following distribution p(2), Z and distortion function w x 2 - I. For given Q 2 0, the rate-distortion function is defined the infimum for which there exist encoding function Jenc Z + [2 and decoding function Idec [2 2M, such that
M M
lim M-+00
[wi
2,) S Ω,
(8)
M
2m1
where the sequence (εi)E((M) denotes the reconstruction of the original sequence (Z) at the decoder output, i.e.,
( = Gdec o Jenc ((z:)ett))
Intuitively, the rate-distortion function represents the minimum number of bits required to compress a given random variable, such that the decompressing error upper-bounded by a constant Ω.
In general, deriving the rate-distortion function is challenging, as it entails computing mutual infor- mation terms between the input sequence and the reconstructed sequence. However, for the case of finite-sample zero-mean multivariate Gaussian distribution with squared-error distortion measure, the rate-distortion function has a closed-forn solution. In particular, letting = [ž] Rdx denote the matrix containing a set M d-dimensional samples, for a squared-error distor- tion of c², the rate-distortion function is given by (M+d) log det. (T + - 72") where / denotes the x d identity matrix (Ma et al. (007). Quite interestingly. the rate-distortion function, when normalized by the number of samples, can be viewed as measure of compactness of the given samples in R2d Assuming M >> d, this leads to the coding rate R(Z, e), defined as
2M] E
1 R(Z,,) == log det (1+ d
Mr² 77%
(10)
The coding rate in (10) can be leveraged in a representation learning setup, where z;'s 's are the repre- sentations produced by the backbone h2s For representations to be useful, the representations within one class should be as compact as possible, whereas the entire set of representations should be as diverse as possible. For a given class E [K]. let II; E IR X be a diagonal binary matrix, whose 7t diagonal element is 1 if and only if the 2th samples belongs to class k. Then, the average per-class coding rate given the partitioning = (IT, ]e[K]] can be written as
Rd R7, [II = 2M
Σ tr((Te) log det (I+ tr([I ) 2T KE[K]
(11)
where tr(-) represents the trace operation.
The principle of maximal coding rate reduction (MCR²) proposed by Yu et al. 2020) defines the backbone objective (2; σ) the difference between the average per-class coding rate R RZ, III in ( and the average coding rate over the entire dataset, R(Z,e) E) in (TO). More precisely,
(D; ()) (Z e(T) R(Z(D))),),
(12)
where the dependence of the representations Z on the dataset D and the set of backbone parameters is explicitly shown.
PROPOSED METHOD
Learning low-dimensional representation be posed collaborative objective, where each client in the network benefits from the collaboration. In federated learning, the dataset D is dis- tributed among of clients, i.e., D Dhe[M]D,,, where D,, is the dataset located the 1h client.
Since the 2CR backbone objective in 122 is monotonically decreasing with scaling the representations 7, in practice, the representations need to be constrained, e.g., to the unit hypersphere go-1, the Frobenius of per-class representations should bounded by the number of per-class samples.
4

---

Under review as a conference paper at ICLR 2023
We leverage the MCR² principle to introduce the global objective of our proposed FL method, which we refer to as Federated Now-Dimensional Representation Learning, or FLOW, as follows,
(I+ I+
min fp(D;o) = 2M
Σ kE[K]
log det
(T+
d
Σ
Σ
nE[N] mED.nMa
2 log det
(1
1+
d Σ nE[N] mED,
Σ
(13)
M
where for a given class k E Mr denotes the set of samples that belong to the kihh class. Note that in [3)., we have made the dependency of the objective function on explicit, that is 2m = ho (Fm)
worth noting that the objectives (D; Φ) in (12) and (3) are equivalent, as
=
zas] = (11(
ha(xMM) and Zz
Σ ME[M]
Z' Fmm
(14)
and the partition matrix II, has its mhth diagonal element equal to one and only if the m belongs to MM. Therefore, learning low-diment www-dimessiona representations in a distributed manner is equivalent to solving (1).
Note that opposed to common FL. implementations, our approach optimizes common objective, as opposed to a summation over different objectives. However, this comes at a cost; the objective in (13) is not separable, i.e., it does not immediately follow that each client can take local gradient descent steps. In what follows, we will demonstrate interesting properties of problem (13), namely (i) that it is in each client's 's self interest to obtain a collaborative solution, and (ii) that solution to problem (13) can be found in a distributed manner without clients needing to share their local datasets with each other.
3.1 MOTIVATION
Learning low-dimensional representations is collaborative objective, and it is in each client's 's self interest to obtain better representation. The choice of maximizing the coding rate reduction is well motivated by properties of the solution of problem (I3), as can be shown in the following theorem. Theorem 1. Consider set of dimensions {d& k= such that with rank rank(Zi) S dg. tfe embedding space is large enough, i.e., d 2 Ek= dy and the coding precision is high enough, i.e. e²
MMldd
minke[k] Mdj
then:
The optimal subspaces associated with each class are orthogonal even from data across clients, iee,, ha- (Fm))² has (Imm) = 0 for any m MR. in E M, with & k; and,
* Each class subspace 7% =x=xa =EEE. hag (Im)has (Im)² achieves its maximal dimension rank(Z?) [Ma), and the largest [MR] - singular values of Z% are equal.
Proof. The proof follows from Yuu et al. 2020, Theorem 2.1) noting that problem (3 is equivalent optimizing the centralized objective [2]. A similar proof can also be found in (Chan al. 2022 Theorem 1).
Theorem Tis important because it shows that the benefits of our method are two-fold: (i) the solution of the problem is orthogonal between classes, even from data coming from different clients, and (ii) the obtained representations for each class are maximally diverse. Theorem D is notable given that are not sharing data between clients, and are still able learn representations that orthogonal between classes. That is to say, if two samples x E RD and x' E IR belong to different classes, their corresponding low-dimensional representations and z' will be orthogonal regardless of which client owns the datum. What is more, the subspace associated with class j is maximal across clients, which translates into having a rich and diverse representation, even in low dimensions.
Note that clients were to solve the problem individually, there would be two undesirable proper- ties. First, even if the representations of samples of different classes for given client are orthogonal,
5

---

Under review as a conference paper at ICLR 2023
Algorithm FLOW: Federated LOW Dimensional Representation Learning
1: Set coding precision e, step size 7, embedding space dimensionality d, aggregation period
2: Initialize backbone parameters #n-
3: for round 1 to T do
4: if mod с * 0 then
5: Client does: Update model locally,
0 de
with Sa given in 12
6:
else
7: Server does: Average models: @t+1 = N Σ ot.
8: end if
9:
end for
that orthogonality might be violated when we move across clients, since there is no guarantee that per-class subspaces are aligned across clients. Therefore, having a common representation is desir- able property as will enforce orthogonality between samples that do not co-exist at the same client. Second, the fact that the class subspace achieves its maximal dimension makes the representations more diverse, grouping similar samples together. Again, this property is desirable, and collaborating between clients is in each client's best interest. Note that these properties properties of central- ized approach Yu et al. (2020), which our proposed method inherits and maintains in the distributed setting.
3.2 ALGORITHM CONSTRUCTION
The optimization problem in (13) is non-separable between clients, that to say, the global objective equal summation, or an average, individual objectives. Given that obtaining closed- form solution of ø cannot be done in practice, we turn into an iterative SGD-based procedure. In short, each round each client receives the current state the model ø', and utilizes own data to maximize its own MCR2 loss, as follows,
0i+ -. - Vol
(15)
with 7) being a non-negative step size. Every T rounds, the clients communicates their backbone parameters back to the central server. The central server's job is to average the received backbone parameters. Notice that these framework has two advantages: (i) clients do not need to share any of their private data, (ii) the computing is done at the edge, the clients. Moreover, averaging the models between the clients can be done utilizing Homomorphic Encryption (HE), preventing the central client from revealing clients' gradient information. An overview of proposed method can be found in Algorithm
3.3 CONVERGENCE OF FLOW
In this section we analyze the convergence of FLOW (cf. Algorithm D To do so, require the following assumptions,
Assumption 1. The MCR² loss is G-smooth with respect to the parameters o, i.e.,
[V: (D.o(D. Gll Ó2l1.
(16)
Assumption I is a standard assumption for learning problems. What this assumption implies is smoothness on the gradient of the function with respect to the parameters 6. In the case of neural networks as the parameterization, this a mild assumption, given the continuity of the non-linearity and its linear filters.
Theorem 2. Consider the iterates generated by Algorithm 7 Under Assumption 7 if the client gradients are homogeneous unbiased estimates of af (D: Φ), i.e. Ep, Vofo (D.: = V Vola (D (D; Φ), and the variance of the estimates of the gradients bounded, ie. (ll

---

Under review as a conference paper at ICLR 2023
Vofo(D:o)1 o², then
T
G [D)
a² fo Fe (4
(17)
T
2N
with S L/L
Proof. See Appendix A
datasets Dn are composed of samples that are sufficiently similar, individual gradients taken at each client can be modeled unbiased gradients of the gradients taken over the whole dataset, i.e., [ oVoJo o)] = V f (DD Φ). Theorem provides a standard convergence result for the case of non-convex loss, which indicates that the summation of the norm of the gradient square does not diverge. The convergence of the summation implies that the norm of the gradient in fact decreasing, which means that the iterates of the algorithm approaching a first order stationary point.
We also provide proof of convergence of our algorithm in the in which the distributions are not uniform in the clients.
Theorem 3. Consider the iterates generated by Algorithm 7 Under Assumption 7 if the client gradients are biased estimate of of; Φ), i.e. V. (Dni o) (D; o) + pns with I| V (D((D; e) S 6, and [E[| (D; Φ) (D 1 52 0², then
1
T
C - -o(l;:o)-((Dior))))
o²
(D)
+
- 8,
(18)
T
T
2N
with S L/L.
Proof. See Appendix B
Theorem provides a convergence result of Algorithm | in the case of non-uniform clients. We model the non-uniformity of the client distributions by introducing a jtn discrepancy vector for each client n. Notice that the key difference between Theorems 2 and 3 is the presence of ó, which is a bound on the maximum norm of the discrepancy between the gradients. The consequence of such dissimilarity is mild, as we can still obtain a convergent sequence.
4 EXPERIMENTS
We Algorithm I in two federated learning settings, with N = =5:0, 50, and with N 100 agents, in both cases, we run full participation, i.e. all agents were part of the communication rounds. For the dataset, we utilized CIFAR 10. and for the parameterization, ResNet18. The low dimensional representation has dimension d = 128. To model the agent mismatch, we distributed the samples per class according to Dirichlet distribution prior with a literature (Shen et al. (2021): Hsu et al. (2019): Acar et al. (202T). In all cases we run for 500 epochs, with a learning rate of 0.3, we utilized a batch size of 500 samples, and we run local epochs per agent.
this distribution is widely used in the
4.1 LEARNING CURVES
In figure we plot the learning curves for the MCR2, well as the R loss, and the RC loss. It can be seen that in all cases, the centralized MCR paraetttettirizaatoo outperforms the Federated learning case. This is expected,
distributing the datasets tends to have negative effect on performance.
The number of agents also affects the loss, as the parameterization able to get a better performance on N 50 than on N 100. This has to do with the unbiasness of the local gradients, that as the number clients increases, so does the bias term. In all, figure shows that the MCR"loss can be learned in distributed manner.

---

Under review as a conference paper at ICLR 2023
Apenic
Sprics
-10
20
10
Agrrc
410
250
130
(a) Joss
(b) Discriminative loss RC
Compgesssvve Joss Rt
Figure 1: Learning curves for MCR'in Federated and Centralized settings for CIFAR-10.
1.0
1.0
10000
20000
0.6
20000
0.6
30000
40000
40000
0.2
0.2
Contratized MCX
(b)
Eatropy:
10000
20000
30000
40000
10000
20000
30000
40000
0.8
0.8
10000
10000
30000
0.4
30000
0.4
40000
40000
0.0
0.0
(c) Federated Leaming with N
(d) Federated Leurning with N 100
Figure 2: Orthogonality of the low dimensional representation.
8

---

Under review as a conference paper at ICLR 2023
(a) Centraliaed.
(b) Federated Leuming 50
(c) Federated Learnngg 100
Figure 3: Decreasing order of magnitude of singular values of the subspaces associated with each class.
4.2 ORTHOGONALITY OF REPRESENTATIONS
Figure 2 shows the cosine similarities between all the elements of the dataset. Upon training, we obtained the low dimensional representation of cach sample, and computed the pairwise cosine ordered that the first 10000 samples
correlation between them. In order to plot the samples, belong to the first class and so on so forth. As expected by Theorem [ samples different classes tend to be orthogonal between themselves, and samples of the same class maximally diverse. Consistently with the worse value of the loss observed in Figure [ we can visually verify that the orthogonality between samples is worse the number of clients increases. Nevertheless, for the most part, we are able to obtain an orthogonal representation for the samples. This, is as expected by Theorem[[]] 3 As opposed to the centralized case, in our federated learning procedure, samples of different agents are never shared, which adds merit to Figure 2 The value of using the MC a loss is seen when compared the representations learned with the cross entropy loss. To obtain this representation, we train centralized architecture (i.e. ResNet 18) with 128 features before the fully connected layer. Figure 2 shows that learning orthogonal representations is not obtained unless enforced. Moreover, the block diagonal elements of the cross entropy matrix darker, which means that the numbers are closer 1. This comes to no surprise, as the sole objective the cross-entropy loss is to separate samples of different classes. However, the MCR also seeks for diverse representations, allowing samples of the same class to have different alignments.
Finally, Figure shows the distribution of the cigenvalues of the per-class matrices Z Z,ZZi or the singular values Zμ for different classes in centralized and federated cases. Again, we see that proposed approach lead to similar distributions of the principal components of the learned representation subspaces, where each class ends occupying low-dimensional subspace, even though each client does have direct access the data samples hosted by other clients.
CONCLUSION
In this paper introduced principled procedure to learn low-dimensional representations in distributed manner. In the context of Federated Learning, we introduce collaborative loss based on the maximal coding rate reduction (MCR²), which individually benefits all the agents in a self interested way. We refer to our federated fow-dimensional representation learning algorithm by FLOW. Theoretically, we show that (i) the solution of FLOW generated orthogonal representations for samples of different classes, and maximizes the dimension of each class subspace, and (ii) that under mild conditions, FLOW converges to first order stationary point. Empirically, we compare our method to the centralized procedure, validating all the claims that put forward.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. arXiv preprint arXiv:2111.04263, 2021.
Yücel Altug, Aaron B Wagner, and Ioannis Kontoyiannis. Lossless compression with moderate eπror probability. In 2013 IEEE International Symposium on Information Theory, pp. 1744-1748. IEEE, 2013.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varionce-ivarrnne---vviineee for self--uper learning. arXiv preprint 1rXiv:2105.04906, 2021.
haw.a//aaaaaaaoooooooooonnaariia regularization
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Kwan Ho Ryan Chan, Yaodong Yu, Chong You, Haozhi Qi, John Wright, and Yi Ma. Redunet: A white-box deep network from the principle of maximizing rate reduction. Journal of Machine Learning Research, 23(114):1-103, 2022.
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. In International Conference on Learning Representations, 2021.
Huili Chen, Jie Ding, Eric Tramel, Shuang Wu, Anit Kumar Sahu, Salman Avestimehr, and Tao Zhang. Actperfl: Active personalized federated learning. In Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022), pp. 1-5, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared repre- sentations for personalized federated learning. In International Conference on Machine Learning, pp. 2089-2099. PMLR, 2021.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Fedavg with fine tuning: Local updates lead to representation learning. arXiv preprint arXiv:2205. 13692, 2022.
Thomas M. Cover and Joy A. Thomas. Elements of information theory (2. ed.). Wiley, 2006.
Caan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21283 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non- non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06333 2019.
José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- plug and play geometric loss for deep learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8118, 2018.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach, David Brent, Ruslan Salakhutdinov, and Louis- Phils-ph Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523, 2020.
Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 2969):15461156. 2007.
Adeel Mahmood and Aaron B Wagner. Lossy compression with universal distortion. In 2022 IEEE International Symposium Information Theory (ISIT), pp. 596-601. IEEE, 2022.
10

---

Under review as a conference paper at ICLR 2023
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication effficient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1773-1282. PMLR, 2017.
Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani. Linear convergence in federated learning: Tackling client heterogeneity and sparse gradients. Advances in Neural Information Processing Systems, 11114666616119 2021.
Nesterov. Introductory Lectures on Convex Optimization: Basic Course, volume 87. Springer Science & Business Media, 2013.
Jaehoon Oh, SangMook Kim, and Se- Youy Yun. Fedbabu: Toward enhanced representation for federated image classification. In International Conference on Learning Representations, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.00 00748, 2018.
Zebang Shen, Juan Cervino, Hamed Hassani, and Alejandro Ribeiro. An agnostic approach to fed- erated learning with class imbalance. In International Conference on Learning Representations, 2021.
Andrew Silva, Katherine Metcalf, Nicholas Apostoloff, and Barry-John Theobald. Fedembed: Per- sonalized private federated learning. arXiv preprint 000ii::2202.09422,2222 2022.
Sinem Unal and Aaron B Wagner. Vector gaussian rate-distortion with variable side information.
IEEE Transactions on Information Theory, 6365)-116,-5178, 2017.
Aaron B Wagner and Johannes Ballé. Neural networks optimally compress the sawbridge. In 2021 Data Compression Conference (DCC), pp. 143-152. IEEE, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In International Conference Machine Learning, pp- 9929-9939. PMLR, 2020.
Qiang Yang. Yang Liu, Yong Cheng, Yan Kang, Tianjian Chen, and Han Yu. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3):1-207. 2019.
Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. Advances in Neural Information Processing Systems, 33:9422-9434, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, pp. 12310- 12320. PMLR, 2021.


Reviews:
[{"review_id": "KWfxClqW08A", "reviewer": "Reviewer_LtxX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed to replace the cross-entropy loss with the MCR2 loss in federated learning. They provided a theoretical analysis of learning convergence of the global model using the MCR2 loss. They also empirically show the convergence of the loss and the orthogonality of the class-wise subspaces learnt. ", "summary_of_the_review": "The paper makes a rather simple change to the standard federated learning setting by using a different loss function. The empirical evaluation does not show the advantage of performing that change.", "strength_and_weaknesses": "Strength:\n\nThe idea of separating the backdone network from the project head is an interesting idea.\n\nWeakness:\n\nThe idea proposed seems rather trivial. And the empirical evaluation is not done in comparison with some natural baselines. It is difficult to see the significance of the proposed work. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "AvVsawhCeF", "reviewer": "Reviewer_bnnY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an algorithm to learn low-dimensional representations in the Federated Learning setting. The paper borrows idea from a recent work which formulated a loss that could maximize the code rate difference between the entire dataset and summation of individual classes. The work takes a further step by identifying that learning a low-dimensional representation could be a collaborative objective and introduces an algorithm to solve the learning objective in a distributed manner.\n", "summary_of_the_review": "This is an well-written paper proposing an interesting extension for the MCR loss in the federated learning setting. The experimental parts are a bit weak and the overall novelty is also limited.\n", "strength_and_weaknesses": "Strength: The work is overall well-written and easy to read, all the details of the cocepts, setups are clearly stated. The theoretical results demonstrate that in the distributive setting,  the learning objective could still be successfully solved.\n\nWeakness: The novelty of the work is quite limited, although the usage of the MCR loss in federated learning is new, both the loss and the algorithm design are not new. Besides, the experimental part is quite weak as it only demonstrates the results of the MCR loss under Federated Learning setting, no comparison has been done to show the supremacy of the proposed algorithm.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9Dz30luBKy", "reviewer": "Reviewer_fuw1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the federated learning problem and proposes to\nminimize the difference between two log determinant functions instead of\nthe cross-entropic loss.\n\n", "summary_of_the_review": "As detailed above, the paper is clearly below the bar of ICLR in several\naspects. In its current form, the paper is better suited for some less\nselective conferences.", "strength_and_weaknesses": "# Weakness\n1. The work is not solid. My\n   judgment is based on the following:\n   - The background section is unnecessarily long. For example, the\n     review of Section 2.2 is redundant, and the paper can simply say\n     that, as motivated by some rate-distortion theory, Yu et al.\n     (2020) proposed this MCR2 objective.\n   - Theorem 1 is basically a copy-and-paste from a follow-up work\n     (Ryan et al., 2022) of Yu et al. (2020), called ReduNet.\n     Moreover, the copy is wrong: It misses the key assumption that the\n     objective admits a low-rank solution, see Theorem 12 of Ryan et al.\n     (2022). Finally, the paper did not mention that this theorem is\n     due to Ryan et al. (2022).\n\n   - Theorem 2 is a very\n     standard result for smooth optimization. Moreover, \n     footnote 1 of the paper has noted that the MCR2 objective\n     requires some non-convex constraints for itself to be well-posed,\n     but Theorem 2 (and perhaps also the implementation) simply\n     ignored these constraints, which are required for Theorem 1. This\n     creates a great mess.\n   - Figures similar to Figure 2 have already been reported by Ryan et\n     al. (2022). The purpose of such a repeat is not very clear.\n\n2. It is quite confusing why the paper keeps saying that MCR2 is a\n   principle (Similar objectives to MCR2 already exist, cf. Lezama et\n   al., (CVPR 2018), why their objectives are not principles?). Also,\n   the coding rate reduction explanation is only applicable to Gaussian data.\n   And one gets no interpretation when dealing with real data. It is\n   also not very clear to me why diverse representations are\n   practically important, and the paper did not show the performance\n   benefits compared to the cross-entropy loss.\n\n3. Here are the references that I mentioned above:\n   - https://arxiv.org/pdf/2105.10446.pdf (Ryan et al., 2022)\n   - https://arxiv.org/abs/1712.01727 (Lezama et al., CVPR 2018)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9BbJ4CDvS2", "reviewer": "Reviewer_L6k4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission proposes a federated methodology to learn low-dimensional representations from a dataset distributed among several clients. Basically, the idea lacks novelty and is with limited contribution as it is a simple application of the MCR2 objective to a federated learning setting. The optimization is still using fedAVG and the proofs are also routine as FedAVG as the embedding parameters are optimized based upon SGD. ", "summary_of_the_review": "Since the work lacks novelty and the contribution is very limited, I thus do not champion acceptance. Basically, it is only a simple adoption MCR2 to the federated setting. Since the optimization is also SGD and fedavg, there are no new challenges to transferring MCR2 objective to a federated setting including the proofs of convergence and others. ", "strength_and_weaknesses": "Strength:\n\nS1). The paper is clearly presented. Basically, the paper is well-organized and easy to follow. The authors give sufficient background on both federated learning and MCR2. \n\nS2). The references reviewed are somehow complete. The authors presented both traditional Federated learning and personalized FL.\n\nWeakness:\n\nW1). the work lacks novelty and the contribution is very limited. Basically, it is only a simple adoption of MCR2 to the federated setting. Since the optimization is also SGD and fedavg, there are no new challenges to transferring MCR2 objective to a federated setting including the proofs of convergence and others. \n\nW2). Moreover, since the optimization needs to collectively sum over different labels (K), however, in the personalized FL setting, it is possible that the same sample in different clients is assigned with different labels, as mentioned in paper https://arxiv.org/pdf/1910.01991.pdf\nIn this case, how the proposed obj of MCR2 would work in FL setting?\n\nW3). The representation learning framework MCR2 is still using class labels, why not use cross-entropy loss? what's the extra merits? The authors need to clarify the usage of such supervised embedding learning compared to popularly used cross-entropy loss with information bottleneck guidance. \n\nW4). The experiments are far from sufficient. Basically, the authors only present the convergence and effectiveness of learning a compact representation learning. Basically, what are the usefulness and the quality of the learned representations? What's the superiority of the optimization framework fast converge at the FL setting compared to traditional FedAVG?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject