Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The submission proposes a federated methodology to learn low-dimensional representations from a dataset distributed among several clients. Basically, the idea lacks novelty and is with limited contribution as it is a simple application of the MCR2 objective to a federated learning setting. The optimization is still using fedAVG and the proofs are also routine as FedAVG as the embedding parameters are optimized based upon SGD. 
strength_and_weaknesses: Strength:

S1). The paper is clearly presented. Basically, the paper is well-organized and easy to follow. The authors give sufficient background on both federated learning and MCR2. 

S2). The references reviewed are somehow complete. The authors presented both traditional Federated learning and personalized FL.

Weakness:

W1). the work lacks novelty and the contribution is very limited. Basically, it is only a simple adoption of MCR2 to the federated setting. Since the optimization is also SGD and fedavg, there are no new challenges to transferring MCR2 objective to a federated setting including the proofs of convergence and others. 

W2). Moreover, since the optimization needs to collectively sum over different labels (K), however, in the personalized FL setting, it is possible that the same sample in different clients is assigned with different labels, as mentioned in paper https://arxiv.org/pdf/1910.01991.pdf
In this case, how the proposed obj of MCR2 would work in FL setting?

W3). The representation learning framework MCR2 is still using class labels, why not use cross-entropy loss? what's the extra merits? The authors need to clarify the usage of such supervised embedding learning compared to popularly used cross-entropy loss with information bottleneck guidance. 

W4). The experiments are far from sufficient. Basically, the authors only present the convergence and effectiveness of learning a compact representation learning. Basically, what are the usefulness and the quality of the learned representations? What's the superiority of the optimization framework fast converge at the FL setting compared to traditional FedAVG?


clarity,_quality,_novelty_and_reproducibility: Clarity is fine
Quality is low because of insufficient experiments.
Novelty is very limited.
Reproducibility is fine.
summary_of_the_review: Since the work lacks novelty and the contribution is very limited, I thus do not champion acceptance. Basically, it is only a simple adoption MCR2 to the federated setting. Since the optimization is also SGD and fedavg, there are no new challenges to transferring MCR2 objective to a federated setting including the proofs of convergence and others.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm to learn low-dimensional representations in the Federated Learning setting. The paper borrows idea from a recent work which formulated a loss that could maximize the code rate difference between the entire dataset and summation of individual classes. The work takes a further step by identifying that learning a low-dimensional representation could be a collaborative objective and introduces an algorithm to solve the learning objective in a distributed manner.

strength_and_weaknesses: Strength: The work is overall well-written and easy to read, all the details of the cocepts, setups are clearly stated. The theoretical results demonstrate that in the distributive setting,  the learning objective could still be successfully solved.

Weakness: The novelty of the work is quite limited, although the usage of the MCR loss in federated learning is new, both the loss and the algorithm design are not new. Besides, the experimental part is quite weak as it only demonstrates the results of the MCR loss under Federated Learning setting, no comparison has been done to show the supremacy of the proposed algorithm.

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to understand and the code is released for the reproducibility. The novelty of the paper is quite limited.

summary_of_the_review: This is an well-written paper proposing an interesting extension for the MCR loss in the federated learning setting. The experimental parts are a bit weak and the overall novelty is also limited.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed to replace the cross-entropy loss with the MCR2 loss in federated learning. They provided a theoretical analysis of learning convergence of the global model using the MCR2 loss. They also empirically show the convergence of the loss and the orthogonality of the class-wise subspaces learnt. 
strength_and_weaknesses: Strength:

The idea of separating the backdone network from the project head is an interesting idea.

Weakness:

The idea proposed seems rather trivial. And the empirical evaluation is not done in comparison with some natural baselines. It is difficult to see the significance of the proposed work. 


clarity,_quality,_novelty_and_reproducibility: The paper is overall well-written and very readable. But it lacks novelty.

summary_of_the_review: The paper makes a rather simple change to the standard federated learning setting by using a different loss function. The empirical evaluation does not show the advantage of performing that change.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper considers the federated learning problem and proposes to
minimize the difference between two log determinant functions instead of
the cross-entropic loss.


strength_and_weaknesses: # Weakness
1. The work is not solid. My
   judgment is based on the following:
   - The background section is unnecessarily long. For example, the
     review of Section 2.2 is redundant, and the paper can simply say
     that, as motivated by some rate-distortion theory, Yu et al.
     (2020) proposed this MCR2 objective.
   - Theorem 1 is basically a copy-and-paste from a follow-up work
     (Ryan et al., 2022) of Yu et al. (2020), called ReduNet.
     Moreover, the copy is wrong: It misses the key assumption that the
     objective admits a low-rank solution, see Theorem 12 of Ryan et al.
     (2022). Finally, the paper did not mention that this theorem is
     due to Ryan et al. (2022).

   - Theorem 2 is a very
     standard result for smooth optimization. Moreover, 
     footnote 1 of the paper has noted that the MCR2 objective
     requires some non-convex constraints for itself to be well-posed,
     but Theorem 2 (and perhaps also the implementation) simply
     ignored these constraints, which are required for Theorem 1. This
     creates a great mess.
   - Figures similar to Figure 2 have already been reported by Ryan et
     al. (2022). The purpose of such a repeat is not very clear.

2. It is quite confusing why the paper keeps saying that MCR2 is a
   principle (Similar objectives to MCR2 already exist, cf. Lezama et
   al., (CVPR 2018), why their objectives are not principles?). Also,
   the coding rate reduction explanation is only applicable to Gaussian data.
   And one gets no interpretation when dealing with real data. It is
   also not very clear to me why diverse representations are
   practically important, and the paper did not show the performance
   benefits compared to the cross-entropy loss.

3. Here are the references that I mentioned above:
   - https://arxiv.org/pdf/2105.10446.pdf (Ryan et al., 2022)
   - https://arxiv.org/abs/1712.01727 (Lezama et al., CVPR 2018)
clarity,_quality,_novelty_and_reproducibility: The novelty of the work is extremely limited. The writing becomes unclear when it comes to whether the normalization constraints are imposed on the MCR2 objective.
summary_of_the_review: As detailed above, the paper is clearly below the bar of ICLR in several
aspects. In its current form, the paper is better suited for some less
selective conferences.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach to federated representation learning based on the so-called principle of maximal coding rate reduction. The approach is connected to personalized federated learning, and the aim is to learn a shared backbone (feature extractor) across clients, while clients learn a personalized prediction head. The paper claims to contribute theory ensuring that optimal subspaces will be learned under certain assumptions, and that the proposed federated method FLOW converges. Experiments are reported for simulations of federated training of a ResNet18 on CIFAR10.

The proposed approach may be promising based on the CIFAR10 results.

Overall, there were several weaknesses noted about this work. The main weakness is the limited empirical evaluation; the paper could be substantially strengthened by exploring this method on several datasets in several settings, to both illustrate the advantages over other personalized FL methods, and also to identify potential limitations. Several reviewers questioned the novelty and expressed the opinion that just applying the idea of MCR2 loss in the context of federated learning brought limited novelty.
justification_for_why_not_higher_score: Insufficient evaluation to support all of the claims made, and limited novelty.
Decision: Reject