Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a new learning scheme in Federated Learning that communicates knowledge between clients in the form of meta knowledge generated by dataset condensation. In this case, the global model aggregation in the cloud is replaced with a "meta training" process in the cloud using the combined meta knowledge from all participating clients in each communication round. The authors also proposed several modules that co-work with the main meta knowledge scheme, such as pseudo knowledge in the cloud using a generating model and dynamic weights.

The idea is clearly novel and the empirical effect of reducing communication rounds needed is impressive. The authors also show that other designed modules all contribute to the superiority of the overall framework.
strength_and_weaknesses: Clearly the meta knowledge sharing in place of the classic model weight aggregation is the most important and strong contribution of this work. In so heterogeneous scenarios in federated systems, model weight aggregation is really the pain point and potentially very bad in terms of communication efficiency. However, this work pointed out an alternative with clear empirical evidences that support the superiority of  it.

The only weakness I found in this work is the lack of discussion about the influence of the proposed condensation mechanism on the privacy preserving property of FL. This is crucial because the local meta knowledge will finally be sent to the cloud. While the authors didn't include discussions on privacy in the draft, I highly encourage them to at least add some references to existing analysis work about dataset condensation, which is not new technically in non-FL scenarios.
clarity,_quality,_novelty_and_reproducibility: The work is clearly written and easy to follow. The main technical contributions are novel.
summary_of_the_review: This is an outstanding work with novel technical contributions and impressive empirical performances.

=====================================

Note for the late submission of this review. I am very sorry that I submitted my review this late. I think I wrote the review locally on my computer but forgot to post it to OpenReview.

My original evaluation on this work without referring to other reviews was very positive, with only  some minor issues on the math formulation and some statements, such as:

1. In equations (4,5,7) of the original draft and the equations (3,5,6,8) of the updated draft as of 11/30/2022, the $\nabla$ operator misses subscripts that indicate with respect to which the derivatives are taken. The subscripts are needed because the derivatives are taken w.r.t. different inputs to the loss function $\mathcal{L}^c$.
2. In equation (6) of the original draft, the trailing parenthesis is missing. It is still missing in the updated draft.
3. The $\mathcal{L}^c$ notation is abused in the equation (7) of the original draft. The authors should mention that the notation comes with a little abuse.
4. The authors said above equation (7) in the original draft that "Apparently, the weight of each sample is inversely proportional to its prediction loss", which is contrary to the formulation. It seems to have been fixed in the updated draft.
5. There has been some unclarity of the bilevel optimization formula, but it has been addressed in the discussions between the authors and other reviewers.

I encourage the authors to address the above points, while they have not influenced my overall evaluation of this work so I hope it won't make the rebuttal process messy.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper develops a new FL solution. In the designed solution, the local clients distill meta knowledge based on local private data and shared knowledge from other clients, the distilled meta knowledge is uploaded to the server for global model training. As the global model training is based on meta knowledge from all active clients, the bias issue can be mitigated and the convergence can be accelerated experimentally. Additionally, the authors designed two novel mechanisms to improve the meta knowledge extraction on local clients, i.e., meta knowledge sharing and dynamic weight assignment. Both of the designed mechanisms are novel and technically correct. The authors conduct extensive experiments and ablation studies to demonstrate the efficacy and efficiency of their method.
strength_and_weaknesses: Strength:
 
-- The authors designed a new learning solution. Unlike prior FedAvg-based methods, the authors conduct meta knowledge extraction and upload the meta knowledge to a server. They use the meta knowledge from all active clients as normal training data for training a global model, mitigating the bias issue. The meta knowledge extraction and global model training are conducted alternatively.
 
-- The designed mechanisms, i.e., meta knowledge sharing and dynamic weight assignment, are novel and technically correct.
 
-- The authors conduct extensive experiments and ablation studies, and provide convincing experimental results to demonstrate the efficacy and efficiency of their method.
 
-- The writing is clear and easy to understand. The whole work is organized well.
 
Weakness:
Two minor issues:
-- It would be better to place the Algorithm pseudo code in the draft rather than supplementary material.
-- Why name the extracted knowledge as meta knowledge? Is there any specific reason? If yes, it would be better to provide the reason in the paper.
 
Clarity, quality, novelty and reproducibility
Clarity: The paper is well-written and easy-to-follow.
Quality: The technical quality is good. The method proposed in this paper is extensively evaluated. Novelty: The idea of this paper is novel. I haven't seen the same idea in prior works. Reproducibility: This work is with good reproducibility.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and easy-to-follow.
Quality: The technical quality is good. The method proposed in this paper is extensively evaluated. 
Novelty: The idea of this paper is novel. I haven't seen the same idea in prior works. Reproducibility: This work is with good reproducibility.
summary_of_the_review: This work is interesting and novel. The authors designed a new learning paradigm and designed two novel mechanisms in the learning. The motivation is explained well and the technical details are clear. They conduct extensive experiments to demonstrate the efficacy and efficiency of the designed solution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new meta-knowledge driven FL paradigm to optimize the FL training with limited communication rounds. Besides, two strategies namely Dynamic Weight Assignment and Meta Knowledge Sharing are proposed to improve the performance. 
strength_and_weaknesses: Strength:
S1: A new FL paradigm is proposed to deal with the limited communication budgets problem by leveraging condensate meta knowledge instead of the raw training data.
S2: The proposed paradigm is significantly better than existing FL solutions when facing limited communication rounds.
S3: Extensive experiments are conducted to show the effectiveness of the proposed method in terms of accuracy, communication efficiency (rounds) and also data leakage.

Weaknesses:
W1: I wonder if it is fair to evaluate the communication budgets using the metric of communication rounds, because unlike the competitors that the clients and the servers transmit the gradients or model weights, the FMKE framework proposed in this paper needs the clients to send the meta data, which have different amount of size.
W2: It is unknown if the bi-level optimization is optimal for FMKE framework. It is expected to compare the bi-level optimization with other baselines, e.g., simultaneously optimizing the two objectives.
W3: While dynamically adjusting the weights of the training samples has been widely studied, it lacks of comparisons between the proposed Dynamic Weight Assignment with the existing methods, such as AdaBoost.

clarity,_quality,_novelty_and_reproducibility: Clarity: Overall, the main idea, methodology, and the key experimental results are present clearly, except for some issues that could be further improved as mentioned above.

Quality: Goodness. The paper presents an effective solution to deal with an important problem.

Novelty: The core idea of the paper is novel, while some of the techniques are based on or similar to existing solutions.

Reproducibility: The experimental settings are clarified in this paper, but it would be better if the source code is made publicly available.

summary_of_the_review: Overall, the paper presents a new solution to tackle the FL training problem in case of limited communication rounds. The core performance is well evaluated. However, there still remains some issues that could be further improved. Thus, I recommend this paper a borderline accept score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach for Federated learning. The idea is to first distill the private data of each client (with respect to the current initialization of their model weights) and send them to the server. On the server side, a global model will learn from these synthetic data and broadcast the learned weight back to the clients. The proposed method is demonstrated to outperform other FL benchmarks on several datasets.
strength_and_weaknesses: Strength: Idea is clear and straight forward. The empirical result seems great. 
Weakness: The objective formulation seems problematic and I cannot make sense why it works in practice.

-- The notation in Eq. (3) does not make sense to me. $D_c$ is a constant, not a variable. Why is it being optimized? I take it as this is some random variable $D'$ that has the same desired dimension as the condensed dataset. If so, how is the local dataset used?

-- Another major concern is that Eq. (3) seems to have a trivial solution? First off, i think it is improper to have loopy dependency in an optimization objective ($\widehat{D}_c$ depends on $w_\ast$ and vice versa). Maybe it's better to write it as $\widehat{D}_c, w_\ast = \underset{w_c, D_c}{\mathrm{argmin}} \dots $, i.e., the bilevel optimization is an approximate procedure to solve for this objective and should not be the objective itself. Regardless of the notation, I think the optimization will converge when $\widehat{D}_c$  and $w_\ast$ are obtained such that $\mathcal{L}(\widehat{D}_c, w_\ast) = 0$. This is probably simple to achieve when there is no constrained on the meta dataset. For example, we can construct a dataset where all datapoints are labelled as class 1, and a model that always predicts class 1 with zero uncertainty (setting the bias of the prediction layer to $\mathbf{e}_1$ and zeroing out all other weights would probably suffice). Thus, a trivial solution exists and theoretically this could result in very bad performance. This seems intriguing given the very good result observed in the empirical studies. My best explanation for this good performance is that the local models only perform their bilevel optimization for a few steps, which is likely not enough to find such an exploit. 

-- This objective does not seem to explicitly prevent the condensed dataset from being exactly the same as the local data. After all, they live in the same input space, so it could happen. I understand that the paper conducts an empirical study to verify this is not the case, but when you deal with privacy, that is simply not good enough. What will happen when the global model somehow has zero loss on the client data? Then, the outer loop doesn't need to do anything but returning a subset of its local data (since loss can't be improved further). This seems like a big privacy vulnerability to me --- Imagine a client with only 10 data points on a binary classification task, then a malicious server would only need to send $2^10$ specifically constructed weights to eventually hit the perfect loss and hence recover exactly the client data? I think the meta dataset somehow has to have a different dimension than the original dataset for this to work.

Some other (minor) concerns:

--  We run three trials and report the mean accuracy performance (MAP) -- is the MAP averaged over both trials and clients? I would recommend showing some standard deviations.

-- How many update steps per round of local data distillation?

-- How is evaluation done? Does each local model of FMKE fine-tune the global weights before testing on their own data, or do they directly use the global weights trained on synthetic data? Is it the same for other benchmarks?

-- Can the authors show the MAP curve of FMKE beyond iteration 10 in Fig. 2? It's not just about achieving high performance, but also about demonstrating the stability of the method. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Idea is generally clear. As mentioned, eq. (3) is confusing and I will need clarification from the authors.
Quality: I currently think the approach is not sound. I hope the author can clarify if I have misunderstood anything. Despite my doubt, the empirical result seems great.
Novelty: Most components are applications of previous work (as acknowledged by the authors), but the overall idea is quite novel.
Reproducibility: No code was provided in the supplementary material. No anonymous github page was provided. 
summary_of_the_review: I recommend a rejection since I am not fully convinced about the distillation objective. I'm willing to change my score if the authors can clarify my concerns. I'm personally very curious about the exceptionally good results. I do hope that my assessment is wrong, because the method otherwise seems like a good contribution. I would suggest the authors to release the implementation on an anonymized github repo, so reviewers can properly investigate.

-------

Upon reading the authors' responses to my questions and revisions of the manuscript, I am convinced that the method is sound (Another minor comment: The revision is technically not a bilevel optimization anymore since both parameter updates happen simultaneously). As promised, I'm happy to change my scores accordingly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a federated learning framework which communicates condensed datasets obtained using meta-learning, which are then used as pseudo-training examples to train a global model at the server, along with the synthetic samples generated by a generator that is trained with the pseudo-training examples. The method is validated on multiple benchmark datasets with multiple federated learning algorithms, and is shown to obtain significant improvements over the base models. 

The reviewers in general considered the idea of communicating meta-knowledge instead of model parameters as novel, the proposed dataset condensation and dynamic weight assignment for obtaining meta-knowledge from each client as reasonable, the experimental validation including the ablation studies as extensive, and the paper well-written.

However, the reviewers also had a common concern on the potential data leakage from the condensed datasets that are transmitted to the server. In response, the authors provided some experimental results which show that it is very difficult to restore the original data from the meta-knowledge, and referred to the theoretical analysis in [Dong et al. 22], which dealt away with some of their concerns. As this resolved the critical concerns from the reviewers, all reviewers recommended to accept the paper after the discussion period.

There was also an additional concern on my side on the lack of discussion and experimental comparison against a high related work, FedMix [Yoon et al. 21], which leverages averaged data from each client as well as augmented samples generated with mixup regularization, and [Kim et al. 22] which proposes a federated learning framework based on dataset condensation. The authors provided some discussions and experimental comparisons against them in the interactive discussion period, as well as some experimental results. I strongly suggest them to include the discussions as well as full experimental comparisons against them in the final revision, and adjust the claimed contributions of the proposed work accordingly. Also, since data leakage could be critical for a federated learning scenario, there should be a more extensive analysis of the potential threats to the data privacy introduced by the proposed method. 

[Yoon et al. 21] FedMix: Approximation of Mixup Under Mean Augmented Federated Learning, ICLR 2021
[Kim et al. 22] Stable Federated Learning with Dataset Condensation, JCSE 2022
justification_for_why_not_higher_score: - Novelty is relatively weak, since similar approach has been proposed in previous works such as [Yoon et al. 21] and [Kim et al. 22].
- The privacy concern is not fully resolved.
justification_for_why_not_lower_score: The proposed method obtains impressive performance over compared baselines.
Decision: Accept: poster