Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This work proposed applying hypernetwork for personalized federated reinforcement learning. Empirical results show that the proposed method achieves better performance compared to FedAvg.
strength_and_weaknesses: Weaknesses:
- The novelty of this work is limited. It seems that this work is a simple application of previous works. The problem being studied is also restricted (one specific environment). There's no evidence in the paper that the proposed method works for general RL problems.
- Significantly missing important related works in both FL / personalized FL / RL.
- The motivation of this work is unclear. What is the unique challenge in RL that prevents existing personalized FL framework to be deployed (for example, pFedMe [1], Ditto [2], etc.)

[1] T Dinh, C., Tran, N., & Nguyen, J. (2020). Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33, 21394-21405.

[2] Li, T., Hu, S., Beirami, A., & Smith, V. (2021, July). Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning (pp. 6357-6368). PMLR.
clarity,_quality,_novelty_and_reproducibility: Limited quality and novelty.
summary_of_the_review: This paper doesn't demonstrate enough novelty and contribution to the field of RL and FL. This is below the standard for ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes the first step to combine personalized federated hypernetworks (PFH) with reinforcement learning (RL) and applies their method in the specific field of price-setting for energy demand response. The proposed method can work when the common centralized training with decentralized execution (CTDE) framework in multi-agent reinforcement learning (MARL) is not applicable. Empirically, this paper demonstrates that PFH enables efficient personalized model learning in multi-task MARL on experiments with synthetic datasets when the heterogeneous tasks have moderate diversity.
strength_and_weaknesses: Strength

1. This paper is the first to combine PFH with RL.

2. The experiment analyses are sufficient. The authors give detailed discussions on the possible reasons why their proposed method outperforms or is outperformed by baseline methods in different environments.

Weaknesses

1. Novelty: My greatest concern is about novelty. This paper seems to be a bit incremental in the sense that it directly combines the PFH with the proximal policy optimization (PPO) algorithm and then applies it in the specific energy pricing domain. The authors also do not seem to give any discussions about whether there are any technical difficulties when combining PFH with PPO and how they tackle these difficulties. 

2. Technical Soundness: Though the proposed method applies the federated learning (FL) framework and trains all the personalized models without the communication of raw data, I still find it a bit overclaiming that the proposed method is for “privacy preservation”. As shown in Figure 1, the gradients of personalized models are sent to the central server. However, several works have shown that privacy leakage is still possible, even when only the gradients are communicated [1,2]. 

3. Experiments: 

(a)  The proposed method indeed outperforms all the baseline methods when the heterogeneous tasks have medium diversity. However, all the experiments are conducted on synthetic datasets, making the results less convincing. It may be better if the authors can conduct more empirical evaluations on environments using real-world datasets.

(b)  The decentralized MARL method using networked agents also does not require the CTDE framework and can even be trained in a fully decentralized manner [6,7]. Combining with the differential privacy (DP) framework [8] to protect the information in the communication between networked agents, the decentralized MARL method is rigorously privacy-preserving, and can also be applied to the energy pricing domain. I would suggest the authors also include the DP-based privacy-preserving decentralized MARL method as a baseline to make the experiments more convincing.

(c)  Is there any criterion or clue about when the diversity is "medium"?

4. Literature Review: Several works have already shown the feasibility of combining FL with RL or MARL [3,4,5], which are missing in the current related work section. I would suggest the authors give more discussions and comparisons with these works.

5. Presentation: Overall, the presentation is clear but some parts of the paper can be further improved. For instance, before the method section, it would be better if a section introducing the setting or the preliminaries of the problem concerned in this paper is given (say, the basic preliminaries of RL). Besides, the sizes of some figures and tables in the experiment section can be reduced to put the related work section from the appendix to the main body of the paper for better readability. Also, it is unclear to me what Eq. (1) means, and it would be better if further explanations are given. For Eq. (2), it may be inappropriate to say that some RL agent aims to maximize “a reward”. In general, the objective of RL is to maximize the long-term expected total rewards instead of an instantaneous reward at some time step.

[1] Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019, May). Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP) (pp. 691-706). IEEE.

[2] Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017, October). Deep models under the GAN: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (pp. 603-618).

[3] Xu, X., Li, R., Zhao, Z., & Zhang, H. (2021). The gradient convergence bound of federated multi-agent reinforcement learning with efficient communication. arXiv preprint arXiv:2103.13026.

[4] Kwon, D., Jeon, J., Park, S., Kim, J., & Cho, S. (2020). Multiagent DDPG-based deep learning for smart ocean federated learning IoT networks. IEEE Internet of Things Journal, 7(10), 9895-9903.

[5] Zhang, W., Yang, D., Wu, W., Peng, H., Zhang, N., Zhang, H., & Shen, X. (2021). Optimizing federated learning in distributed industrial IoT: A multi-agent approach. IEEE Journal on Selected Areas in Communications, 39(12), 3688-3703.

[6] Zhang, K., Yang, Z., & Başar, T. (2021). Decentralized multi-agent reinforcement learning with networked agents: Recent advances. Frontiers of Information Technology & Electronic Engineering, 22(6), 802-814.

[7] Zhang, K., Yang, Z., Liu, H., Zhang, T., & Basar, T. (2018, July). Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning (pp. 5872-5881). PMLR.

[8] Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4), 211-407.
clarity,_quality,_novelty_and_reproducibility: Reproducibility: I appreciate the details of how the hyperparameters are selected are given in the appendix, but there seems no reproducibility statement in the current paper. 

Clarity and Novelty: Please see the detailed reviews above.

summary_of_the_review: Though this paper makes the first step to combine PFH with PPO, the combination seems a bit incremental, and all the experiments are only conducted on the environments using synthetic datasets. Besides, the current literature review seems not sufficient. Therefore, I am afraid that this paper is not ready for publication as an ICLR paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work is an application paper that deployed the existing federated learning technologies namely hypernetwork to achieve personalized learning in reinforcement learning across multiple agents. The problem statement is based on a simplified hypothetical situation where multiple communities are powered by renewable energy with battery to assist the grid. With assumption of consumer will adjust their behavior based on the price changes. The training process of system contains two parts, local training, and global aggregation. The global aggregation is done by a hypernetwork where gradient from different agents upload their gradient to update this hypernetwork. 
strength_and_weaknesses: Strength:
The problem formation is clear. The real problem of energy demand between different utility node and consumer behavior is complex. Author made some reasonable simplification to scale down the problem into a manageable format that could be analyzed by simulation. 

Weaknesses:
The weakness of the paper is experiment part is not strong enough to show advantage of their algorithm. The experiments show your algorithm is not always work, where only a selected scenario it outperformed baseline.
The formation for total profit should be explained more since it is based on the process of utility market and consumer behavior. The constrain of consumer behavior would be stated since it is unrealistically to have all the consumer to cook at non-dinner period or not use AC during the middle of day. 

clarity,_quality,_novelty_and_reproducibility: Many parts of experiment need additional information to clarity since it is based on a simplified hypothetical simulation. Since it is an application paper, the novelty is more valued on the contribution to solve the application problem. In this paper, the augment of advantage of using this algorithm in this application is not strong. Many critical paper settings were not explained here, which made to reproduce the same experiment challenging. 
summary_of_the_review: This paper is an application paper which deployed a personalized federated learning reinforcement learning in energy optimization situation. The technical novelty of algorithm itself is not strong which is acceptable for application paper. But the analysis and experiment on the simulation does not support the paper to become a high-quality application paper. More experiment, problem constrains and explanations for the simulation should be included in the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach that combines personalilzed federated networks with RL

Strengths: 
* Personalized federated RL is a problem that has not be explored extensively
* The applicatioh of energy demand response is nice

Weaknesses:
* Low novelty (straightforward combination of PFH and PPO)
* Weak experimental evaluation (only one environment suite)

Given the incremental nature of the work and weak experiments, this work is not ready for publication.
justification_for_why_not_higher_score: * Low novelty (straightforward combination of PFH and PPO)
* Weak experimental evaluation (only one environment suite)

justification_for_why_not_lower_score: NA
Decision: Reject