Personalized Federated Hypernetworks for Privacy Preservation in Multi-Task Reinforcement Learning
Authors: Doseok Jang
Larry Yan
Lucas Spangher
Selvaprabu Nadarajah
Costas Spanos
Venue: Submitted to ICLR 2023 (2023)
Keywords: microgrid clusters
energy demand response
transactive energy control
neural networks
multi-agent reinforcement learning
reinforcement learning
multi-task learning
transfer learning
hypernetworks
federated learning
personalized federated learning
microgrids

Abstract: Multi-Agent Reinforcement Learning currently focuses on implementations where all data and training can be centralized to one machine. But what if local agents are split across multiple tasks, and need to keep data private between each? We develop the first application of Personalized Federated Hypernetworks (PFH) to Reinforcement Learning (RL). We then present a novel application of PFH to few-shot transfer, and demonstrate significant initial increases in learning. PFH has never been demonstrated beyond supervised learning benchmarks, so we apply PFH to an important domain: RL price-setting for energy demand response. We consider a general case across where agents are split across multiple microgrids, wherein energy consumption data must be kept private within each microgrid. Together, our work explores how the fields of personalized federated learning and RL can come together to make learning efficient across multiple tasks while keeping data secure.

Under review as a conference paper at ICLR 2023
PERSONALIZED FEDERATED HYPERNETWORKS FOR PRIVACY PRESERVATION IN -ASTI-TASK REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-Agent Reinforcement Learning currently focuses on implementations where all data and training can be centralized to one machine. But what if lo- cal agents are split across multiple tasks, and need keep data private between each? We develop the first application of Personalized Federated Hypernetworks (PFH) to Reinforcement Learning (RL). We then present a novel application PFH to few-shot transfer, and demonstrate significant initial increases in Iearn- ing. PFH has never been demonstrated beyond supervised learning benchmarks, apply PFH
important domain: RL price-setting for energy demand response. We consider general case across where agents are split across multiple microgrids, wherein energy consumption data must be kept private within each microgrid. Together, our work explores how the fields of personalized federated learning and RL can come together make learning efficient across multiple tasks while keeping data secure.
INTRODUCTION
As Reinforcement Learning (RL) is brought to bear pressing societal issues such the green energy transition, the types of environments that RL must perform well in may display characteristics exotic to classical RL environments. Real applications at scale may require privacy guarantees which are not provided by modern multi-agent RI algorithms they may train privileged or corporate data (Lowe al. 2017, Sunehag et al. 2017 Rashid et al. 2018); any app that personalizes an RL agent to individual users must take care to protect their privacy by not storing all their data in a central server. Real world applications will also likely feature heterogencous tasks; every user, robot, energy system will have different traits that cannot be accounted for by "one size fits all" algorithms. As previous work in
RL (Qi et al.) 2021 Wang et al.) 2020cc Ren et al.) 2019 Anwar & Raychowdhury [022] does not extend to personalized models, the competing goals of privacy and personalization must be accomplished at the other's 's expense.
One approach toward privacy preservation by decentralizing data servers within supervised learning federated learning (Shokri & Shmatikov] 2015). Federated learning algorithms train global model from gradient updates sent by individual clients training on their own data, which is never sent to the central server. An extension of federated learning technique is personalized federated learning using hypernetworks (PFH, Shammsia et al. (202T), which allows for behavior tailored to individual heterogeneous tasks by splitting the model into a global common component (i.e. the hy- pernetwork), and local individual component local network generated by the hypernetwork), which is tailored to each client. This task specialization allows for learning common features to- gether in the global component while allowing for learning client-specifie knowledge in the local component.
We present a novel application PFH to RL a realistic power systems setting that requires both privacy and heterogeneity in agents accommodate diverse, sensitive environments. An RL controller optimizing hourly transactive energy pricing has been shown to optimize energy usage (Li & Hong 2014, Spangher 2021 Siizuuez-Canter
al. 2019 Agwan al. 2021) by incentivizing consumers, at the scale of groups of buildings (microgrids) or office workers within buildings, to shift demand to different times of day. By guiding consumers to defer energy demands to hours

---

Under review as a conference paper at ICLR 2023
when solar is especially active, it is possible to drop building's carbon impact 48% of normal operation through RL price-setting (Jang et al., 2022), which could have massive implications for grid sustainability. However, RL can be extremely data hungry; prior transactive control attempts required about 80 years of training data (Agwan et al. 202T).
To increase the amount of data available, consider multiple RL agents, each managing their own (slightly different) microgrid through energy prices and collecting data in parallel. This microgrid environment a multi-task, multi-agent setup in which the management of each microgrid, through prices, constitutes a task. We characterize our problem as multi-agent because we have multiple RL agents optimizing a shared reward (total profit), and multi-task because optimization of profit each of the different microgrids presents tasks that are related but also independent due to differences in size, number of batteries in each building, etc. We hypothesize we can accelerate training by incorporating data from multiple microgrids with different characteristics. Learning to set prices using data from multiple microgrids (source tasks) also opens the door to few-shot learning in new microgrids (target tasks), wherein we learn to generate near-optimal prices for microgrid very quickly.
However, energy data is data in which privacy concerns are paramount. Is our hope contribute to privacy protection by aggregating learning, not data, to one central source. Not only would keeping data of buildings' energy consumption at one central location present major privacy concern if this central machine is compromised, but message passing of the raw data could present an additional source of vulnerability. Although each microgrid might have access the data of a few buildings at time, the scale of damage would be much larger if data was stored in a central server across multiple microgrids.
We now present a hypothetical setting in which our architecture would be useful. One could imagine a hacker being able to learn when the hypothetical company CovertAl trains their new 80 quintillion parameter language model CPT-4 from the energy consumption of CovertAl's compute warehouses. The hacker could sabotage power lines at the right moment to erase learning gains. They may then turn their attention to residential neighborhoods. Here, they could figure out when people are not home from the energy consumption of domestic buildings, timing a theft; they could also disaggregate energy signals to learn the appliances the homeowner has glean sensitive health information medical devices produce noticeable patterns in energy consumption.
Applying PFH to the energy application remedies both of these competing issues. PFH takes prraccypreservattoon into account by design, and accounts for heterogeneous tasks by generating RL agents individualized to each microgrid's size, number of solar panels, batteries, etc. We demon- strate that PFH learns the underlying factors that define an environment by applying PFH to the microgrid price-setting problem, where observe increases the scale of millions of dollars in total microgrid profit (reward) over federated and local learning. We also demonstrate how PFH can be used for few-shot transfer learning for new local agents entering the system by reporting drastic training speed-ups (> 100x) when transferring from source tasks to target tasks. Thus PFH drastically increases the feasibility of RL for energy price-setting.
Methodologically, our paper is novel in its presentation of an adaptation of state of the art privacy- preserving algorithm to RL. To our knowledge we are the first to explicitly apply personalized federated learning to multi-task, multi-agent RL when centralized learning and joint action-values are unavailable. Application-wist our paper is also novel in its improvement in energy demand response across heterogeneous microgrids. We hope our work highlights important microgrid environment to the RL community, helps establish the use of PFH within RL, and allows for RL to address problems where learning speed and privacy are fundamental.
2 DEFINITIONS
Energy Demand Response is technique used by grid operators incentivize consumers to shift demand to times when it better for grid stability/climate emissions, such as when solar energy peaks. Demand response has the same function as grid-level batteries would in casing the volatility of wind and solar energy and is seen as an important tool in the energy transition (Albadi & El-
(Saadany] 2007).
See Appendix A for discussion of related work.
2

---

Under review as a conference paper at ICLR 2023
A microgrid is defined as a small group of buildings that transacts energy with each other through some energy aggregator, governed by an hourly energy pricing scheme. One may imagine they are situated close together with respect to not only geography but also the wiring topology of the grid, making trading within the microgrid preferable to trading with the grid. We will refer to groups of microgrids "microgrid clusters" ".
A prosumer is an entity that both consumes and produces energy, like a building with rooftop solar.
We wish to disambiguate between multi-task and multi-agent for the reader's convenience. We use them in the conventional sense: multi-task relates to multiple, related settings (in our case slightly different MDP's in each different microgrid) whereas multi-agent refers multiple different poli- cies.
A hypernetwork is neural network that outputs the weights of another neural network.
A Privacy-preserve algorithm is one that does not require communication or storage of raw data samples to central server.
METHODS
3.1 LEARNING ENVIRONMENT
The Microgridl Leamn (Agaan et al. 2021 environment is OpenAl Gym (Brockman et al. 2016) environment used to study RL-set pricing in prosumer aggregations. Specifically, the environment is structured such that an RL agent and an energy utility both broadcast a day's worth of hourly buy and sell prices; Ag, A., and U.. U.. respectively. to microgrid's 's simulated prosumers, who choose at the beginning of the day which hours they will transact with the RL agent and which hours they will transact with the energy utility. Each prosumer an office building composed of a year's worth of historical data and user-defined, non-negative battery and photovoltaic capacities At every step, i.e. one day where all 24 hours are considered, every prosumer solves a convex optimization optimizing their battery bharging/discharginn u4, ū_, to maximize their individual profit; i.e.:
arg max 17
+ U.). (E.+ ū.)) - (min(A,, 0.)
U.). +
(1)
Where és, EG are inflexible energy generation and consumption, respectively, of the prosumers, (a, b) is dot product, and the min and max are taken elementwise. The first term, i.e. the element-wise maximum, is thus the gross profit from energy each prosumer sells, and the second term, i.e. the element-wise minimum, is the gross expenditures from energy each prosumer buys. Please note that every vector here may be considered a 24 hour vector, and that opposing actions are exclusive (i.e. sell and buy, or charge and discharge.) Thus, entries in the sell vector in which the prosumer is buy- ing are represented by O's. By ensuring that each prosumer has the ability to transact with either the utility or the microgrid, we incentivize the microgrid output prices that are better than the utility, guaranteeing a better experience for prosumers under this microgrid structure. One important sim- plification we have made is that we model human behavior as fixed in E relative to the price signal; we do not expect humans to change behaviors (e.g., eating lunch at a different time to take advan- tage of cheaper energy prices). We only model how distributed batteries could be automatically controlled to maximize the prosumer's profit.
In an environment with this collection of prosumers, the RL agent solves an MDP defined by state space 5 = (Ü., U. G. 1,,,-1,,,,-11 E 224424+24424421 where ÿ is the day's 's solar prediction and the fb/s,t-1 1 are prosumer buying and selling energy from the previous day. The RL controller emits actions A ;= (A,, A.) E 24 The agent seeks to maximize long term discounted reward defined by its individual profit, i.c.:
Please note that while "microgrid clusters does appear in the literature, there is no clear consensus around the term being the only appropriate term for multiple connected mierogrids.
'The photovoltaic output is defined by a fixed year's 's worth of solar generation for one unit panels, which is then scaled by the number of photovoltaic panels assigned at initialization.
"The day's 's solar predictions simply read from csv.
3

---

Under review as a conference paper at ICLR 2023
(Ao, Fio) (A.r Fia)
arg max As
We somewhat abuse the E notation to conveniently define Ebjs as the total amount of energy bought from or sold to the RL agent hour to hour.
A
RL Price Controller
Utiliy
Microgrid Aggregator
Im
B
Environment Embedding Vector
Buy/sell prices from utility
OOOO
60.
10 02
6e,
Buy/sell to buildings
Figure I: Microgrids and PFH: A. We imagine prosumer that can, each hour of the day, choose to sell energy surplus or purchase ummet energy demand from the larger utility or to the microgrid aggregator. The microgrid aggregator's ss energy buy/sell prices determined by RL controller. B. A Hypernetwork for Personalized Federated Learning (PFH) receives gradient updates from RI. controllers and sends back weights. C. The hypernework takes input environment embedding vector and outputs weights for an RL controller. The RI. agent takes as input buy/sell prices from the utility and outputs buy/sell prices to the buildings the microgrid the agent manages. The RL agent sends back gradient update to the hypernetwork, which uses the update to compute the gradient update for the hypernetwor'ss 's own weights.
Because the agent is an aggregator that does generate its energy, the profit the agent gener- ates comes from the difference in price between the energy it buys from prosumers at that timestep and the energy it sells to prosumers that timestep. Any excess supply or demand is transacted with the energy utility. In this way, the environment neatly models a realistic transactive system.
3.2 REINFORCEMENT LEARNING
We use Proximal Policy Optimization (Schulman et al. 2017) (PPO), a popular actor-critic based algorithm, to train all of our RL agents solve the MDP introduced in 3.1 because PPO is reliable and highly performant. Note that both algorithms introduced in 3.3 and 3.4 are agnostic to the architecture of the local policies, so could any gradient-based model.
3.3 FEDERATED LEARNING
In order to learn shared model between multiple microgrids in privacy preserving manner, turn to federated learning. McMahan et al. (2017) presented what is now the most popular federated learning scheme: Federated Averaging (FedAvg). FedAvg is simple parameters for the policy for microgrid i at timestep as 0,,. All the 8 are initialized with the same weights, 010 02n ... 0no, etc. Then each policy trains on its microgrid for
implement. Denote the
4

---

Under review as a conference paper at ICLR 2023
local steps, producing new 8t for each microgrid, which has adapted to be better at price-setting in microgrid i than the original f... All the 8y are transmitted back to central server, where they compute the shared model for the next iteration by averaging all the *
1 T
IIk = 02x = I..
(3)
Then the local models train their own, send trained models back central server, and repeat. Sending model information only preserves privacy because only the parameters 0 are commu- nicated with the central server, never any data. Note that in our setup, every client participates in the weight exchange process, not just a sampled subset of the clients. While FedAvg is a simple algorithm that performs well in supervised learning, it learns global policy for all the price-setting agents, In our case, global model is not ideal as microgrids may have different energy consump- tion/supply behaviors.
3.4 HYPERNETWORKS FOR PERSONALIZED FEDERATED LEARNING (PFH)
To learn a shared model that is still able to personalize to individual microgrids, we turn to hy- pernetworks for personalized federated learning. (Shamsian al. (001). Personalized federated learning algorithm has found great success in supervised learning, beating FedAvg and personal- ized federated learning approaches based on meta-learning (Fallah al. 2020), Moreau Envelopes
Dinh et al. 2020). and Personalization Layers (Arivazhagan et al. 2019). However, personalized federated learning has never been used before for RL.
Now we will describe PFH more formally. Please refer to Fig. I for a visual of the algorithm, and Algorithm| [ for pseudocode. Consider again Oiz E R" as an m dimensional vector denoting the parameters of the policy for microgrid i at timestep t. A hypernetwork is a neural network that outputs the parameters of another neural network. We will have one global Hø. RR' + R"
parameterized by e. Hø, takes as input an environment embedding vector v; E E which is learned for each environment along with the hypernetoork. We initialize 0. = H (1;) Vi E [1, n).
Then local agent trains for l steps, producing new parameters 6/ Then, Adi = 6/, - 0... is sent back the central server, where it used update the hypernetwork:
1
de 60
0 00
A0
Since the hypernetwork outputs neural networks conditioned on the environment, it is able create RL agents that are personalized to the needs of each microgrid. We also still preserve privacy by only communicating parameters with the central server instead of data.
3.5 DIVERSITY AND OPTIMAL USE OF PFH
One factor that could affect the relative performance of PFH is the heterogeneity the scenario. A homogeneous scenario (imagine cookie-cutter residential neighborhood) could be suitable for federated learning methods due to similarity in behavior. In contrast, an extremely heterogeneous scenario (imagine mixed-use city blocks with night-life, shopping, and residential real estate) could have wildly different energy demands, which may be better learned by individual local networks without any mechanism to share learning. We hypothesize PFH will perform competitively some average of these two extremes. If local environments are diverse yet share similar underlying mech- anisms, PFH will be able to fit local conditions while sharing information on common trends.
'Note our setup is slightly different from the original PFH (Shmssian al. 2021 every client participates in each round, not just sampled subset of clients. We made this small variation to better understand whether scaling the algorithm larger numbers of microgrids would be useful.
5

---

Under review as a conference paper at ICLR 2023
Algorithm Personalized Federated Hypernetworks
Input: Environments E and hypemettork Hg. For each environment E, an RL policy A(e) and hyppemettorr--ppecffiedd parameters (e). Hyperparameters: Number of rounds R, number of local training steps per hypernetwork update K, learning rate o.
for ..... . do for environment E E do
Get parameters ((e) 8(e) == Ha(e). for = 1,..... K do
Collect rollouts R from e using policy A(e) with parameters 0(e). Update (()) using PPO with rollouts R. Initialize @update == 0.
for environment E E do (f(e) = (()) - (e))
V (e)(e) Ad(e)
@update == @update +
Update hypernetwork parameters
Pupdate-
4 EXPERIMENTAL SETUP
4.1 SIMULATING DIVERSE MICROGRIDS
Because each microgrid is defined by a distribution of photovoltaics and battery sizes, we propose simple way to tweak the amount of diversity in system. We sample photovoltaic and battery sizes from normal distributions, changing the variance o² as the diversity parameter, and round outcomes to the nearest intege We sample from N(u 100, for medium diversity and (100, 50) for high. We note here that we have chosen the low, medium, and high cases such that 95% of samples (i.e. standard deviations around the mean) in the high case hit realistic bounds in the environment; i.e., O (an obvious lower bound) and 20
10) for low diversity cases, NN000 30)
4.2 BASELINES
We compare PFH against FedAvg and two other baselines. First, observe what happens with no RL control at all; the microgrid aggregator outputs prices that are exactly the same as the utility's. ss. We assume buildings choose meet half their energy demand/surplus with the utility and half with the aggregator. Our second baseline is the approach used in Agwan et al. (202T): training all the local RL controllers with only their own data: central model or inter-microgrid communication. These two baselines, no RL and local control, are designed to highlight the added value of RI the task of price-setting for energy demand response in microgrids, and the added value of having some central model that aggregates learning across multiple microgrids, respectively.
For specification on how we selected hyperparameters, please see Appendix E
As we are sampling from "hyper" distribution to instantiate houses, means of the distribution are not as important as variances instantiating diversity.
200 is a realistic upper bound in both solar panels and batteries: 200 solar panels would require area of 60 70 ft, which bounds the square footage of many commercial roofs, and 200 batteries would be realistic upper bound of entities not engaging in commercial grid services.
"The most common non-RL methods for microgrid price-setting are iterative pricing methods (IP) Liu
al. 2017 Wang & Huang 2016) which buildings "bargain" with microgrids reach equilibrium prices. exclude these baselines because they require each building develop their own demand forecasts. This requirement raises the computational barrier for entry by an order of magnitude. For comparison, if we had 10 microgrids with 10 buildings each, local RL requires training 10 models (10 microgrids), PFH and FedAvg requires 11 (10 microgrids central model), and IP requires 100 (10 buildings 10 microgrids), (Agan et al. (202T) also showed RL results in less volatile pricing curves and better performance compared to TP.

---

Under review as a conference paper at ICLR 2023
Table I: Cumulative profits above base utility pricing after 10,000 days, in hundred thousands.
Scenario
PFH
FedAvg
Local Baseline
Simple, 5 agents
39.23
45.75
43.72
Simple, 10 agents
42.11
41.65
43.18
Simple, 20 agents
40.85
34.52
42,82
Medium, 5 agents
47.50
40.95
40.12
Medium, 10 agents
46,89
43.82
41.73
Medium, 20 agents
48.22
39.38
39.66
Complex, agents
34.77
32.66
35.60
Complex, 10 agents
43.01
42.08
45.24
Complex, 20 agents
44.39
38.70
40.78
4.3 MULTI-TASK TRANSFER
An interesting feature of our hypernetwork- -ased setup is the potential for multi-task learning and few-shot transfer learning. The optimization problem of setting prices for each microgrid can be viewed as an individual task. Since the hypernetwork should learn some common strategies for each task, we tested whether it can generalize unseen tasks with little training. To test this hypothesis, we simply take a hypernetwork that has trained for to manage a microgrid cluster with 20 microgrids of medium diversity and train the hypernetwork manage a new microgrid cluster of 20 microgrids with the same level of diversity. By pretraining our hypernetwork on 20 varied source tasks, we hope encode enough knowledge applicable the new target tasks to make few-shot transfer learning possible. We will refer to such pretrained hypernetwork Few-Shot PFH.
RESULTS AND DISCUSSION
5.1 PFH ACCELERATES LEARNING IN MEDIUM DIVERSITY MICROGRID CLUSTERS
Fig. [ shows average daily profit gained by each microgrid in microgrid cluster with 10, and 20 microgrids, with varying amounts of diversity. The middle column of Fig. 2 shows PFH is more efficient and profitable for microgrid cluster than microgrid cluster under FedAvg or local control scheme. As shown in Table PFH results in up to $8, 500,0000 of additional cumulative profit after 10,000 days over the local control baseline a microgrid cluster with 20 microgrids. However, this advantage does not carry over to cases of small or large diversity. For less diverse scenarios, PFH was comparable or less profitable than FedAvg or local control. For more diverse scenarios, local control was generally more profitable. The number of microgrids in microgrid clusters also did not seem to have much effect on learning speed here.
5.2 FEDAVG RECOVERS LOCAL PERFORMANCE AT BEST
Curiously, our results indicated that FedAvg presented did not improve the management of a mi- crogrid cluster over collection of local agents. We had expected FedAvg to perform better in the homogeneous case, and to scale with the number of agents, but neither effect appears in our results. Although FedAvg may perform well in supervised learning (MeMahan et al. (017). it may not ex- tend well to RL. We explain FedAvg's 's poor performance as follows: unlike supervised learning, RL requires exploration and already suffers from non-lID data. When aggregating learning across different heterogeneous environments, this issue of learning from non-lID data may have been ex- acerbated, slowing down learning. Furthermore, the setting of federated learning may have made the RL algorithm more sensitive to hyperparameters, as the set of hyperparameters that works for all tasks is likely smaller than the set that works for any one task. We conducted an extensive hyperpa- rameter sweep that is documented in Appendix E to account for this issue with combining federated

---

Under review as a conference paper at ICLR 2023
small o
medium a
large o
PFH
600
FedAvg
Mean Microgrid Profit
5 agents
Local RL
No RL
200
600
10 agents
200
600
20 agents
200
13000
7500
10000
5000 10000
Time (days)
mm Mm
Tox
Microgrid Diversity
Figure 2: RL Agent Performance: The performance of the RL price-setting agent as a function of the number and diversity of the microgrids in the microgrid cluster Performance is measured by looking at the average daily profit gained by each microgrid.
learning with RL. Mcanwhile, the hypernetwork is able to learn how to build RL policies that less sensitive to these hyperparameters because it outputs agents personalized to each task.
5.3 PFH ENABLES FEW-SHOT LEARNING
Fig 3 A shows the hypernetwork adapted to a set of microgrid management tasks extremely quickly. On average, within 1.5 months (42 days), each new microgrid achieved $380 in daily profit, which is about the daily profit of the local agents baseline after 13 years (5000 days) of train- ing. The original, randomly initialized PFH required 3000 days to achieve similar performance. Thus, Few-Shot PFH achieved 119x speedup over local agents and 71.4x ized PFH over the first 1.5 months. Within 7 months (210 days), Few-Shot PFH achieved a daily profit of $565: 44% higher profit than the local agents ever achieve. A randomly initialized PFH required Re 22 years (8000 days) to achieve similar performance: a 38x speedup in the first 7 months of training. Cumulatively, having pretrained PFH on 20 microgrids saves $1,500,000 over the course of training on the new microgrid management tasks compared to randomly initialized PFH.
randomly initial-
5.4 FEW-SHOT LEARNING CAPABILITY SCALES WITH MICROGRID CLUSTER SIZE
When we tried the same experiment with hypernetworks that were trained for 10,000 days on microgrid management tasks and 10 tasks in Fig. B and tested on 5 and 10, respectively, we saw significantly smaller boosts in the mean reward over groups of new tasks with fewer training tasks. The smaller scale of benefit was expected given multi-task learning strategy with fewer source tasks and data. Indeed, when trained on tasks, there was hardly any initial training speedup. Starting from 10 tasks, we observed a large initial boost (although not as large with 20.) Rather strikingly, Few-Shot PFH pretrained on and 10 tasks converged to lower reward curves than even the baseline PFH (i.e., randomly initialized hypernetwork.) With 20 tasks, training speed and no adverse impact on long term training. We hypothesize the fewer microgrid source tasks provided, the more information stored in the environment embedding, which makes
both large initial boost in
8

---

Under review as a conference paper at ICLR 2023
Mean Microgrid Profit (9)
A
Nutrared PM B
Local Beseiin Soselin PPM
600
400-
Pretrained
Peetrained
Pretrained
200
Local Baseline Besaliww Pirt
1000
3000
2500
7500
10000
Time (days)
Time (days)
C
Can you help manage microgrids?
Figure 3: PFH Enables Few Shot Learning: A. Mean microgrid profit of PFH pretrained on 20 microgrids learning manage 20 new microgrids ("Pretrained PFH"), compared to randomly initialized PFH ("Baseline PFH") and the local agents baseline ("Local Baseline"), over training days on the new microgrids. on new set of microgrids, over longer time than A. C. A plausible scenario in which PFH may need quickly adapt to new microgrids.
Mean microgrid profit of PFH pretrained on 10, and 20 microgrids
the hypernetwork brittle to new environments. Thus in the and case, net has not learned enough shared dynamics in the other parameters to generalize to settings. In the case of 20 and above, we expect that enough shared dynamics are learned that the net can generalize. The range of training speed benefits observed suggested the potential in some configurations for Few-Shot PFH to quickly adapt to new tasks depends on how many tasks it was initially trained on.
6 LIMITATIONS AND FUTURE WORK
Technically, our work is limited in several ways. We presented "goldilocks" zone in which PFH outperforms other methods, but as we tested only in simulation, it is unclear where this goldilocks zone would appear in the real world. Second, we protect privacy by only communicating parameters, but it is possible to reconstruct data from parameters for some models (Carlini et al. (020)).
We would like to address these issues in future work. First, we would like explore other environments to determine whether the "goldilocks" phenomenon is unique to the MicrogridLearn environment. Second, we would like to combine our PFH training procedure with differential pri- vacy measures like those in Abadi et al. (2016) to further impede reconstruction of training data.
For two more extended examples of future work we are excited to test, as well as further discussion on potential societal impact of the application of our algorithm, please see Appendix C and D
CONCLUSION
We seek a privacy preserving mechanism for improved training speeds on profit-driven energy ag- gregation in microgrid cluster. To this end, we are the first to demonstrate PFH for RL to output local model gradient updates and show improved training times. We hypothesize that PFH shines when the setting is diverse enough differ meaningfully between systems, but not so diverse that system behavior diverges. We prove our hypothesis and demonstrate the efficacy PFH for few shot learning approaches.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. Federated learning based dynamic regularization, 2021. URL
bttts::/arxiv..om http//awwww...gg////////////
Utkarsha Agwan, Lucas Spangher, William Arood, Tarang Srivastava, Kameshwar Poolla, and Costas J Spanos. Pricing in prosumer aggregations using reinforcement learning. In Proceed- ings of the Twelfth ACM International Conference on Future Energy Systems, pp. 220-224, 2021.
Mohamed Albadi and Ehab El-Saadany. Demand response in electricity markets: An overview. In 2007 IEEE power engineering society general meeting, pp. 1-5. IEEE, 2007.
Aqeel Anwar and Arijit Raychowdhury. Multi-task federated reinforcement learning with adver- saries. arXiv preprint arXiv:2103.06443, 2021.
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary, Fed- erated learning with personalization layers. arXiv preprint arXiv: 191200/912 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1rXiv:1606.01 540, 2016.
N Carlini, F Tramer, Wallace, M Jagielski, A Herbert- Voss, Lee, A Roberts, T Brown, D Song, U Erlingsson, et al. Extracting training data from large language models. arxiv. Preprint posted online December, 14, 2020.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning: A meta- learning approach. arXiv preprint arXiv:2002.07948, 2020.
G. Han, S. Lee, J. Lee, K. Lee, and J. Bac. Deep-learning- and reinforcemen--leaminng profitable strategy of a grid-level energy storage system for the smart grid. Journal of Energy Storage, 41, 2021. doi: 10. 10.10166..es22221110666 102866. URL https: www. scopus. com/
2021. |aw.::/000.aaaaaaaaaaaaaaaaaa3.3 cited By
Briland Hitaj, Giuseppe Ateniese, and Fernando Perez-Cruz Deep models under the gan: Infor- mation leakage from collaborative deep learning, 2017. URL htt.ttss///xiii.rr//////
1742.07464]
Doseok Jang, Lucas Spangher, Selvaprabu Nadarajah, and Costas Spanos. Decarbonizing buildings via energy demand response and deep reinforcement learning: The deployment value of supervi- sory planning and guardrails. Available SSRN 4078206, 2022.
Dohyun Kwon, Joohyung Jeon, Soohyun Park, Joongheon Kim, and Sungrae Cho. Multiagent ddpg- based deep learning for smart ocean federated lcarning iot networks. IEEE Internet of Things Journal, 7(10):9855-99)) 2020.
Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning, 2019, URL https: : arxiv. org/
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pp. 6357- 6368. PMLR, 2021.
Xiao Hui Li and Seung Ho Hong. User-expected price-based demand response algorithm for a home-to-grid system. Energy, 64:437 449, 2014.
Paul Pu Liang, Terrance Liu, Liu Ziyin, Nicholas B Allen, Randy P Auerbach. David Brent, Ruslan Salakhutdinov, and Louis Phillpph Morency. Think locally, act globally: Federated learning with local and global representations. arXiv preprint 2:X15.2001.01523, 2020.
10

---

Under review as a conference paper at ICLR 2023
Nian Liu, Xinghuo Yu, Cheng Wang. Chaojie Li, Li Ma, and Jinyong Lei. Energy-sharing model with price-based based demand response for microgrids of peer-to-pect prosumers. IEEE Transactions on Power Systems, (2(5):3569-33555 3583, 2017.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed coperraive-commpeitive environments. Advances in neural informa- fion processing systems, 30, 2017.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication--fficcine learning deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273- 1282. PMLR, 2017.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. feature leakage in collaborative learning, 2018. URL Betpss//arxxw..orr/bbb////
04049
Chetan Nadiger, Anil Kumar, and Sherine Abdelhak. Federated reinforcement learning for fast personalization. In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 123-127. IEEE, 2019.
Jiaju Qi, Qihao Zhou, Lei Lei, and Kan Zheng. Federated reinforcement learning: techniques, applications, and open challenges. arXiv preprint arXiv:2 /887, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International Conference on Machine Learning, pp.
PMLR, 2018.
Jianji Ren, Haichao Wang, Tingting Hou, Shuai Zheng, and Chaosheng Tang. Federated learning- based computation offloading optimization edge crapputingsuupporte internet of things. IEEE Access, 7:691 1949194-99 69201, 2019.
David Rolnick, Priya L Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, Andrew Slavin Ross, Nikola Milojevic-Dupont Natasha Jaques, Anna Haldman-Brownn et al. Tackling climate change with machine learning. ACM Computing Surveys (CSUR), 55(2):1-96, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v:7777 2017.
Matthias Seeger. Gaussian processes for machine learning. International journal of neural systems, 14(02))69--10) 106, 2004.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks. In International Conference Machine Learning, pp. 9489 -5029 PMLR, 2021.
Amin Shojacighadikklaei, Arman Ghasemi, Kailani R Jones, Alexandru G Bardas, Morteza Hashemi, and Reza Ahmadi. Demand responsive dynamic pricing framework for prosumer dom- inated microgrids using multiagent reinforcement learning. In 2020 52nd North American Power Symposium (NAPS), pp. 6. IEEE, 2021.
Reza Shokri and Vitaly Shmatikov. Privacy-pree preecccpreeervingg deep learning. In Proceedings of the 22nd ACM SIGSAC Conference Computer and Communications Security, CCS '1,, pp. 1310-1321 New York, NY, USA, 2015. Association for Computing Machinery. ISBN 978 9781450338325, doi:
00.1145/2810103.22 266677.. 3687. URL httttt:://deee or0/10. 1 01 03. 281 3687
Lucas Spangher. Transactive multi-agent reinforcement learning for distributed energy price local- ization. In Proceedings of the 8th ACM International Conference Systems for EneeryyEfficient Buildings, Cities, and Transportation, pp. 244-245, 2021.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. ceooppsecomooiitoon networks for cooperative multi-agent learning. arXiv preprint arXiv: 1706.0526660 2017.
11

---

Under review as a conference paper at ICLR 2023
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau velopes. Advances in Neural Information Processing Systems, 33:21 394 111111144422.. 2020.
José R Vázquez-Cantelh Jérôme Kämpf, Gregor Henze, and Zoltan Nagy. Citylearn vl. O: An openai gym environment for demand response with deep reinforcement learning. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficien Buildings, Cities, and Transportation, pp. 356-357, 2019.
Hao Wang and Jianwei Huang. Incentivizing energy trading for interconnected microgrids. IEEE Transactions on Smart Grid, 94))2647722577 2016.
Hao Wang. Zakhary Kaplan, Di Niu, and Baochun Li. Optimizing federated learning non-iid data with reinforcement learning. In IEEE INFOCOM 2020-IEEE Conference on Computer Commu- nications, pp. 1698-1707. IEEE, 2020a.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaza- cni. Federated learning with matched averaging, 2020b. URL htLp::s////// 2002.06444
Xiaofei Wang, Chenyang Wang. Xiuhua Li, Victor CM Leung, and Tarik Taleb. Federated deep reinforcement learning for internet of things with decentralized cooperative edge caching. IEEE Internet of Things Journal, (10):9441-9455 2020c.
Lulu Wen, Kaile Zhou, Jun Li, and Shanyong Wang. Modified deep learning and reinforce- ment learning for an incentive-based demand response model. Energy. 205: 18019, 2020. ISSN 0360-5442. doi: https:/dboorr//100.0bf....or....
URL httpp::/wwww
sciencedirect. com http://c........oooo.o//.././///
Chulin Xie, Keli Huang, Pin-Yo Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against feder-
ated learning. International Conference on Learning Representutions, 2019.
Xing Xu, Rongpeng Li, Zhifeng Zhao, and Honggang Zhang. The gradient convergence bound of federated multi-agent reinforcement learning with efficient communication. arXiv preprint arXiv:2103.13026 2021.
Kaiqing Zhang, Zhuoran Yang, Han Liu, Tong Zhang, and Tamer Basar. Fully decentralized multi- agent reinforcement learning with networked agents. In International Conference Machine
Learning, pp. 5872-5881. PMLR, 2018.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Decentralized multi-agent reinforcement learning with networked agents: Recent advances. Frontiers of Information Technology Electronic Engineering, 2266)8802-814, 2021a.
Michael Zhang, Karan Sapra, Sanja Fidler, Serena Yeung, and Jose M. Alvarez. Personalized fed- erated learning with first order model optimization, 2020. URL http:::://arxii..rrr//eee/e 201 2. 08565
Weiting Zhang, Dong Yang, Wen Wu, Haixia Peng, Ning Zhang, Hongke Zhang, and Xuemin Shen. Optimizing federated learning in distributed industrial iot: A multi-agent approach. IEEE Journal Selected Areas in Communications, 39( 39(12):3688-371 3668377333 202Ib..
A RELATED WORK
We position our literature within an ecosystem of work related transactive pricing microgrids. A price-setting RL agent first shown to help energy aggregator improve demand response and generate a profit (Agwan et al. 2021 Since then a number of works have explored the issue
(Shojaeighadikolaei et al.) 2021 Wen al. 2020 Han al. 2021 Rolnick al. 2022 with some work exploring different configurations of RL.
We wish to provide example of federated learning. "Distributed Selective Stochastic Gradient De-
scent". Le. DSSGD (Shokri & Shmatikov 2013). is an interesting example which deserves further exploration from the interested reader. DSSGD has each local model exchange select parameters and
12

---

Under review as a conference paper at ICLR 2023
gradient updates with the central server. In contrast, FedAvg (McMahan al. 2017) just averages all local model gradient updates and synes all local model parameters. In personalized federated learning, there have been techniques other than PFH to facilitate the federated learning of person-
alized models such as Moreau envelopes T Dinh et al. (2000), multi-task learning [i et al. 2021 personalization layers Arivazhagan al. 20T9), and local representations/ Liang al. (2020) How-
ever, Shamsian et al. (2021) compared PFH against several of these algorithms and performed better
in supervised learning.
Existing multi-agent environments are often solved through multi-agent RL algorithms like MAD- DPG (Lowe et al., 2017). VDN (Sunehag et al.) 2017). and Q-Mix (Rashid al. (2018), but these aggregate data from all the agents onto one central machine during training, and take advantage of joint action-values from all agents. Other works use federated hypernetworks for multi-task setups, but specifically not those in RI.
The combination of the two fields, federated multi-agent reinforcement learning, has focused mainly on learning global models, not personalized models for heterogeneous tasks Qi et al. (2021): Wang al. (2020c); Ren et al. (2019): Anwar & Raychowdhury (2021); Kwon al. (2020): Zhang et al. (202Tb); Xu et al. (202 Wang et al. (2020a); Nadiger et al. (019)) Decentralized multi-agent reinforcement learning does learn personalized models Zhang et al. 2221aa 2018). but it may be difficult to scale up decentralized system such that each agent
benefit from the experiences of all the others without large communication costs. This is not as much of an issue for federated learning communication only needs occur between clients and server rather than clients and all their peers. Although decentralized systems have their benefits, we focus mainly on federated systems in this work.
We note that that the transmission of gradients can allow one recreate private data (Xie al. 2019: Hitaj al., 2017 Melis et al., 2018). Thus, while we note that our work guarantees privacy to the extent of other works within the field of federated learning (Wang al.) 2020b) Li al., 2019) Acar al.,
might be to an extent an overstatement, as works have shown
2021 Zhang et al., 20200). one should apply the term urivacy-preservee with the same caveats to our work as the rest of the field.
B REPRODUCIBILITY STATEMENT
B. OVERVIEW
We wish to provide all details sufficient for the reader to reproduce our experiments. We will overview how setup the code, which we provide in a zip file for analysis, and discuss hyper- parameter selection.
B.2 CODE SETUP
First, download and extract the anonymized code provide in the submission.
Install dve (with google drive support). On linux this is pip install
dvc [gdrive]"
Install Docker, if you have not already.
Run
python3 dve remote add -d gdrive gdrive://
Run python3 -m dve pull
Run pip install requi reguntement.cro
.
Run docker build
B.3 HYPERPARAMETER SELECTION
We select hyperparameters for each algorithm by conducting hyperparameter sweeps, with parame- ters proposed by Bayesian hyperparameter optimization (Seeger 2004) trying maximize the mean reward across all agents. For the local agent baseline we run sweeps, one for each level of diver- sity, each level has different distribution of microgrids. For FedAvg and PFH, we conducted
13

---

Under review as a conference paper at ICLR 2023
sweeps, varying diversity (simple, medium, complex), and the number of agents (5, 10, 20). The number of agents is relevant to FedAvg and PFH because they learn from data across multiple agents. Each local agents and FedAvg sweep had 50 runs, and each PFH sweep had 100 runs, because PFH had Ra double the number parameters to optimize.
To minimize the effect of outliers, run from each sweep. Detailed parameter bounds for hyperparameter sweeps can be found in the appendix in Table 2 and simple analyses of the hyperparameters can be found in Appendix E.2
the hyperparameters from the third highest performing
C MORE FUTURE WORKS
C.O.1 "COST OF PRIVACY"
We wish to further investigate the "cost" of privacy in terms of the negative impact it may have on training time and thus on cumulative aggregator profit. order to create a true apples-to-apples comparison, would need mechanism that aggregates information across microgrids in suitable way. Some ideas on this front include multi-agent RI. that shares the critic but personalizes policies, and hicrarchical RL with a global aggregator.
C.0.2 VERTICAL INTEGRATION OF THE HIERARCHY
In the future, PFH may enable further exploitation of the hierarchical nature of price setting for energy demand response. The energy grid can be imagined responding to energy prices set by microgrids, which respond to energy prices set by city utilities, which respond to energy prices by state utilities, etc. In the future, may have loT devices adjusting demand to energy prices set at the building level. At any level of the energy grid, the task the same: set prices for agents beneath you elicit demand response. In this work have only looked at one level of this energy hierarchy, but the methods we have used could be applied to other layers of the hierarchy as well, and even multiple levels of the hierarchy. One could imagine hypernetwork that learns from price-setting agents at every level of the hierarchy, and can be used to rapidly initialize agents to manage any new entrants to the energy grid, all while preserving privacy at different levels of the tree.
hierarchical tree, with buildings
D DISCUSSION OF SOCIETAL IMPACT
What are potential negative societal effects of our work? Overall, negative effects to prosumers are limited, as the focus of our work is in protecting consumer information. Furthermore, prior work demonstrated that the presence of an aggregator consistently reduced energy costs for consumers.
However, persistent danger of AI is that it is often deployed through centralized profit-scekers. Our work is no different this regard. Although our specific innovation protects prosumers, it may improve the economic viability of profit-seeking entity whose scale may eventually enable it to further its own profit at the expense of prosumers.
Also, the act of setting prices in systems may raise faimess concerns. If initial training microgrids are biased towards wealthier residents, the PFH may initialize new policies with pricing that benefits consumption habits of wealthier clients but not poorer clients. A vivid illustration may be seen in the types of prosumers who are best poised to benefit from economic aggregation: prosumers with large solar panels and batteries are able to shield themselves from profit off of high prices by consuming their own energy, and may fully charge their batteries when prices are low. Prosumers with smaller or no storage capabilities do not have this luxury, and thus are more vulnerable the negative effects of price fluctuation.
E APPENDIX FOR HYPERPARAMETER EXPLORATION
F. HYPERPARAMETER SWEEP SPECIFICATIONS
See Table for the bounds of the hyperparameter sweeps that performed.
14

---

Under review as a conference paper at ICLR 2023
Table 2: Hyperparameter Sweep Bounds All sweeps swept over the first hyperpa- rameters. AFL sweeps additionally swept over the "AFL # of Local Steps" parameter. PFH sweeps additionally swept over the last 7 parameters. Due the high dimensional- ity of the sweep, used Bayesian hyperparameter optimization. We refer to four types of distributions: Uniform, Int Uniform, Log Uniform, and Int Log Uniform. The "Int" type distributions simply quantize the underlying distribution (e.g. if x is sampled from uniform distribution. [x] is returned by an int uniform distribution). The Log Uniform distribution samples uniformly over the log of the value, so the probability of sampling e is the same as e etc.
Parameter
Lower Bound
Upper Bound
Distribution
Batch Size
16
le
Int Uniform Log Uniform
Learning Rate
e-8
# of Hidden Layers
1
7
Int Uniform Int Log Uniform Uniform Int Uniform
of Neurons per Hidden Layer
[e]
PPO Clipping Param
0.01
1.0
PPO of SGD Iterations
30
AFL # of Local Steps
2
[c0]
Int Log Uniform
PFH Dropout
e
10
Log Uniform
PFH Embedding Dim
512
Int Uniform
PFH L2 Regularization
10
Log Uniform
PFH Learning Rate
Log Uniform
PFH of Hidden Layers
Int Uniform
PFH # of Neurons per Hidden Layer
1024
Int Uniform
PFH Local Steps
100
Int Uniform
E.2 REGRESSION EXPLORATIONS OF HYPERPARAMETER SWEEPS
We present, for the reader's interest, a regression fit on the hyperparameters that were swept over. In this regression, each observation is single run of the sweep, the dependent variable in both is the reward mean of the learning trajectory, and the independent variables are those listed in the rows. We believe that this regression contains some interesting information; specifically the direction of coefficients (i.e. whether they are negative or positive) and on which parameters were significant in producing positive reward. We note that the basic assumption of linear regression, that observations are sampled IID from distribution, is not the case here; observations are loosely dependent on each other as the parameter configurations each batch are determined by the performance of parameters in the previous batch. Thus, we relegate these results as a curiosity for supplementary material only. We are more confident in the negative results of this regression, i.e. which variables are insignificant after controlling for the others, than the positive results, as this indicates parameters that the sweep chose not to focus on. We believe that further work in regressions of hyperparameter values may be an interesting research endeavor for understanding ML models as well as for ML applications like AutoML.
E.2. FULL REGRESSION MODEL (TABLE
Of specific interest in the full regression are which hyperparameters did and did not effect the aver- age reward. Many variables in the hypernetwork itself do not seem to matter: the hypernetwork ss learning rate, number of layers, and L2 regularization did not matter. However, whether or not the hypernetwork selected for dropout did matter, and it hurt the performance, implying that hypernet- work fitting was more important than robustness. Some parameters of the PPO agents, such as the clip parameter number of gradient updates, did not matter much either.
15

---

Under review as a conference paper at ICLR 2023
E.2.2 REDUCED REGRESSION MODEL WITH ONLY SIGNIFICANT COEFFICIENTS INCLUDED (TABLE 4)
We believe that this test is more interesting for examining the direction of significant variables after controlling for the other variables. Here, it is interesting to note greater sizes of policy networks, ("sizes"), has negative effect. while number of layers of policy networks have positive effect, offering mixed view on whether policy complexity important. The learning rate is negatively important, implying that more stability in network is preferred. The number of local model update steps allowed is positively correlated mean reward, implying that the more the local models are allowed to fit, the better. The combination of negative effeet of batch size and positive effect of learning rate implies that the local RL agents found it easier to take few steps (lower batch sizes) but update policies at more conservative rate (lower learning rates), which makes sense in the RL context.
16

---

Under review as a conference paper at ICLR 2023
Table 3: Full regression model
Dep. Variable:
Avg Reward OLS
R-squared:
0.362
Model:
Adj. R-squared:
0.348
Method:
Least Squares
F-statistic:
24.62
Date:
Thu, 19 May 2022
Prob (F-statistic):
1.08e-43
Time:
08: 16:34
-3188.0
No. Observations:
533
AIC:
6402.
Df Residuals:
520
BIC:
6458.
Df Model:
12
coef
std err
t
P> [
[0.025
0.975]
Intercept
344.6188
29.679
11,612
0.000
286.314
402.924
sizes
-1.9453
0.777
-25004
0.013
-3,472
-0,419
l.layers learning rate
5.2696
2.806
1.878
0.061
-0.243
10.782
1835.5718
128.380
-14.298
0.000
-0877.799
1583.364
ppo cli param hnet Ir
49.4402
38,009
1.301
0.194
-25.230
124.110
77.4179
63.747
1.214
0.225
47.815
202.651
hnet auet_num__u Jocal steps
1.0579
0.150
7.043
0.000
0.763
1.353
ppo_num wwm.nummsgg sum_sgd_iim iter
-0,3889
0.741
-0,525
0.600
-1.845
1.068
mmet_num_layer
-2.3439
2.571
-0,912
0.362
-7.395
2.708
batch_size
-0.6685
0.171
-3,910
0.000
-1.004
-0.333
hnet_ene werteembdddin..e dim
0.0077
0.028
0.271
0.786
-0.048
0.063
hnet 12 reg hnet dropout
100.4226
19,912
-0.526
0.599
-99.601
28.636
-68.5849
21.070
-3.255
0.001
-109.978
-27.192
Omnibus:
16.401
Durbin-Watson
1.670
Prob(Omnibus);
0,000
Jarque-Bera (JB):
19.555
Skew: 0.336
Prob(JB):
5.67e-05
Kurtosis:
3.655
Cond. No.
9.08e+03
Table 4: Reduced regression model
Dep. Variable:
Avg Reward
R-squared:
0.357
Model:
OLS
Adj. R-squared:
0.349
Method:
Least Squares
F-statistic:
48.63
Prob (F-statistic):
1.79e-47
Log-Likelihood:
-3190.3
No. Observations:
533
AIC:
6395.
Df Residuals:
526
BIC:
6425.
DE Model:
coef
std err
P>
[0.025
0.975]
Intercept
364.7212
15.515
23.507
0.000
334.242
395.201
sizes
-2.1970
0.627
-3.502
0.001
-3,429
-0.965
n.layers learning rate
5.1216
2.759
1.856
0.064
-0.299
10.542
-1829.2013
127.609
-14.334
0.000
-2079.887
-1578.516
hnet num www.omwmm..laacccccmmmm local_steps steps
1.0482
0.149
7.055
0.000
0.756
1.340
batch size
-0.6409
0.167
-3.811
0.000
-0.969
-0.313
hnet dropout
-65.1701
20.923
-3.115
0.002
-106.273
-24.067
Omnibus:
17.230
Durbin- :atson:
1.669
Prob(Omnibus):
0.000
Jarque- Bera (JB):
19,988
Skew:
0.360
Prob(JB):
4.57e-05
Kurtosis:
3.618
Cond. No.
2.57e+03
17


Reviews:
[{"review_id": "eHQaBKEUha", "reviewer": "Reviewer_a54n", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes the first step to combine personalized federated hypernetworks (PFH) with reinforcement learning (RL) and applies their method in the specific field of price-setting for energy demand response. The proposed method can work when the common centralized training with decentralized execution (CTDE) framework in multi-agent reinforcement learning (MARL) is not applicable. Empirically, this paper demonstrates that PFH enables efficient personalized model learning in multi-task MARL on experiments with synthetic datasets when the heterogeneous tasks have moderate diversity.", "summary_of_the_review": "Though this paper makes the first step to combine PFH with PPO, the combination seems a bit incremental, and all the experiments are only conducted on the environments using synthetic datasets. Besides, the current literature review seems not sufficient. Therefore, I am afraid that this paper is not ready for publication as an ICLR paper.", "strength_and_weaknesses": "Strength\n\n1. This paper is the first to combine PFH with RL.\n\n2. The experiment analyses are sufficient. The authors give detailed discussions on the possible reasons why their proposed method outperforms or is outperformed by baseline methods in different environments.\n\nWeaknesses\n\n1. Novelty: My greatest concern is about novelty. This paper seems to be a bit incremental in the sense that it directly combines the PFH with the proximal policy optimization (PPO) algorithm and then applies it in the specific energy pricing domain. The authors also do not seem to give any discussions about whether there are any technical difficulties when combining PFH with PPO and how they tackle these difficulties. \n\n2. Technical Soundness: Though the proposed method applies the federated learning (FL) framework and trains all the personalized models without the communication of raw data, I still find it a bit overclaiming that the proposed method is for “privacy preservation”. As shown in Figure 1, the gradients of personalized models are sent to the central server. However, several works have shown that privacy leakage is still possible, even when only the gradients are communicated [1,2]. \n\n3. Experiments: \n\n(a)  The proposed method indeed outperforms all the baseline methods when the heterogeneous tasks have medium diversity. However, all the experiments are conducted on synthetic datasets, making the results less convincing. It may be better if the authors can conduct more empirical evaluations on environments using real-world datasets.\n\n(b)  The decentralized MARL method using networked agents also does not require the CTDE framework and can even be trained in a fully decentralized manner [6,7]. Combining with the differential privacy (DP) framework [8] to protect the information in the communication between networked agents, the decentralized MARL method is rigorously privacy-preserving, and can also be applied to the energy pricing domain. I would suggest the authors also include the DP-based privacy-preserving decentralized MARL method as a baseline to make the experiments more convincing.\n\n(c)  Is there any criterion or clue about when the diversity is \"medium\"?\n\n4. Literature Review: Several works have already shown the feasibility of combining FL with RL or MARL [3,4,5], which are missing in the current related work section. I would suggest the authors give more discussions and comparisons with these works.\n\n5. Presentation: Overall, the presentation is clear but some parts of the paper can be further improved. For instance, before the method section, it would be better if a section introducing the setting or the preliminaries of the problem concerned in this paper is given (say, the basic preliminaries of RL). Besides, the sizes of some figures and tables in the experiment section can be reduced to put the related work section from the appendix to the main body of the paper for better readability. Also, it is unclear to me what Eq. (1) means, and it would be better if further explanations are given. For Eq. (2), it may be inappropriate to say that some RL agent aims to maximize “a reward”. In general, the objective of RL is to maximize the long-term expected total rewards instead of an instantaneous reward at some time step.\n\n[1] Melis, L., Song, C., De Cristofaro, E., & Shmatikov, V. (2019, May). Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE symposium on security and privacy (SP) (pp. 691-706). IEEE.\n\n[2] Hitaj, B., Ateniese, G., & Perez-Cruz, F. (2017, October). Deep models under the GAN: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (pp. 603-618).\n\n[3] Xu, X., Li, R., Zhao, Z., & Zhang, H. (2021). The gradient convergence bound of federated multi-agent reinforcement learning with efficient communication. arXiv preprint arXiv:2103.13026.\n\n[4] Kwon, D., Jeon, J., Park, S., Kim, J., & Cho, S. (2020). Multiagent DDPG-based deep learning for smart ocean federated learning IoT networks. IEEE Internet of Things Journal, 7(10), 9895-9903.\n\n[5] Zhang, W., Yang, D., Wu, W., Peng, H., Zhang, N., Zhang, H., & Shen, X. (2021). Optimizing federated learning in distributed industrial IoT: A multi-agent approach. IEEE Journal on Selected Areas in Communications, 39(12), 3688-3703.\n\n[6] Zhang, K., Yang, Z., & Başar, T. (2021). Decentralized multi-agent reinforcement learning with networked agents: Recent advances. Frontiers of Information Technology & Electronic Engineering, 22(6), 802-814.\n\n[7] Zhang, K., Yang, Z., Liu, H., Zhang, T., & Basar, T. (2018, July). Fully decentralized multi-agent reinforcement learning with networked agents. In International Conference on Machine Learning (pp. 5872-5881). PMLR.\n\n[8] Dwork, C., & Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4), 211-407.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "PerMukrqJDY", "reviewer": "Reviewer_D68C", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed applying hypernetwork for personalized federated reinforcement learning. Empirical results show that the proposed method achieves better performance compared to FedAvg.", "summary_of_the_review": "This paper doesn't demonstrate enough novelty and contribution to the field of RL and FL. This is below the standard for ICLR.", "strength_and_weaknesses": "Weaknesses:\n- The novelty of this work is limited. It seems that this work is a simple application of previous works. The problem being studied is also restricted (one specific environment). There's no evidence in the paper that the proposed method works for general RL problems.\n- Significantly missing important related works in both FL / personalized FL / RL.\n- The motivation of this work is unclear. What is the unique challenge in RL that prevents existing personalized FL framework to be deployed (for example, pFedMe [1], Ditto [2], etc.)\n\n[1] T Dinh, C., Tran, N., & Nguyen, J. (2020). Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems, 33, 21394-21405.\n\n[2] Li, T., Hu, S., Beirami, A., & Smith, V. (2021, July). Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning (pp. 6357-6368). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "CR6g2B-Wkr", "reviewer": "Reviewer_C5ih", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work is an application paper that deployed the existing federated learning technologies namely hypernetwork to achieve personalized learning in reinforcement learning across multiple agents. The problem statement is based on a simplified hypothetical situation where multiple communities are powered by renewable energy with battery to assist the grid. With assumption of consumer will adjust their behavior based on the price changes. The training process of system contains two parts, local training, and global aggregation. The global aggregation is done by a hypernetwork where gradient from different agents upload their gradient to update this hypernetwork. ", "summary_of_the_review": "This paper is an application paper which deployed a personalized federated learning reinforcement learning in energy optimization situation. The technical novelty of algorithm itself is not strong which is acceptable for application paper. But the analysis and experiment on the simulation does not support the paper to become a high-quality application paper. More experiment, problem constrains and explanations for the simulation should be included in the paper.", "strength_and_weaknesses": "Strength:\nThe problem formation is clear. The real problem of energy demand between different utility node and consumer behavior is complex. Author made some reasonable simplification to scale down the problem into a manageable format that could be analyzed by simulation. \n\nWeaknesses:\nThe weakness of the paper is experiment part is not strong enough to show advantage of their algorithm. The experiments show your algorithm is not always work, where only a selected scenario it outperformed baseline.\nThe formation for total profit should be explained more since it is based on the process of utility market and consumer behavior. The constrain of consumer behavior would be stated since it is unrealistically to have all the consumer to cook at non-dinner period or not use AC during the middle of day. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject