## 3 Demand-Response Aggregator Frameworks in Future US Electricity Markets
### Uncertainty in Prosumer-Aggregator Game-Theoretic Frameworks
The most important aspect of decision-making under uncertainty in the aggregator's problem is adaptivity. In energy trading, the players have to be able to adapt to changes in prosumers' demand (highly uncertain at the residential level [117]), price elasticity, dis-utility, players' strategy (i.e., dishonesty), and changes in the market side that affect players' payoffs, i.e., introduction of a new player, change in a player's strategy, etc. Reinforcement learning (RL) is a popular adaptive method where past knowledge is used (exploitation of learning), in combination with the exploration of actions to further maximize the players' payoffs, and this has not yet been implemented for the DR aggregator. However, DR approaches related to consumers and utilities have been proposed as bi-level non-cooperative Stackelberg-Nash games [225]. In another RL approach, [PERSON]. [226] proposes a Markov decision process (MDP) for aggregators to learn better pricing strategies. This approach is based on Q-learning, assuming competition between aggregators, but it lacks a game theoretic approach and does not consider prosumer modeling (no prosumer flexibility). It has been stated that decision-making under uncertainty for DR aggregators, especially under partially observed environments, is still an open question [227]. Game-theoretic approaches under uncertainty for the DR aggregator problem are limited, as mentioned by the two literature review papers [227, 228]. Additionally, the problem of incomplete information regarding the prosumer's energy-selling or energy-purchasing patterns was not addressed properly [229] and remained a roadblock until [230] proposed an RL mechanism, in which an EMS (with a role similar to a DR aggregator) coordinates the energy exchange between the utility and prosumers, modeling them in a competitive way via uncertainty quantification and solving it in a combined manner. The scalability of the solution was proven to be satisfactory with the number of prosumers, while heterogeneous prosumer behavior was also modeled. In another work, an RL-based DR strategy is designed for the DR aggregator to learn the best charging decisions for prosumer batteries based on market reward outcomes (no privacy and minimum prosumer flexibility modeled), utilizing deep Q-networks. Scalability was not addressed, but it is dependent on the state and action's dimensionality. Lastly, a Q-learning framework in [231] allows customers to engage in a cooperative game with the utility, which performs both DR and economic dispatch (ED), while Stackelberg equilibria are found with the help of a pre-trained neural network. The latter provides evidence that learning Stackelberg game equilibria with RL is possible but does not specifically consider the DR aggregator problem. RL can provide model-free frameworks, i.e., frameworks without a specific utility function, opening paths for adaptivity in the prosumer-DR aggregator games while playing close to the equilibrium. Before any learning can be applied, adaptivity must be ensured for the algorithm. Online metrics such as regret minimization [232] have been used in playing two-player zero-sum Nash games [233] in an adaptive fashion, where the difference between the total payoff under uncertainty and the total best payoff is minimized in a horizon of N games. In [234], players can play non-zero-sum Stackelberg games with uncertainty in the follower's action, where the leader is trading off the estimated follower's action with the true value revealed. In summary, Bayesian approaches require data to construct distributions for the uncertain variable, online learning does not require any a priori knowledge and learns on the fly, and online adaptivity combined with RL methods can trade-off exploration and exploitation compared to robust methods. Decision-making frameworks under uncertainty for prosumer DR aggregators are still an open question due to various challenges and very limited market data access.