# Online Learning Algorithms for the Real-Time Set-Point Tracking Problem
## 2 Mathematical Framework of Online Decision Making
Therefore, a projection step is performed in each step as well. The OGD-based algorithms are widespread and typically very efficient; however, as explained above, they rely on a projection step whenever a decision is taken outside the domain of interest (i.e., the infeasible decision value), which can limit their potential in multiple applications. The projection step indicates obtaining the closest point inside the decision-making set and could require solving a convex quadratic program each time the optimization variable is decided. In many settings of practical interest, linear optimization can be carried out more efficiently. In this avenue, another type of online algorithm is proposed based on the Frank-Wolfe algorithm, which does not require projection at any step and is also very easy to implement. In the context of the Frank-Wolfe approach, it is assumed that the linear optimization step is computationally cheap compared to the projection counterpart. The linear programming optimization in some applications can simplify the implementation of the whole optimization as well. The online Frank-Wolfe optimization is given in Algorithm 2[28]. ```
Inputs \(x_{1}\in\mathcal{K}\), \(\eta\in\mathbb{R}^{+}\) for\(t=1\) to \(T\)do  Use \(x_{t}\) and observe \(f_{t}



## 3 Set-Point Tracking Modeling

Electric utilities, along with power system operators, consider DR as a practical tool to activate the demand side to participate in grid functions when necessary. When utilized appropriately, demand response could prove to be an economical and sustainable solution benefiting both customers and grids simultaneously. DR can have different attributes, including the DR duration, frequency, and response time. Indeed, the practical implementation of DR could take different time scales, automatizing levels, and response size. For instance, large demand loads can be called by phone by system operators hours before an event of DR (the frequency is low). Opposite to large loads, we have small appliances that are faster, and the frequency is higher but would need to respond autonomously, with small intervals for updating their response (ranging from seconds to a few minutes and a higher frequency of DR). Here, we investigate set-point tracking that requires fast responses from the engaged load in the DR program. We assume an aggregator (central utility) managing certain types of residential appliances that provide ancillary services to the power grid. A regulation signal [29] is received by the aggregator, and the task is to convert this signal into the state decisions of the individual TCLs. Therefore, for each TCL, an adjustment signal is provided at each optimization interval to track a power set-point via the aggregated consumption. Each household appliances' energy consumption is managed by an EMS installed in the customer's home, which receives online control signals from the aggregator. We consider \(N\) flexible loads in the order of hundreds or thousands. The decision variables are the amount of adjustment at each interval, which translates into \(x_{i}\in[\underline{\Delta x_{i}},\overline{\Delta x_{i}}]\). Thus, at each time, the loads can be adjusted to match the set-point signal \(s_{t}\in\mathcal{R}\). We assumed that the response of the loads is under uncertainty. In this regard, the following loss function is chosen to penalize the large deviation from the set-point signal:

\[L_{t}(x_{t})=(s_{t}-(I+\omega_{t})^{T}x_{t})^{2} \tag{2}\]

Here, \(L_{t}\) is the loss function, and \(\omega_{t}\) represents the uncertainty of the load behavior. The responsive loads are considered to be TCLs. We assume the cooling phase (air condition); however, the model can be extended for heating.