# Online Learning Algorithms for the Real-Time Set-Point Tracking Problem
## 2 Mathematical Framework of Online Decision Making
After receiving the complete observation of the loss function, the first stage of updating step is calculated as demonstrated in the algorithm. The result could be outside of the decision-making set. Thus, the final decision is updated and outputted after one projection, which is translated as solving a quadratic program with constraints that define the decision-making set. In Algorithm 4, similar steps are taken in the first part of the procedure regarding the input, realization, and calculation of the loss function. After computing the loss function,the direction variable is calculated accordingly, which is then utilized in solving the given LP problem in (5). \[\min z_{t}^{\prime}x_{t}\] (5a) s.t. \[Ax_{t}\geq b \tag{5b}\] Figure 2.0

Here, \(z_{t}\) is determined by the direction variable, the decision-making set boundaries are translated into \(A,b\) matrices, and the lower and upper bound are already known from the TCL data. The solution of this LP problem is utilized in the next step to determine the value of \(x_{t+1}\). Finally, \(x_{t+1}\) along with \(d_{t}\) are outputted to be used in the next optimization interval. In the next section, we fully investigate and compare the performances of these algorithms and provide hindsight on how to use them in similar decision-making problems to their full potential. ## 5 Numerical Study

In this numerical study, we present three different cases:

**Case I:** In this case, we assume that the fluctuation of set-point tracking is small and that there is no complicating constraint that defines the feasibility set for decision-making. **Case II:** In this case, the set-point signal fluctuation range is higher, which makes the tracking more difficult and less predictable. **Case III:** Finally, in this case, not only is the fluctuation high but also there are constraints that further complicate the decision-making set. The numerical experiments were performed in Python, and CVXP solvers were utilized where necessary to solve the optimization problems (linear and quadratic optimization problems). ### Case I

For this case, we assumed that \(s_{t}=\frac{155N}{100}+sin(0.1t)\) and \(w_{t}\sim N_{[-0.1,0.1]}(0,0.5)\), a truncated Gaussian variable, models the TCLs' response uncertainty at each optimization interval. This uncertainty is related to the limitation of temperature modeling, for instance, the impact of the radiant house heating from the sun or windows being open, etc. The TCL parameters are sampled uniformly from the thermal parameters depicted in Table 1, and the desired temperature is uniformly sampled in the range of 20 \({}^{\circ}\)C to 25 \({}^{\circ}\)C for all loads. The ambient temperature is supposed to be fixed to 30 \({}^{\circ}\)C considering the fact that the operation time is generally limited, and this temperature, on average, does not vary much. Both OGD and OFW algorithms are applied to the set-point tracking problem as explained in the previous section. The loss function is calculated for the case where \(N=100,T=100\), and its evolution is depicted in Figure 3 with a logarithmic y axis in Figure 2.1

As depicted in these figures, the loss starts to decrease after a few steps of optimization. This decrease is quicker in the OFW case, but both approaches depict a similar behavior where, in some steps, OGD decreases the loss function even further. Next, the regret analysis is conducted in Figure 2.2 We can see that both algorithms attain a sub-linear regret, which is an important goal when dealing with an online optimization framework, meaning \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline R & 1.5–2.5 \({}^{\circ}\)C/kW \\ C & 8–12 kWh/\({}^{\circ}\)C \\ \(P^{tr}\) & 10–18 kW \\ \(\eta\) & 3.5–5.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: TCL Parameters. that, when the time increases, the average regret tends to zero or, in other words, online learning is successfully carried out. Figure 4: Comparison of the loss values for each method with a logarithmic \(y\) axis.