# Online Learning Algorithms for the Real-Time Set-Point Tracking Problem
## 2 Mathematical Framework of Online Decision Making
From [28], the temperature evolution of a TCL can be represented as follows:

\[\theta_{i,t+1}=a_{i}\theta_{i,t}+(1-a_{i})(\theta_{t}^{a}-q_{i,t}\theta_{i}^{ g}) \tag{3}\]

Here, \(\theta_{i,t}\) is the TCL \(i\) internal temperature at interval \(t\), \(a_{i}=\exp\frac{-\Delta t}{R_{i}C_{i}}\) where R and C are the thermal resistance and capacitance, respectively, \(\theta_{t}^{a}\) is the ambient temperature, \(q_{i,t}\) is the control variable, and finally \(\theta_{i}^{g}=P_{i}^{tr}R_{i}\). We adopt the model so that the energy level can be changed in a continuous range, and this way the acquired temperature can be achieved easier. In addition, this complies more with the structure of the state of art TCLs found in residential departments. We assumed that the air conditioner is working to give the desired temperature; thus, at each interval that TCL is participating in DR, this temperature may be increased or decreased depending on the control signal provided by the central operator, which, in this case, is the aggregator. Thus, the aforementioned loss function is rewritten as given in (4) to be utilized in the online algorithms introduced in the next subsection. \[L_{t}(x_{t})=(s_{t}-\overline{q}^{T}p-(I+\omega_{t})^{T}x_{t})^{2} \tag{4}\]

where \(p=P^{tr}/\eta\) and \(\overline{q}=\frac{\theta^{a}-\theta^{d}}{P^{tr}R}\). ## 4 Solution Methodology

The previous section presented the set-point tracking problem and prepared the optimization model to be utilized within an online decision-making framework. Here, wegive two online algorithms based on the OGD and OFW algorithms. The OGD algorithm acts a the benchmark and represents the online algorithms that require the projections at each step. Opposed to this, we introduce the OFW-based algorithm that utilizes the linear approximation to solve an LP problem at each stage without the utilization of any projections. Both of these algorithms are introduced in Algorithms 3 and 4. ```
Inputs \(x_{t}\), \(a\theta^{d}\), \(\alpha\), \(R\), \(C\), \(P^{tr}\) Define Decision-making set \(\mathcal{K}\) Begin:  Utilize the decision made in the previous time interval: \(x_{t}\)  Realize uncertain parameters: \(s_{t}\), \(\omega_{t}\)  Compute the loss function input parameter: \(\overline{q}=\frac{\theta^{q}-\theta^{d}}{P^{tr}R}\)  Calculate the loss function: \(L_{t}(x_{t})=(s_{t}-\overline{q}^{T}p-(I+\omega_{t})^{T}x_{t})^{2}\)  Compute: \(y_{t}=x_{t}+2\alpha(I+\omega_{t})(s_{t}-\overline{q}^{T}p-(I+\omega_{t})^{T}x _{t})\)  Project and Update: \(x_{t+1}=\prod_{\mathcal{K}}(y_{t})\) End Output: \(x_{t+1}\)
```
**Algorithm 3**OGD-Based Set-Point Tracking Algorithm at Time \(t\)
```
Inputs \(x_{t}\), \(a\theta^{d}\), \(\gamma\), \(\rho\), \(R\), \(C\), \(P^{tr}\) Begin:  Utilize the decision made in the previous time interval: \(x_{t}\)  Realize uncertain parameters: \(s_{t}\), \(\omega_{t}\)  Compute the loss function input parameter: \(\overline{q}=\frac{\theta^{q}-\theta^{d}}{P^{tr}R}\)  Calculate the loss function: \(L_{t}(x_{t})=(s_{t}-\overline{q}^{T}p-(I+\omega_{t})^{T}x_{t})^{2}\)  Compute: \(d_{t}=(1-\rho)d_{t-1}-2\rho(I+\omega_{t})(s_{t}-\overline{q}^{T}p-(I+\omega_{t })^{T}x_{t})\)  Compute LP optimization parameters: \(z_{t}\) and \(A,b\) according to the  decision-making set  Solve LP optimization and calculate solution as \(v_{t}\) accordingly  Update \(x_{t+1}=(1-\gamma)x_{t}+\gamma v_{t}\) End Output: \(x_{t+1},d_{t}\)
```

**Algorithm 4**OFW-Based Set-Point Tracking Algorithm at Time \(t\)

The algorithm given in Algorithm 3 represents the process of deciding the next step for TCL operation at time \(t\). There are some input parameters that are known prior to the start of the optimization interval. These parameters include the decision made in the previous time interval, the TCL parameters, the step size, and the desired temperatures. At the start of the time step \(t\), the \(x_{t}\), which is set in the previous step, is utilized, and the related uncertain parameters are realized afterward. The loss function could be calculated accordingly.