## 3 Method
### Decision Policy Design  
An RL-based VESS operation strategy determines the decision rule of the current action among the feasible action ranges at each stage in the state-action space. The decision rule is designed to maximize the reward provided by the VESS operation. The goal of the VESS operation is to minimize the forecast uncertainty presented in the objective function in (9). At stage \(t\), the operator is unaware of the uncertainty of forward time over \(t\). Therefore, the uncertainty included in the VESS operation at stage \(t\) is expressed as  
\[ \begin{array}{rl}\mathcal{O}_{t}(a_{t}\mid\varepsilon_{t})&=\frac{1}{T}\sum \limits_{m=t}^{T}\sum\limits_{u\in\mathcal{U}}\left|\varepsilon_{m}^{u}\right| \\ &=\frac{1}{T}\sum\limits_{u\in\mathcal{U}}\left|\varepsilon_{t}^{u}\right|+ \frac{1}{T}\sum\limits_{m=t+1}^{T}\sum\limits_{u\in\mathcal{U}}\left| \varepsilon_{m}^{u}\right|\\ &=\frac{1}{T}\sum\limits_{u\in\mathcal{U}}\left|\varepsilon_{t}^{u}\right|+ \mathcal{O}_{t+1}(\hat{a}_{t+1}\mid\hat{\varepsilon}_{t+1}),\end{array}  \tag{14}\]  
where the values with a hat represent the expected values. As shown in Equation (14), the uncertainty comprises the accumulated uncertainty of the current and expected uncertainties at the future time. In the RL approach, the current uncertainty performance of the VESS operation is defined as the instantaneous reward value from the current decision action at each stage. The reward at state \(t\) is presented as  
\[r_{t}=\frac{1}{T}\sum\limits_{u\in\mathcal{U}}\left|\varepsilon_{t}^{u}\right|. \tag{15}\]  
Moreover, the accumulated uncertainty is defined as the return that is the accumulated reward from time \(t\) onward, and is calculated as  
\[ \begin{array}{rl}R_{t}&=r_{t}+\gamma r_{t+1}+\cdots+\gamma^{T-t}r_{T}\\ &=r_{t}+\gamma R_{t+1},\end{array}  \tag{16}\]  
where \(\gamma\) is the discount factor, \((0,1]\), which reduces the risk of the expected value from the onward decision time. The return in Equation (16) becomes the weighted uncertainty performance of the VESS operation in Equation (14). The RL-based decision-making approach is used to determine the VESS operation action to minimize the reward, which is the uncertainty performance. For this, the state-action value function is defined and presents the quality of an action, \(a_{t}\), at a given state \(s_{t}\), as follows  
\[ \begin{array}{rl}Q(s_{t},a_{t})&=\mathbb{E}[R_{t}|s_{t},a_{t}]\\ &=\mathbb{E}[r_{t}+\gamma Q(s_{t+1},a_{t+1})|s_{t},a_{t}].\end{array}  \tag{17}\]As the transaction probability of an action at each state is known, \(\pi=\Pr(a_{t}|s_{t}),\ \forall\ t\in\mathcal{T},s_{t}\in S,a_{t}\in A_{t}\), the optimal state-action value function \(Q^{*}(s_{t},a_{t})\) is measured using the Bellman optimality equation [34]  
\[ \begin{array}{ll}Q^{*}(s_{t},a_{t})&=\mathbb{E}[r_{t}+\gamma\min_{a_{t+1}\in A _{t+1}}Q(s_{t+1},a_{t+1})|s_{t},a_{t}]\\ &=\mathbb{E}[r_{t}+\gamma Q^{*}(s_{t+1},a_{t+1})|s_{t},a_{t}].\end{array}  \tag{18}\]  
and the optimal action is determined as  
\[ \begin{array}{ll}a_{t}^{*}&=\underset{a_{t}\in A_{t}}{\arg\min}Q^{*}(s_{t}, a_{t})\\ &=\underset{a_{t}\in A_{t}}{\arg\min}\mathbb{E}[r_{t}|s_{t},a_{t}]+\gamma Q^{* }(s_{t+1},a_{t+1}).\end{array}  \tag{19}\]  
However, it is impractical to attempt to determine the transaction probability of an action at each state. In the RL approach, the state-action value function is estimated by learning. In widely used Q-learning-based RL approaches [26; 27; 28; 29], the state-action value function is estimated as  
\[Q^{QL}(s_{t},a_{t})\leftarrow(1-\alpha)Q^{QL}(s_{t},a_{t})+\alpha\bigg{[}r_{ t}+\gamma\min_{a_{t+1}\in A_{t+1}}Q^{QL}(s_{t+1},a_{t+1})\bigg{]}, \tag{20}\]  
where \(\alpha\) is the learning rate in \((0,1]\). Moreover, the action is determined as  
\[a_{t}^{QL}=\underset{a_{t}\in A_{t}}{\arg\min}\bigg{[}r_{t}+\gamma\min_{a_{t+ 1}\in A_{t+1}}Q^{QL}(s_{t+1},a_{t+1}).\bigg{]} \tag{21}\]  
However, the WPG has a high uncertainty variance [19].