## 3 Method
### RL-Based VESS Operation Strategy  
Data clustering
1:Initialization
2: Set the number of clusters to \(K\). 3: Initialize centroids \(c_{k}\) using historical WPG forecasting data. 4:Data clustering
5: Set cluster \(k\) as \(k=arg\!\min_{k}\sum\limits_{u\in\mathcal{U}}\parallel\hat{g}^{u}-c_{k} \parallel_{2}\). 6: Update \(c_{k}\) including \(\hat{g}^{u}\). 7:Policy learning
8:Initialization
9: Set \(Q^{SARSA}\) as \(Q_{k}\) from \(Q=\{Q_{1},\cdots,Q_{K}\}\). 10: Set \(s_{1}\) as the current SoC and \(A_{1}\) using (13). 11:Policy learning
12:For \(t=\{1,\cdots,T\}\),
13: Set \(d_{t}^{SARSA}\) in \(A_{t}\) using (22). 14: Update \(s_{t+1}\), \(A_{t+1}\) and \(Q^{SARSA}\) using (12) and (23). 15:endfor
```

**Algorithm 1**Proposed RL-based VESS operation algorithm

First, to apply the k-mean clustering, the number of clusters is set to \(K\), and using the historical WPG forecast data, the centroids are initialized to solve (24) (steps 2 and 3). The cluster number \(k\) of data sets is selected as the cluster that has the minimum Euclidean distance to the cluster centroid in step 5. The cluster number is used to select the active state-action value function for the policy learning process. Moreover, the centroid of the selected cluster is updated considering the dataset in step 6. Combined with \(K\) clustering, \(K\) state-action value functions are required. In the policy learning process, the k-th state-action value function is loaded as the active state-action value function, \(Q^{SARSA}\), according to the cluster number \(k\) in step 8. The initial state \(s_{1}\) is set as the current VESS condition and the feasible action range \(A_{1}\) is determined by the current state \(s_{1}\) in step 9. During the operation time horizon \(\mathcal{T}\), the VESS operation action is selected using (22) and the values of the state and the state-action value function are updated according to the selected action in steps 12 and 13. ## 4 Results and Discussion

To verify the performance of the proposed strategy, the simulation results were evaluated. In the simulation, five WPG datasets that were recorded by the National Renewable Energy Laboratory to develop eastern wind resources in the United States of America from 2004 to 2006 were employed [38]. Each WPG had a capacity of 20 megawatt (MW). Day-ahead forecasting data were provided with 1-h time resolution. Therefore, the operation time horizon was set to 24 h, \(\mathcal{T}=\{1,\cdots,24\}\). The simulation results were measured using the data from the first 14 days of December 2006, and the other datasets were used for RL training. Moreover, for policy learning, the learning rate and discount factor were set as \(\alpha=0.1\) and \(\gamma=0.95\), respectively. The cluster size was assumed to be three. However, a discussion of the cluster size is also presented here. A lithium-ion based ESS system was considered, which is widely used with renewable energy systems [39]. The charging/discharging efficiency \(\eta\) was set as, 0.95, which provided a 0.9 round-trip efficiency, and the DoD margin that restricts the minimum and maximum operable ES capacity was 0.1. The ES capacity was expressed as the normalized WPG capacity, that is, per unit (p.u. ), and the service time was 2 h, which provided a 0.5 charging rate (C-rate). The simulations in this study were implemented on a 64-bit PC with a 4 GHz Quad-Core Intel Core i7 CPU and 32 GB RAM, using MATLAB R2020a with an IBM CPLEX optimization studio. ### Performance Results of Proposed Strategy

Figure 3 shows the uncertainty management performance as MAE, with varying VESS size. The black line with circles, the red line with squares, and the blue line with diamonds present the results obtained when applying the optimal solution, the proposed method, and the stochastic method, respectively. The optimal solution is the solution to problem (9), for known information including the future time, and the stochastic method is the VESS operation according to the probabilistic information of the WPG suggested in [40].