## 3 Methodology
### Proposed Scheme
#### 3.2.2 Deep Reinforcement Learning-Based Dynamic Threshold  
PPO is one of the DRL agent algorithms that can be utilized for both discrete action and continuous action spaces. PPO was utilized because it outperformed other policy gradient methods, such as Advantage Actor-Critic (A2C) and Actor-Critic with Experience (ACER) [58]. The objective function of the PPO was the following:  
\[L^{CLIP}(\theta)=\hat{\mathbb{E}}_{t}[\min(z_{t}(\theta)\hat{A}_{t},\text{clip }(z_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})] \tag{7}\]  
In the clip term, \(\epsilon\) was a hyperparameter for clipping the probability ratio \(z_{t}(\theta)\). The default value of \(\epsilon\) was 0.2. The clipping effect prevents an excessively large policy update. The details of the algorithms are described in [58]. In our proposed scheme, we used the PPO as a DRL agent for setting dynamic thresholds. For \(\epsilon\), the default value was used. We utilized each agent for each estimation model and denoted our DRL agent as DRL-DT. Action  
For action, we used a continuous action space setting, in which the Gaussian policy was utilized. There were two output nodes in the last layer of the policy function of the DRL-DT: one was \(\mu_{s_{t}}\), and the other was \(\sigma_{s_{t}}\). Sigmoid function was utilized to constrain a range of \(\mu_{s_{t}}\), and Softplus function was utilized for \(\sigma_{s_{t}}\). During training, dynamic threshold values were sampled in the Gaussian distribution with \(\mu_{s_{t}}\) and \(\sigma_{s_{t}}\). \[a_{t}\sim N(\mu_{s_{t}},\sigma_{s_{t}}) \tag{8}\]  
During the test, we used \(\mu_{s_{t}}\) as threshold values because sampling thresholds from distribution caused performance uncertainties. \[a_{t}=\mu_{s_{t}} \tag{9}\]  
State  
The same weather information (irradiance \(D_{t}\), ambient temperature \(T_{t}\), wind speed \(W_{t}\), humidity \(H_{t}\)) was used as the input of the estimation model. This observation provided the agent information necessary for properly setting dynamic thresholds under varying weather conditions. A state vector, representing an observation at a time step, is as follows:  
\[\mathbf{s}_{t}=(D_{t},T_{t},W_{t},H_{t}) \tag{10}\]  
Reward  
We designed a reward function for setting dynamic thresholds to detect inverter faults under the condition that there were no fault labels. Feedback signals from the reward function can play a key role in anomaly or fault detection tasks in systems in which fault events are sparse. Once an estimation model is trained, residual values can be obtained by using an estimated value as follows:  
\[e_{t}=|X_{t}-\hat{X}_{t}| \tag{11}\]  
In the fault detection task, if \(e_{t}\) exceeded a threshold value \(a_{t}(a_{t}\geq 0,\forall a_{t})\), we considered that an inverter fault occurred at the used time step. Therefore, if \(\frac{e_{t}}{a_{t}}\) was larger than, or equal to, 1, it was considered that a fault had occurred. Otherwise, it was considered that the PV systems were operating in a normal way. Our proposed method was for dealing with cases in which fault labels were either sparse or not. For this reason, we trained our agent with only the normal data of the NIST canopy dataset. By utilizing the \(\frac{e_{t}}{a_{t}}\), we could provide feedback to our agent on whether it  
Figure 8: DRL structure for setting dynamic thresholds. chose proper thresholds without fault labels or not. Our designed reward function was as follows:  
\[r_{t+1}= \begin{cases}\beta(\frac{e_{t}+\epsilon}{a_{t}+\epsilon}),&0\leq\frac{e_{ t}+\epsilon}{a_{t}+\epsilon}<1\\ \gamma(-\frac{e_{t}+\epsilon}{a_{t}+\epsilon}),&\frac{e_{t}+\epsilon}{a_{t}+ \epsilon}\geq 1\end{cases}  \tag{12}\]  
We assumed that the training data for the agent was normal. If the agent chose a threshold value, \(a_{t}\), larger than a residual value, \(e_{t}\), the agent received a positive reward, such as \(\beta(\frac{e_{t}+\epsilon}{a_{t}+\epsilon})\). Otherwise, the agent received a negative reward, such as \(\gamma(-\frac{e_{t}+\epsilon}{a_{t}+\epsilon})\).