# Lithium-Ion Battery Prognostics through Reinforcement Learning Based on Entropy Measures
## 4 Methodology
### Predictive Models
#### 4.2.3 Reinforcement Learning  
Reinforcement Learning (RL) is a type of multi-layered neural network, and has become a focus of research in modern artificial intelligence. The concept is based on rewarding or punishing an agent's performance in a specific environment. A state is a description of the environment made to provide the necessary information for the agent to decide at each time step. For each and every state \(s\), the agent has a number of selecting actions \(a\) to make decisions from. A policy is required, based on a cost function, to map each state to the optimal action with the consideration of maximizing its reward function during the episode [33].  
Reinforcement Learning has real-life applications in various fields such as driving cars, landing rockets, trading and finance, diagnosing patients, and so on. This Deep Learning technique differs from supervised learning, as it does not require correct sets of actions and labeled input/output pairs [34]. Instead, the goal is to find a balance between exploration and exploitation. Figure 10 illustrates the schematic of a general Reinforcement Learning structure and its Equations are described as follows.  
\[a_{t} \sim \pi(a_{t}|s_{t}) \tag{15}\] \[s_{t+1} \sim f_{state}(s_{t+1}|s_{t}.a_{t})\] (16) \[r_{t+1}=f_{reward}(s_{t}.\ d_{t}.s_{t+1})\] (17) \[R=\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}\] (18) \[Q^{new}_{s_{t}.\ a_{t}}=Q^{old}_{s.\ a_{t}}+\alpha\left(\overbrace{ r_{t}+\gamma maxQ^{next}_{s_{t+1}.\ a}}^{Target}-\overbrace{Q^{old}_{s_{t}.\ a_{t}}}^{Prediction}\right) \tag{19}\]  
Figure 10: Reinforcement Learning Schematic.  
In this study, we have considered the permutation entropy of the battery voltage as the states and the capacities as the actions, which should be taken at each state based on the given entropy. An algorithm for the RL model is presented in the following.  
```
States:\(s=\{PE_{1}\). \(PE_{2}\). \(\dots\)\(PE_{n}\}\): Permutation Entropy of Battery Voltage; Actions:\(a=\{Capacity_{1}.Capacity_{2}\). \(\dots\)\(Capacity_{n}\}\): Battery Capacity;  Define the optimal policy;  Initialize the parameters \(a\) and \(\gamma\); for\(t\) in range (epoch) do  Calculate \(a_{t}\) using the optimal policy  Determine \(s_{t+1}\) as a function of the state and the previous state and action  Compute \(r_{t+1}\) and \(R\)  Update \(Q_{s,~{}a_{t}}\) using Equation (19)  Evaluate the estimation using the following loss function as in Equations (20)-(22) end
```  
**Algorithm 4** Reinforcement Learning  
The hyperparameters of the proposed models define how they are structured. Optimal hyperparameters are approximated so that the loss is reduced. In other words, we explore various model architectures and search for the optimal values in the hyperparameter space to minimize the resulting performance metrics; for instance, Mean Squared Error. For this purpose, in the three models, grid search is used for tuning the hyperparameters and achieving reliable comparisons between the numerical results from the models. A model is built for each possible combination of all of the hyperparameter values; next, the models are evaluated based on the performance metrics, and then the architecture which produces the best results is selected. The results and findings are reported in the following section.