# Lithium-Ion Battery Prognostics through Reinforcement Learning Based on Entropy Measures
## 4 Methodology
### Permutation Entropy  
To compute a \(D\) order permutation entropy for a one-dimensional set of time series data with \(n\) data points, the following steps are taken [29]. First, the data is partitioned into a matrix with \(D\) rows and \(n-(D-1)\tau\) columns, where \(\tau\) is the delay time. \[V= \begin{bmatrix}v(1)&v(1+\tau)&\ldots&v(1+(D-1)\tau)\\ v(2)&v(2+\tau)&\ldots&v(2+(D-1)\tau)\\ \vdots&\vdots&&\vdots\\ v(n)&v(n+\tau)&\cdots&v(n+(D-1)\tau)\end{bmatrix}  \tag{1}\]  
After rebuilding the data, \(\pi\) is defined as the permutation pattern for \(V\) columns:  
\[\pi=\{l_{0}.l_{1}.\ldots.l_{D-1}\}=\{0.1.\ldots.D-1\} \tag{2}\] \begin{table}
\begin{tabular}{l l} \hline \multicolumn{2}{c}{**Indices**} \\ \hline \(n\) & \multicolumn{2}{c}{Number of time series data} \\ \(T\) & \multicolumn{2}{c}{Number of times the permutation is found in time series data} \\ \hline \multicolumn{2}{c}{**Variables**} \\ \hline \(x_{t}\) & \multicolumn{2}{c}{Input variable (permutation entropy of battery voltage) at step \(t\)} \\ \(y_{t}\) & \multicolumn{2}{c}{Observed battery capacity at step \(t\)} \\ \(\hat{y}_{t}\) & \multicolumn{2}{c}{Output variable (predicted battery capacity) at step \(t\)} \\ \(h_{t}\) & \multicolumn{2}{c}{Previous state at step \(t\)} \\ \(c_{t}\) & \multicolumn{2}{c}{Current state at step \(t\)} \\ \(\widehat{c}_{t}\) & \multicolumn{2}{c}{Intermediate cell state at step \(t\)} \\ \(i_{t}\) & \multicolumn{2}{c}{Input gate at step \(t\)} \\ \(f_{t}\) & \multicolumn{2}{c}{Forget gate at step \(t\)} \\ \(o_{t}\) & \multicolumn{2}{c}{Output gate at step \(t\)} \\ \(p\) & \multicolumn{2}{c}{Order of auto-regression} \\ \(d\) & \multicolumn{2}{c}{Order of difference} \\ \(q\) & \multicolumn{2}{c}{Order of moving average} \\ \(s_{t}\) & \multicolumn{2}{c}{State at step \(t\)} \\ \(a_{t}\) & \multicolumn{2}{c}{Action at step \(t\)} \\ \(r_{t}\) & \multicolumn{2}{c}{Reward at step \(t\)} \\ \(R\) & \multicolumn{2}{c}{Sum of the rewards} \\ \(\alpha\) & \multicolumn{2}{c}{Learning rate} \\ \(\gamma\) & \multicolumn{2}{c}{Discount factor} \\ \(Q_{s_{t},\ a_{t}}\) & \multicolumn{2}{c}{Q Table for states and actions at step \(t\)} \\ \hline \multicolumn{2}{c}{**Parameters**} \\ \hline \(PE\) & \multicolumn{2}{c}{Permutation entropy} \\ \(D\) & \multicolumn{2}{c}{Order of permutation entropy} \\ \(\tau\) & \multicolumn{2}{c}{Time delay in data series} \\ \(V\) & \multicolumn{2}{c}{Time series data matrix} \\ \(l_{i}\) & \multicolumn{2}{c}{Columns in \(V\)} \\ \(\pi\) & \multicolumn{2}{c}{Permutation pattern} \\ \(P\) & \multicolumn{2}{c}{Relative probability of each permutation} \\ \(W_{i}\), \(W_{f}\), \(W_{o}\), \(W_{c}\), \(U_{i}\), \(U_{f}\), \(U_{o}\), \(U_{c}\) & Weights in LSTM cells \\ \(b_{i}\), \(b_{f}\), \(b_{o}\), \(b_{c}\) & bias vectors in LSTM cells \\ \(\theta_{i}\), \(\varnothing\) & \multicolumn{2}{c}{ARIMA coefficients} \\ \(\epsilon_{t}\) & \multicolumn{2}{c}{Normal white noise with zero mean} \\ \hline \end{tabular}
\end{table} 
Table 3: Glossary. The relative probability of each permutation in \(\pi\) is calculated as below:  
\[P(\pi)=\frac{T}{n-D+1} \tag{3}\]  
where \(T\) is the number of times the permutation is found in the time series. Finally, the relative probabilities are used to compute the permutation entropy:  
\[PE=-\sum_{i=1}^{D!}P(\pi)\log_{2}P(\pi) \tag{4}\]  
An algorithm for the permutation entropy computation is presented below. **Algorithm 1:** Permutation Entropy  
**Step1** Reshape the data series into a matrix as in Equation (1)  
**Step2** Find the permutation patterns \(\pi\)  
**Step3** Calculate the probability of each permutation in \(\pi\)  
**Step4** Compute \(PE\) as in Equation (4)  
Permutation entropy of the coarse-grained battery voltage is extracted, as in Figure 6. Despite the noise affecting the entropies, in PL11, the differences in the entropies are relatively small compared to the earlier cycles, while the deviations increase as the number of cycles increases.