# Lithium-Ion Battery Prognostics through Reinforcement Learning Based on Entropy Measures
## 2 Related Work  
In the current literature, entropy-based predictive models for battery prognostics, as well as other predictive models, have been researched and tested. Table 1 illustrates a brief overview of some of the most relevant and recently published papers that use data-driven methods for lithium-ion battery prognostics.  
\begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**Ref.** & **Data** & **Methods** & **Results** \\ \hline \multirow{3}{*}{[17]} & NASA Ames Prognostics & \multirow{3}{*}{Deep neural networks (DNN)} & The proposed model successfully predicts the SOH and RUL of the lithium-ion battery but is less effective when real-time processing comes into play. \\  & Center for Advanced Life & \multirow{3}{*}{Deep neural networks (CALCE) at the University of Maryland} & \multirow{3}{*}{The ANN predicts the battery State of Charge values} \\  & Cycle Engineering (CALCE) at the University of Maryland & \multirow{3}{*}{The CNN)} & \multirow{3}{*}{with accuracy using only voltage, current, and charge/discharge time as inputs and achieves an MSE of \(3.11\times 10^{-6}\). \\  & NASA Ames & \multirow{3}{*}{Long short-term memory (LSTM)} & The proposed model has a better performance for the time series problem of li-ion battery prognostics and a stronger learning ability of the degradation process when compared to other ANN algorithms. \\  & NASA lithium-ion battery dataset & \multirow{3}{*}{Long short-term memory (LSTM)} & The method produces exceptional performances for RUL prediction under different loading and operating conditions. \\  & Data repository of the NASA Ames Prognostics Center of Excellence (PCoE) & \multirow{3}{*}{Autoregressive integrated moving average (ARIMA)} & The RMSE of the model for the RUL prognostics varies in the range of 0.0026 to 0.1065. \\  & Lithium-ion battery packs from forklifts in commercial operations & \multirow{3}{*}{Autoregressive integrated moving average (ARIMA)} & The ARIMA method can be used for SOH prognostics, but the loss function indicates further enhancement is needed for the environmental conditions. \\  & NASA prognostic model library & \multirow{3}{*}{Reinforcement Learning (RL)} & RL model enables accurate calibration of the battery prognostics but has only been tested on simulated data and sim-to-real transfer needs to be made to test the proposed algorithm on real data. \\  & SPMeT & \multirow{3}{*}{Reinforcement Learning (RL)} & The proposed method can extend the battery life effectively and ensure end-user convenience. However, experimental validation needs to be implemented for the optimal charging strategy. \\  & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: An overview of different approaches to lithium-ion battery prognostics.  
Figure 1: Prediction system for the lithium-ion batteries.  
The literature review reveals a research gap, which can be summarized as follows. Most of the research undertaken so far has relied on traditional Machine Learning and Deep Learning methods. However, the RL method is recognized as an area with room for exploration. Based on these findings, this paper is devoted to filling this gap in the research. LSTM and ARIMA methods are also studied as state-of-the-art models, which can be developed based on the entropy measures and compared with the RL method.  
The main contribution of our study is the proposal of a Reinforcement Learning model based on the permutation entropy of the voltage sequences for predicting the next-cycle battery capacity. To the best of our knowledge, an RL model for lithium-ion battery prognostics, using entropy measures as the input, has not been previously tested in the literature. Additionally, we compare the numerical results from our proposed entropy-based RL model with the results from the state-of-the-art models (i.e., ARIMA and LSMT), which are built based on entropy measures for a fair and reliable comparison.