# Dynamic Voltage Optimization Based on In-Band Sensors and Machine Learning
## 6 Machine Learning in Voltage Optimization
### Approaches Using ML in Real Time Systems  
ML methods for real-time systems vary in the performance of the models and accuracy of the desired outcomes. For real-time systems, the supervised and reinforcement learning methods are more compatible to provide predictive outcomes as models can be trained in previous experiences. In the supervised learning method, the Recurrent Neural Network (RNN) and the Long Short-Term Memory (LSTM) models provide a time-series approach to predictions. For the reinforcement learning method, the Temporal-Difference (TD) prediction model and the Monte Carlo Tree Search (MCTS) model have been successful examples of decision-time planning and provide an ongoing process of learning optimal decisions that provide the highest rewards. The RNN is a model that is trained by sequences of data, in where the order matters, and evaluates the desired outcome given the sequence presented at the input. This approach of time-series is a good implementation when patterns of a data are analyzed to form a conclusion of what was observed before in the sequence. The model can be trained to recognize certain patterns in the sequence of inputs, and the input layer can be extended to a large number of nodes to analyze longer sequences. However, if the RNN model requires to have a \"good memory\" remembering long sequences of values it will impact the performance of the model and computational resources if not running on high-end servers. There are three categories of sequence modeling with four configurations of the model as shown in Figure 5: many-to-one, one-to-many, and many-to-many. The LSTM model a recurrent network architecture in conjunction with an appropriate gradient-based learning algorithm designed to overcome these error back-flow and learn to bridge time intervals in excess of 1000 steps even in case of noisy, incompressible input sequences, without loss of short time lag capabilities [47]. The LSTM memory cell is design in three stages (gates) that provide a recurrent cell state while considering the previous states. Figure 6 displays the design of a memory cell of an LSTM model. The memory cell has three gates combine the forget gate ( f) from the previous state, the input gate (i) in the current state, and the output gate (o) that provides the current state value and hand-off the next cell. LSTM is a popular implementation for detecting patterns in sequences of events to provide forecast values. The reinforcement learning approach differs from supervised and unsupervised learning in the form of having the ability to have an ongoing learning process and provide an action. Due to this difference, it also takes a long time or a vast amount of data to accomplish the optimal training of the model. However, the benefits of the reinforced models are the capacity to handle more complex  
Figure 5: RNN Categorical Models and Configurations [42]. system operations, learn from mistake, and find the right decisions with greater rewards such as the Monte Carlo Tree Search (MCTS) and Temporal-Difference (TD) prediction models. The MCTS model is an algorithm of decision-time planning with means of accumulating values estimates obtained from the Monte Carlo simulations in order to successively direct simulations toward more highly rewarding trajectories [43]. MCTS is better known for the capability to learn the game Go, developed by the AlphaGo team from Google, and defeated the 18-time world Go champion. This model approach will require many examples to learn and understand what optimal choices are needed for desired decision. However, the MCTS has proven that with enough data and examples to learn from will find optimal actions to take and present to achieve high reward outcomes. The TD model is model-free application approach that uses Monte Carlo simulations to estimate values and adjusts prediction with high accuracy before outcomes are known. TD models either have a tabular, denoted as TD(0), prediction approach or n-steps to use statistical bootstrapping.