# Network Performance Optimization for Low-Voltage Power Line Communications
## 3 LVPLC Network Scheme
### Improved \(Q\) Learning Approach in LVPLC Network Scheme  
#### 3.2.1 Improved \(Q\) Learning Model  
\(Q\) learning Mathematic Model  
Reinforcement learning (RL) methods are used to control data packet delivery decisions and improve network stability. They consist of three abstract learning algorithm events: (1) a station observes the state of the environment, generates a reinforcement signal, and selects an appropriate action; (2) the environment generates a reinforcement signal and transmits it to the station; and (3) the station employs the reinforcement signal to improve its subsequent decision. Therefore, a stationrequires information about the state of the environment, reinforcement signals from the environment, and a learning algorithm (Figure 3). A number of RL algorithms, such as \(Q\)-learning and temporal difference learning, have been used. \(Q\)-learning is a recent form reinforcement learning that does not need a model of its environment and works by estimating the values of state-action pairs. It learns behavior through trial-and-error interactions with a dynamic environment, and has been employed for path selection [24]. The algorithm maintains a \(Q\)-value \(Q\left(s,a\right)\) in a table for every state-action pair. Let \(s_{t}\) and \(a_{t}\) denote the reinforcement signal generated by the environment for performing action \(a_{t}\) in state \(s_{t}\). When the station receives reward \(r_{t+1}\), it updates the \(Q\)-value corresponding to state \(s_{t}\) and action \(a_{t}\) as follows:  
\[Q_{t+1}^{(i)}(s_{t}^{(i)},a_{t}^{(i)})=[1-\alpha]Q_{t}^{(i)}(s_{t}^{(i)},a_{t}^ {(i)})+\alpha[r_{t}^{(i)}(s_{t}^{(i)},a_{t}^{(i)},s_{t+1}^{(i)})+\gamma\max_{a ^{j}\in A^{(i)},j\in Neighbor(i)}Q_{t}^{j}(s_{t+1}^{(j)},a^{(j)})] \tag{9}\]  
where \(r\) (\(0\leq r\leq 1\)) is the discount factor, and \(\alpha\) (\(0\leq\alpha\leq 1\)) is the learning rate. The return is deterministic when \(\alpha=1\). Equation (9) is then changed as follows:  
\[Q_{t+1}^{(i)}(s_{t}^{(i)},a_{t}^{(i)})=r_{t}^{(i)}(s_{t}^{(i)},a_{t}^{(i)},s_{ t+1}^{(i)})+\gamma\max_{a^{j}\in A^{(j)},j\in Neighbor(i)}Q_{t}^{j}(s_{t+1}^{(j)},a^{(j)}) \tag{10}\]  
The agent selects the action with the highest \(Q\)-value, except when making an exploratory move. The state transition model is unknown in our problem; hence, we are motivated to use the model free \(Q\)-learning algorithm idea in the LVPLC network. Improved \(Q\)-Learning Mathematic Model  
In reference to the core ideas of the \(Q\) algorithm, this study probes deeply into the following problems: (1) how to achieve bidirectional learning among stations while ensuring state/action sets and (2) how to model the reward function in a complex channel environment. We map herein the state, action, and reward functions for the multi-constrained improved \(Q\)-learning to the LVPLC network. * State  
The general finite state machine (FSM) model shows the states of stations in different network stages, including the initial state (INIT), uncoordinated CCo (UC_CCo), uncoordinated STA (UC_STA), the coordinated CCo (C_CCo), and coordinated STA (C_STA). The C_CCo and C_STA show the stability states of stations. * Action  
Station states evolve from instability to stability via action \(a_{i}\) triggers. The station selects an action \(a_{i}(i=1,2,\ldots n)\) under control policy \(\pi\), such as sending/receiving the beacon. It defines all actions as set A. The action \(a_{t}^{m}\) in set A can be divided into feasible action \(A_{send\_s}\) and infeasible action \(A_{send\_f}\)  
Figure 3: RL model. The uplink/downlink communication success ratio (csr) threshold value can be defined as \(Th_{r1}\), \(Th_{r2}\). The action is shown as follows when the current csr \(r^{\prime}\) is greater than the threshold value \(Thr_{i}\):  
\[a_{t}^{m}= \begin{cases}A_{send\_s}&r^{\prime}\geq Thr_{i}(i=1,2)\\ A_{send\_f}&otherwise\end{cases}