# A Distributed Intelligent Lighting Control System Based on Deep Reinforcement Learning
## 2 Proposed Methodologies
### Illumination Uniform Control Algorithm Based on DDPG
So, the actual illuminance matrix for each test point is \(\mathbf{H}=\mathbf{W}+\mathbf{G}\), according to the luminaire and natural light illuminance matrix. In this paper, the lighting problem is modeled as a reinforcement learning problem. Assuming that there are \(j\) test points with state \(s=H=[h_{1}\), \(h_{2}\),..., \(h_{j}]\), state \(h_{j}\) represents the actual illuminance value of the j-th detection point, i.e., the data measured by the illuminance sensor. The action is modeled as the selection of the dimming factor for the luminaire: \(a=[a_{1}\), \(a_{2}\),..., \(a_{i}]\), where \(a_{i}\) is taken in the range \([0,1]\). From Equations (2) and (3) the next state light level can be obtained in relation to natural light and the level of luminaire switch-on, as shown in Equation (4). \[\mathbf{S}_{t+1}=\left[ \begin{array}{c}s_{1}\\ s_{2}\\ \vdots\\ \end{array} \right]=\mathbf{G}+\mathbf{W}=\mathbf{G}+\mathbf{E}\times\mathbf{u}^{T}=\left[ \begin{array} []{c}G_{1}\\ G_{2}\\ \vdots\\ \vdots\\ \end{array} \right]+\left[ \begin{array}{ccc}E_{11}&E_{12}&\cdots&E_{1i}\\ E_{21}&E_{22}&\cdots&E_{2i}\\ \vdots&\vdots&\vdots\\ \end{array} \right]\left[ \begin{array}{c}u_{1}\\ u_{2}\\ \vdots\\ \end{array} \right] \tag{4}\]  
In this section and in the control process of Section 2.2, the input of the illuminance value to the control algorithm is not the value obtained directly using sensor measurements, i.e., matrix \(\mathbf{S}_{t}\), but the illuminance value produced by natural light at each test point, as shown in the following equation. \[\mathbf{G}=\left[ \begin{array}{c}G_{1}\\ G_{2}\\ \vdots\\ \vdots\\ \end{array} \right]=\mathbf{S}_{t}-\mathbf{E}\times\mathbf{u}^{T}=\left[ \begin{array}{c }s_{1}\\ s_{2}\\ \vdots\\ \vdots\\ \end{array} \right]-\left[ \begin{array}{ccc}E_{11}&E_{12}&\cdots&E_{1i}\\ E_{21}&E_{22}&\cdots&E_{2i}\\ \vdots&\vdots&\vdots\\ \vdots&\vdots&\vdots\\ \vdots&\vdots&\vdots\\ \end{array} \right] \tag{5}\]  
The goal of system optimization is to achieve uniform illumination while meeting lighting requirements. This means that the difference between the actual light level and the expected light level at each detection point is minimized, resulting in the following reward function definition:  
\[r_{t}=-(\mathbf{s}-\mathbf{s}_{exp})(\mathbf{s}-\mathbf{s}_{exp})^{T}=-\sum_{i=1}^{n}(s_{j}- \mathbf{s}_{exp}^{\prime})^{2} \tag{6}\]  
where \(\mathbf{s}\) represents the actual illuminance value matrix for each test point, \(\mathbf{s}_{exp}\) represents the expected illuminance value matrix for each test point, \(s_{j}\) represents the actual light level value for the \(j\)-th test point, and \(\mathbf{s}_{exp}^{\prime}\) represents the expected illuminance value for the area. Deep reinforcement learning applies deep learning to feature extraction and parameter approximation for reinforcement learning, thereby augmenting the algorithm's perception and learning capabilities. The neural network of DDPG employs the Actor-Critical framework. Actor is a strategy network that interacts with its environment and learns a more effective strategy with the aid of a strategy gradient guided by the Critic value function. Critic is a value network that learns a value function from the Actor's interaction with the environment. This function is used to determine how excellent or bad an action is in its current state and assists the Actor in updating its policies. To enhance the stability of neural network training, DDPG employs the target network and applies a soft update based on the following formula. \[\omega^{-}\leftarrow\tau\omega+(1-\tau)\omega^{-} \tag{7}\]  
where \(\omega\) is a network parameter and \(\tau\) is usually a relatively small number. The strategy of the neural network used in this system has three hidden layers, all of which are fully connected, and its exact composition is shown in Figure 5. The number of neurons corresponding to Fc1, Fc2, Fc3, and the fully connected layer Fca are 128, 256, 32, and 1, respectively. The activation function uses the ReLU function.