# A Distributed Intelligent Lighting Control System Based on Deep Reinforcement Learning
## 2 Proposed Methodologies
### Illumination Uniform Control Algorithm Based on DDPG  
The discipline of artificial intelligence has developed rapidly in recent years. Deep reinforcement learning, one of the emerging technologies in this field, has produced extremely impressive results in decision-making optimization. Deep reinforcement learning integrates the powerful feature extraction capabilities of deep learning with the powerful decision-making capabilities of reinforcement learning, enabling a framework from input perception to output decision. This method closely resembles the human mind in that it perceives the environment, then makes decisions via the brain's neurons, and finally takes the appropriate action. In this paper, the Deep Deterministic Policy Gradient algorithm from deep reinforcement learning is selected as the control algorithm to accomplish uniform indoor lighting. The DDPG algorithm has several advantages over conventional optimization algorithms as well as other reinforcement learning algorithms, including the following:  
* Deep reinforcement learning algorithms are in real time, and unlike general heuristics, they can save the model after training and do not need to be recalculated for each adjustment;
* Having the ability to \"store experience\" and to learn from past experience;
* DDPG can handle high-dimensional and continuous action-state spaces to prevent \"dimensional disasters\". Reinforcement learning is a machine learning method that describes the problem as a Markov decision process (MDP) and learns the optimal policy through continuous interaction between the agent and the environment. Markov decision processes satisfy Markovianity, which means that at any moment \(t\), the state \(s_{t+1}\) at the next moment depends only on the current state \(s_{t}\). The MDP can be represented by a tuple \((S,\,A,\,P,\,R,\,\gamma)\), where \(S\) is the state space, \(A\) is the action space, \(P\) is the set of environmental state transfer probabilities, \(R\) is the reward value, and \(\gamma\) is a discount factor in \([0,1]\). In Figure 4, during the training process, the agent continuously interacts with the environment and obtains the corresponding reward \(r_{t+1}\) and state \(s_{t+1}\) by applying action at to the environment based on the observed state \(s_{t}\) of the environment at moment \(t\). It then learns according to the obtained reward and state and repeats until it converges to the optimal policy. The process of controlling the lighting in this system is based on the principle of illumination superposition. The principle of illumination superposition means that the illuminance value at a point is the sum of the illuminance values produced by the point light sources at that point, illustrated in Equation (1). \[E=\sum_{i=1}^{n}E_{i} \tag{1}\]  
where \(E\) indicates the actual illuminance value at a point, and \(E_{i}\) indicates the illuminance value produced by the i-th point source at that point. Figure 4: The reinforcement learning interaction process. The artificial illumination of each detection point can be expressed in the matrix \(\mathbf{W}\):  
\[\mathbf{W}=\left[ \begin{array}{c}W_{1}\\ W_{2}\\ \vdots\\ \vdots\\ \vdots\\ \end{array} \right]=\mathbf{E}\times\mathbf{u}^{T}=\left[ \begin{array}{ccc}E_{11}&E_{12 }&\cdots&E_{1i}\\ E_{21}&E_{22}&\cdots&E_{2i}\\ \vdots&\vdots&\vdots\\ \vdots&\vdots&\vdots\\ \vdots&\vdots&\vdots\\ \end{array} \right] \tag{2}\]  
\[\mathbf{G}^{T}=\left[ \begin{array}{ccc}G_{1}&G_{2}&\cdots&G_{j}\\ \end{array} \right] \tag{3}\]  
In Equations (2) [43] and (3), \(j\) is the total number of detection points; \(\mathbf{E}\) is the illuminance gain matrix for \(i\) luminaires for \(j\) detection points; \(E_{ji}\) is the illuminance contribution of the i-th luminaire to the j-th detection point; \(u_{i}\) is the openness of the i-th lumininaire, i.e., the dimming factor; \(\mathbf{G}\) is the natural illuminance gain matrix for \(j\) detection points; and \(G_{j}\) is the natural illuminance contribution for the j-th detection point.