# Indoor Positioning by Double Deep Q-Network in VLC-Based Empty Office Environment
## 5 Simulation and Results  
In this section, we describe the simulation parameters and present the results. The indoor VLC environment was implemented using MATLAB 2017b, while the double deep Q-Network model was designed based on Python 3.7 and implemented using the keras-rl2 library. The simulation environment and parameters are summarized in Table 2.  
Figure 6: Block diagram of proposed method. In Table 2, the RPs were positioned at 20 cm intervals within the indoor environment, with a total of 676 RPs set. For the double deep Q-Network model, a total of 500 episodes were run, with each episode limited to a maximum of 100 steps. Then, the replay memory size is set to 81,920. The weights of the target network were updated every 100 episodes in the Q-Network. The learning policy employed the Epsilon-Greedy policy. This policy is a strategy for making optimal decisions at each step without considering the future. In other words, it is an algorithm that assumes that the best choice at each step will lead to the best overall outcome. Of course, it does not always return the best results because it does not take future values into consideration. Table 3 evaluates the learning performance by varying the number of nodes per layer in the designed reinforcement learning model. In this study, the total number of neural network layers was set to three, and a rectified linear unit (ReLU) was applied as the activation function for each layer. Indicators for assessing learning performance include average test scores and learning time. As shown in Table 3, as the number of nodes per layer increases, the learning time also increases, and the average test score improves. However, the learning time is 30 to 60 s longer than other combinations. This will require more learning time in the future if the total number of episodes increases or the search area expands. Conversely, when the number of nodes per layer is small, the average test score is low, but the learning time is the fastest. Therefore, in this study, the number of nodes per layer was selected as 64-32-16 based on the results in Table 3. As shown in the table, with 64-32-16 nodes per layer, a relatively high average test score of 9.0 can be achieved within a learning time of 282.5 s. Figure 7 shows the performance graph of learning through the model. In Figure 7, it can be observed that the reward value rapidly converges to a value greater than 0 as the episode progresses in the double deep Q-Network model. This signifies that the agent is able to locate the adjacent RP to the UE more efficiently as the training proceeds. To test the positioning performance, double deep Q-Network agent training was performed for a total of 500 episodes. Utilizing the trained double deep Q-Network agent, as described earlier, a positioning test was conducted. For this purpose, the trained model was saved in the h5 format of the Keras library, a Python package. To test the saved model, the UE was deployed in approximately 1000 new episodes. Afterward, the positioning \begin{table}
\begin{tabular}{c c c} \hline \hline \multicolumn{1}{c}{**Parameter**} & \multicolumn{1}{c}{**Value**} \\ \hline \multirow{3}{*}{Indoor Environment} & Room size & 5 m \(\times\) 5 m \(\times\) 3 m \\  & No. APs & 4 \\  & No. RPs & 676 \\  & Episode & 500 \\  & Max step of each episode & 100 \\ Double DQN & Replay memory size & 81,920 \\  & Update of target network & 100 \\  & Policy & Epsilon Greedy \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Simulation parameters.