## 3 SD-QGrid Routing Platform and Algorithms
### SD-QGrid Offline Q-Learning Training Process
#### 3.3.3 Learning Unit of the SDN CN  
The learning unit of the SDN CN uses Q-learning to calculate the Q-table of each grid based on Tables 6 and 7, which are aggregated from the information in Tables 4 and 5 that is sent by RSUs in each grid. The SD-QGrid proposes to record the Q-values of the eight neighbor grids for each grid in the Q-table database. Whenever the learning unit of the SDN CN completes its work, the SD-QGrid offline Q-learning training process ends.  
We define Q-learning parameters of SD-QGrid as follows:  
(a) State represents the node holding the packet in Grid \(i\) at time period \(t\).  
(b) Action represents the neighbor grid that is chosen to transmit the packet.  
(c) Reward indicates whether the selected neighbor grid is the grid where the destination node is located. If this is true, we will give this state a reward of 100; otherwise, it receives a reward of 0.  
The learning unit of the SDN CN performs Q-learning to generate a Q-table of each grid. The flow chart is shown in Figure 7, and the flow description is described as follows:  
Step 1: After the SDN CN aggregates Tables 4 and 5 of the corresponding time periods from each RSU of every grid, it calculates the ratio of the average number of nodes in the current grid to the average number of nodes in the map in this time period, according to Table 6. Then, it proceeds to Step 2.  
Step 2: According to Table 7, the SDN CN calculates the ratio of the average number of outgoing nodes from the current node to each neighbor grid to the sum of the average number of outgoing nodes from the current node to all eight neighbor grids. Then, it proceeds to Step 3.  
Step 3: From Equation (1), Q-learning can use the discount rate to judge the packet transfer between grids. When the discount rate is higher, it means that the future reward from the next state will be higher. Relatively speaking, when the discount rate is higher, the future reward will be higher. Therefore, the discount rate is set that will be obtained by sending packets to neighboring grids in different directions according to the ratio of the average number of nodes per grid, and the ratio of the average directionality of each grid calculated above. We define the discount rate \(\gamma_{t}^{k}\) for selecting neighbor \(Grid_{k}\) at time period \(t\) as follows:  
\[\gamma_{t}^{k}=\left(w\times\frac{H_{t}^{s}}{\sum H_{t}}+\mathrm{j}\times \frac{N_{t}^{k}}{\sum_{k}N_{t}^{k}}\right)\times m \tag{4}\]  
where \(k=0,\ 1,\ 2,\ \ldots,\ 7\), representing eight directions. \(H_{t}^{s}\) is the average number of nodes in \(Grid_{s}\) at time period \(t\). \(\frac{H_{t}^{s}}{\sum H_{t}}\) is the ratio of the average number of nodes in the current grid to the average number of nodes in the map at time period \(t\) at Step 2. \(N_{t}^{k}\) is the average number of outgoing nodes in the neighbor grid from \(Grid_{s}\) in the direction \(k\) at the time period \(t\). \(\sum_{k=0}^{7}N_{t}^{k}\) is the sum of the average number of outgoing nodes to the neighbor grid in eight directions from \(Grid_{s}\) at the time period \(t\). \(\frac{N_{t}^{k}}{\sum_{k}N_{t}^{k}}\) is the ratio of the number of outgoing nodes in each direction to the total number of outgoing nodes in all eight directions at Step 3. Parameter \(m\) means that the value of \(\gamma_{t}^{k}\) is between 0.3 and 0.9, as to avoid the value of Q-value being too large, which, in turn, affects the choice of the packet transmission route. Finally, w and j are the weighting factors of these two terms in Equation (4), respectively.  
Step 4: The SDN CN performs the Q-learning calculation with the calculated discount rate to generate the corresponding Q-table of each grid for different time periods, and records the Q-value of each grid for the eight neighbor grids in the Q-table database.