# Deep Reinforcement Learning-Based Dynamic Pricing for Parking Solutions
## 3 Proposed Method
### Deep Reinforcement Learning-Based Dynamic Pricing (DRP-DP)
#### 3.2.7 Reinforcement Learning
\begin{table}
\begin{tabular}{l l l} \hline \hline
**Notation** & **Unit** & **Description** \\ \hline Y & (int) & Year. \\ M & (int) & Month. \\ D & (int) & Date. \\ t & (hour) & One hour interval. \\ T & (hour,..., hour) & A period of hours. \\ V & - & Parking vendor. \\ d & - & Driver. \\ L\({}_{\text{d}}\) & (lat, lng) & Location of driver. \\ L\({}_{\text{P}}\) & (lat, lng) & Location of player. \\ L\({}_{\text{O}}\) & (lat, lng) & Location of opponent. \\ Dr\({}_{\text{P}}\), Do & (meter) & Distance of driver’s destination to location of player and opponent. \\ Fr\({}_{\text{P}}\), F\({}_{\text{O}}\) & (int) & Vehicle arrival flow rate of player and opponent. \\ F\({}_{\text{A}}\) & (int) & Vehicle arrival flow rate after SMA. \\ P\({}_{\text{P}}\), P\({}_{\text{O}}\) & - & Pricing policies of player and opponent. \\ D(\(\cdot\)) & - & Discount function. \\ N & (int) & Number of inflows. \\ N\({}_{\text{T}}\) & (int) & Number of inflows after SMA. \\ N\({}_{\text{C}}\) & (int) & Number of controllable inflows. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Notation of used variables in environment simulation. avoid being trapped at a local optimum, as a trade-off between exploration and exploitation. An episode in this study refers to the selected week to undergo environment simulation. The episode will be repeated three thousand episodes to achieve learning efficiency and accuracy. The proposed approach is summarised in Algorithm 1. The environment entity computes the entry and exit flow of the player and competitors in the reward function. The number of drivers will choose their parking space based on price and distance preference with the computed in and outflows. Algorithm 2 shows the procedure of the reward function. ```
1:Initialise environment variable (number of opponents, price range, datetime, period section and etc.) 2:Get the current time-step
3:N \(\Leftarrow\) number of hours in period section (a period of hours)
4:whileN \(>\) 0 do
5: Gets an action based on the Q values of the Q-table. 6: Performs the action
7: Observe the environment and get the reward
8: Update current state of environment
9: Calculate new Q value and update to Q-table
10: Increment the current time-step
11:\(N\Leftarrow N-1\)
12:endwhile
```  
**Algorithm 1** Implementation of RL with Q-Learning for the parking industry. ```
1:Get player current state
2:Get opponent current state
3: Normalise vehicle stay time from player and opponent
4: Initialise rewards, in and out flow of vehicle to 0
5:\(N\Leftarrow\) number of hours in period section (a period of hours)
6:whileN \(>\) 0 do
7: Define hourly reward to 0
8:\(IN\_N\Leftarrow\) number of inflow of current hour (day and hour)
9:\(IN\_CN\Leftarrow\) controllable inflow by calculating the difference between target vehicle flow after vehicle flow regulation with forecasted inflow using historical parking occupancy data
10:while\(IN\_N>\) 0 do
11: Initialise driver with stay time
12: Driver makes decision on parking area
13: Update reward value
14:\(IN\_N\Leftarrow N-1\)
15:endwhile
16:while\(IN\_CN>\) 0 do
17: Initialise driver with stay time
18: Driver makes decision whether to park or visit next time and its parking area
19: Update reward value
20:\(IN\_CN\Leftarrow N-1\)
21:endwhile
22: Update parking occupancy of player and opponent
23: Add hourly reward to rewards
24: Update occupancy of player and opponent
25: Increment the current time-step
26:\(N\Leftarrow N-1\)
27:endwhile
28:Return rewards and updated environment data
```  
**Algorithm 2** Proposed reward function to return reward at each time step.