# An Electric Fence-Based Intelligent Scheduling Method for Rebalancing Dockless Bike Sharing Systems
## 3 Intelligent Electric Fence Dispatching Algorithm Based on DQN
### Generating Actions through Neural Networks
#### 3.1.2 Bias Calculation  
Markov decision processes (MDPs) offer standard formalisms for describing multistate decision making in a probabilistic environment. More precisely, an MDP is a discrete-time stochastic control process, where at each time step, the process is in some state \(s\) and the decision maker chooses a feasible action \(a\). Accordingly, the process then moves to a new state and awards the decision maker a corresponding reward \(r(s,a,s^{\prime})\). The role of rewards is to provide feedback to a reinforcement learning model about the performances induced by the previous actions. Thus, it is important to define the reward to correctly guide the learning process, which accordingly helps the system take the best action policy. In our system, the main goal is to increase customer satisfaction. Thus, we define the rewards as follows:  
\[r(s_{t},a)=CS_{mean} \tag{7}\]  
To perform the dispatch actions with different vehicle states, we utilize the deep Q-networks to dynamically generate optimized values. This learning technique is widely used in modern decision making due to its adaptability to the dynamic features in a system.  
Figure 5: The neural network of the deep Q-learning (DQN).  
The optimal action value function for an electric fence is defined as the maximum expected achievable reward. Thus, we have  
\[R=\sum_{t=0}^{\infty}\gamma_{n}r(s_{t},a_{t}) \tag{8}\]  
where \(0<\gamma<1\) is the discount factor for the future. At any time slot \(t\), the dispatcher monitors the current state \(s_{t}\) and then feeds it to the neural network to generate an action. Note that we do not use a full representation of \(s_{t}\) to find the expectation. Rather, we use a neural network to approximate the Q function.  
For each electric fence, an action is taken such that the output of the neural network is maximized. The learning process starts with zero knowledge, and actions are chosen using a greedy scheme by following the \(\varepsilon\)-greedy method. For the electric fence, after choosing the action and according to the reward, the \(Q\)-value is updated with a learning factor \(\alpha\) as follows:  
\[Q(s_{t},a):=Q(s_{t},a)+\alpha\bigg{(}r(s_{t},a)+\gamma\underset{a^{\prime}}{ \max}Q\big{(}s_{t+1},a^{\prime}\big{)}-Q(s_{t},a)\bigg{)} \tag{9}\]  
where \(\gamma\in[0,1]\) is a discount factor that defines the discounted reward for the future. This technique is known as the value iteration algorithm, and it converges to the optimal action value function, \(Q(s,a)\rightarrow Q^{*}(s,a)\;\;\text{as}\;\;t\rightarrow\infty\). The action value function can be represented with a neural network, which takes the current system state and action as input and outputs the corresponding \(Q\)-value.  
To update the parameters in the neural network, a target value is defined to help guide the update process: \(TargetQ=r+\gamma\underset{a_{t+1}}{\max}Q(s_{t+1},a_{t+1})\). Let Qtarget(\(s\), \(a\)) denote the target \(Q\)-value at state \(s\) when taking action \(a\). The neural network is updated by the mean square error (MSE) in the following equation:  
\[L(\theta)=E\Big{[}(TargetQ-Q(s,a;\theta))^{2}\Big{]} \tag{10}\]