# Resource Allocation on Blockchain Enabled Mobile Edge Computing System
## 3 Resource Allocation Problem
### Reinforcement Learning Model
#### 3.3.5 TCN Enabled A3C Model
This is followed by the operation of flatten and finally the selection of available action and calculation of state-action value using the fully-connected neural network. With the state-action value, we can calculate the target value, or target reward, after \(n\) steps. Here Target \(V\) is used to represent the final reward after \(n\) steps:  
Figure 3: Network architecture: (**a**) TCN enabled A3C; (**b**) temporal convolutional network. \[\text{Target }V=r_{t}+\gamma r_{t+1}+\ldots+\gamma^{n-1}r_{t+n-1}+\max_{a}\gamma^{n}V(s_ {t+n},a) \tag{30}\]  
where \(\gamma\) is the discount factor. In this way, a certain reward can interfere with \(n\) states before that, enabling the model to imitate the experience better and improving the validity of the model. Then we introduce the advantage function. When updating policy-gradient, discount reward \(R_{t}\) can inform the agent what actions are good, what are bad. We subtract the reward function and state-value function to obtain the advantage function \(A(s_{t}-a_{t})\). For example, based on the same state, if \(A(s_{t}-a_{t})>0\), then the chosen action is good, and vise versa. \[A(s_{t},a_{t})=R(s_{t},a_{t})-V(s_{t}) \tag{31}\]  
To calculate the advantage function, we have:  
\[A(s_{t},a_{t};\mathbf{\theta},\mathbf{w})=\sum_{i=0}^{k-1}\gamma^{i}r_{t+i}+\gamma^{k} V(s_{t+k};\mathbf{w})-V(s_{t},\mathbf{w}) \tag{32}\]  
To avoid the algorithm trapped into local optimum, [PERSON] [40] added entropy \(\beta\
abla_{\theta}H(\pi(s_{t};\mathbf{\theta}))\) in policy \(\pi\). The gradient updating function is:  
\[\
abla_{\theta}\mathbb{E}[R_{t}]= \
abla_{\theta}\log\pi(a_{t}|s_{t};\mathbf{\theta})A(s_{t},a_{t}; \mathbf{\theta},\mathbf{w}) \tag{33}\] \[+\beta\
abla_{\theta}H(\pi(s_{t};\mathbf{\theta}))\]  
Equation (33) means that the higher expectation of the reward, the more likely the action will be chosen. \(\pi(a_{t}|s_{t};\mathbf{\theta})\) denotes the possibility to choose action at state \(s_{t}\). (2) Asynchronous algorithm  
How does A3C exactly work? Simply, it is a process in which many Actor-Critic workers work asynchronously and update the parameters of the one policy function, or global network. To make the Actor-Critic workers work asynchronously, the asynchronous algorithm is introduced. The asynchronous framework of the A3C algorithm is given in Figure 4, and it mainly consists of environment, worker, and global network. Each worker performs as an independent agent to interact with an independent environment, using the result of interaction to update the parameters of the global network. With each worker interacting with a certain environment, the model can obtain through more situations, which speeds up the convergence and lessen the possibility of being trapped in the local optimum. The asynchronous algorithm works as following:  
1. Global network parameter initialization. 2. Workers duplicate global network. 3. Workers interact with their environments. 4. Workers calculate the loss of value and policy. Figure 4: Update principle of A3C algorithm. Workers calculate gradient through loss function. * 6 Workers update the parameters of global network.