# Resource Allocation on Blockchain Enabled Mobile Edge Computing System
## 3 Resource Allocation Problem
### Reinforcement Learning Model
#### 3.3.5 TCN Enabled A3C Model  
A3C, short for Asynchronous Advanced Actor-Critic, is an improvement on the traditional Actor-Critic algorithm. A3C algorithm converges faster, contains more stable policies [40; 41]. As its name shows, the A3C algorithm consists of two parts, the Actor-Critic algorithm and asynchronous algorithm. The policy function and state-value function in Actor-Critic are crucial, as they determine the available action to obtain higher reward. As mentioned above, TCN is applied to these two functions to extract long-time trend and void gradient-vanish. In the following article, firstly, the implementation of TCN in Actor-Critic is introduced. Secondly, the asynchronous algorithm, that is how A3C updates the parameters, is described. (1) TCN and Actor-Critic  
The Actor-Critic algorithm contains two neural networks, one is the Critic, the other is called Actor. The former one, or the Critic, judges the state-value function, which is the estimation of the expectation of future rewards based on current state and the corresponding policy. The latter one, or the Actor, estimates the probability of a set of output actions according to the input state. Combining two models, Actor-Critic is obtained. Furthermore, the advantage function is introduced to the model when updating parameters, it can tell whether the output of the model is good or not. As shown in Figure 3a, here, the combination of temporal convolutional network and fully-connected network is deployed to approximate the state-value function \(V(s)\) and the policy function \(\pi(s)\). Different from previous architecture that only relies on current state \(s\), the prediction of action \(a\) depends on both current state and previous states. Furthermore, with the current and previous states and the taken action \(a\), the neural network can thus tell the state-action value. The state-value function and the policy function are given below, the left side of Equation (28) is the initial definition of the state-value function, and the right side of Equation (28) is the state-value function based on neural network. The same as the policy function shown in Equation (29):  
\[V(s,a)\approx V(s_{n-l+1},\ldots,s_{n-1},s_{n}|a;\mathbf{w}) \tag{28}\]  
\[\pi(s,a)\approx\pi(a|s_{n-l+1},\ldots,s_{n-1},s_{n};\mathbf{\theta}) \tag{29}\]  
where \(\mathbf{w}\) is the weight coefficient of the value network, \(\mathbf{\theta}\) is the weight coefficient of the policy network, \(s\) is the state of the agent, \(a\) is the action of the agent, and \(l\) is the history size. Let us further analyze the process and structure of the neural network. To calculate the state-value function or the policy function, a state matrix needs to be put into the network, that the column of the matrix is state vector given by Equation (21). Then, the matrix is processed by TCN. As shown in Figure 3b, the TCN has two main characteristics, one is that the output size is the same as input size, and the other is that there cannot be any backsliding from the future to the present. To obtain the first characteristic, TCN uses a one dimensional fully-convolutional network, where remain the size of hidden layer and input layer the same. Furthermore, zero padding is added to keep the front and back layers equal in length. Furthermore, to achieve the second characteristic, TCN applies causal convolutions, the calculation of an output is only with elements from previous and now. The implementation of dilated convolution enables a wider range of receptive field. As you can see, if the dilatation factor \(d\) is set as \(1,2\), and \(4\), the output layer can obtain nine units of history size from the second hidden layer, and fifteen units of history size from the input layer. Along with the dilated causal convolution, an \(1\times 1\) convolution is added to avoid gradient-vanishing when the neural network is quite deep. After the process of TCN, we use a one-dimension convolution to reduce the length in the time dimension.