# Compp-DDPG: Computation Offloading with Privacy Preservation in a Vehicular Edge Network
## 5 Cooperative Optimization for Privacy and Priority Based on DDPG
### Reward Space  
The reward is used to evaluate the performance of the selected action. All the ICVs are assumed to use the same reward function. To achieve the goal of minimizing the cost function in the optimal problem, we designed the reward function \(r_{i}(t)\) of the agent ICV-\(i\) during the period \(T\) as  
\[r_{i(t)}=-\lim_{T\to\infty}\frac{1}{T}{\sum_{t=1}^{T}U_{i}(t)\Big{|}s_{i(t)}}. \tag{26}\]  
Figure 2: The structure of COPP-DDPG.  
The average reward of multi-agent ICVs in time slot \(t\) is given by  
\[r(t)=\frac{1}{V}\sum_{i=1}^{V}r_{i(t)},\forall i\in\mathcal{V}. \tag{27}\]  
The overall flow of COPP-DDPG algorithm is based on DDPG, as shown in Algorithm 1. DDPG algorithm is widely used and introduced in a large number of studies [40]. Here we will not describe the details of the algorithm. The goal of DDPG is to make the agents obtain the maximum reward, which is guided by the rewards observed by the interaction with the environment. Due to the interaction between multiple ICVs in the VEN environment, the behavior of one ICV has the potential to influence and change the decisions of other ICVs. Therefore, multi-agent DDPG is considered in this work to meet the increasing complexity of environment and task requirements, which takes full account of the cooperation and competition between agents to maximize joint returns. However, with the preliminary experiments, we found that the centralized multi-agent DDPG leads to the problem of poor convergence. Therefore, the COPP-DDPG we finally chose is a decentralized multi-agent DDPG method.  
```
1 Randomly initialize the critic network \(Q(s,a\big{|}\theta_{i}^{Q})\) and the actor network \(\mu(s\big{|}\theta_{i}^{n})\) with weights \(\theta_{i}^{Q}\) and \(\theta_{i}^{n}\);
2 Initialize the target critic network \(Q^{\prime}\) and the target actor network \(\mu^{\prime}\) with weights \(\theta_{i}^{Q^{\prime}}\) and \(\theta_{i}^{n^{\prime}}\);
3 Initialize the memory replay buffer \(B\);
4 forpisode\(k=1,2,\ldots,K\)do
5 Initialize a random process \(n_{t}\);
6 Receive initial observation state \(S=\Big{\{}s_{(1)},s_{i(2)},\ldots,s_{i(N)}\Big{\}}\);
7 for\(i=1,2,\ldots,N\)do
8 for\(t=1,2,\ldots,T\)do
9 Select action \(a_{i(t)}=\mu(s_{i(t)}\big{|}\theta_{i}^{n})+n_{t}\) according to the current
10 policy and exploration noise \(n_{t}\);
11 Execute action \(a_{i(t)}\) and observe reward \(r_{i(t)}\), the next state \(s_{i(t+1)}\);
12 Store all transitions (\(s_{i(t)}\), \(a_{i(t)}\),\(r_{i(t)}\),\(s_{i(t+1)}\)) in \(B\);
13 Sample a random mini-batch of \(Z\) transitions from \(B\);
14 Set
15 \[y_{i}=r_{i(t)}+\mathrm{Y}Q_{T}(s_{i(t)},a_{i(t)}\big{|}\theta^{Q})\Big{|}_{a_{ i}^{+}=\mu^{\prime}(s_{i}^{n+1})}\] (28) Update the critic network \(Q(s,a\big{|}\theta_{i}^{Q})\) by minimizing the loss
17 \[L\big{(}\theta^{Q})=\tfrac{1}{Z}\sum_{i}y_{i}-Q_{T}\left(s_{i},a_{i(t)}\big{|} \theta_{i}^{Q}\right)^{2}\] (29) Update the actor policy by using the sampling policy gradient
19 \[\
abla_{\theta_{i}^{n}}f\approx\tfrac{1}{Z}\sum_{i}\
abla_{a}Q(s_{i(t)},a_{i( t)}|\theta_{i}^{Q})|_{a_{i}=\mu(s_{i})}\
abla_{\theta_{i}^{n}\mu(s_{i(t)}| \theta_{i}^{n})}\] (30) Update the target networks for each agent \(i\): \[\theta_{i}^{Q^{\prime}/\mu^{\prime}}\leftarrow\eta\theta_{i}^{Q^{\prime}/\mu }+(1-\eta)\theta_{i}^{Q^{\prime}/\mu^{\prime}}\] (31)
22 end for
23
24 end for
```  
**Algorithm 1**Cooperative Optimization for Privacy and Priority based on Deep Deterministic Policy Gradient COPP-DDPG