# D2D-Assisted Adaptive Federated Learning in Energy-Constrained Edge Computing
## 4 Partite Graph-Based FL Scheme with Adaptive D2D Pairing and Aperiodic Local Iterations
\begin{array}{c}Loss(\theta,\phi)=\sum_{k^{\prime}=1}^{k}\min(r_{k^{\prime}} \widehat{A}_{k^{\prime}},clip(r_{k^{\prime}},1-\epsilon,1+\epsilon)\widehat{A} _{k^{\prime}})\\ -c_{1}\sum_{k^{\prime}=0}^{k}(v_{\phi}(s_{k^{\prime}})-\widehat{A}_{k^{\prime }})^{2}+c_{2}\sum_{k^{\prime}=1}^{k}S(\pi_{\theta})\end{array} , \tag{39}\]

where \(S\) is an entropy function, \(\widehat{A}_{k^{\prime}}\) is the evaluation of the critic network, \(\widehat{A}_{k^{\prime}}=\sum_{k^{\prime}=1}^{k}\gamma_{k^{\prime}}r_{k^{ \prime}}-v_{\phi}(s_{k^{\prime}})\), \the improvement of the new policy over the old policy; \(\widehat{A}_{k^{\prime}}\) is an advantage function, which indicates the superiority of action \(a_{k^{\prime}}\) relative to the average strategy, and can be calculated by the difference between the reward \(r_{k^{\prime}}\) and the value function \(v_{\phi}(s_{k^{\prime}})\). The clip function modifies the surrogate objective by clipping the probability ratio, which removes the incentive for moving \(r_{k^{\prime}}\) outside of interval \([1-\epsilon,1+\epsilon]\), where \(\epsilon\) is a hyperparameter [34]. The second term is used to update the value function, which represents the difference between the current state \(s_{k^{\prime}}\) and the actual reward \(\widehat{A}_{k^{\prime}}\), calculated using squared-error loss. The third term is the entropy regularization term; \(S(\pi_{\theta})\) is the entropy of the policy \(\pi_{\theta}\). The larger the value of \(S(\pi_{\theta})\), the more random and exploratory the policy is, which can help prevent the policy from converging to a suboptimal solution in the early training stage; \(c_{1}\) and \(c_{2}\) are weight coefficients. To sum up, we describe the main procedure of the graph learning-enabled D2D pairing scheme in Algorithm 2. In Algorithm 2, we first initialize the parameters of graph \(G\), the actor network, the critic network, and the clipping ratio \(\epsilon\), and we clear the replay buffer, which is a data structure for temporarily saving the agents' observations and allowing our learning procedure to update on them multiple times. Then, the algorithm enters a loop with \(epi^{\max}\) episodes. Each episode consists of \(K\) iterations. In each iteration, given the current state \(s_{k}\), we obtain action \(a_{k}\) of the actor network according to (37). Then, we calculate the reward \(r_{k}\) and obtain the new state \(s_{k+1}\) caused by executing action \(a_{k}\). The tuple \(<s_{k},a_{k},r_{k},s_{k+1}>\) is stored in the replay buffer for future parameter updates. If the condition in step 15 of Algorithm 2 is met, it means that the training time or energy consumption exceeds the limit, and the iteration of this episode will be terminated. Otherwise, \(I\) tuple records will be randomly selected from the replay buffer as samples for PPO strategy training, and the parameters of the actor and critic networks will be updated by maximizing the value of \(Loss(\theta,\phi)\). The above process is repeated until the preset maximum number of episodes is reached.