# D2D-Assisted Adaptive Federated Learning in Energy-Constrained Edge Computing
## 4 Partite Graph-Based FL Scheme with Adaptive D2D Pairing and Aperiodic Local Iterations
The initial embedded feature of the GNN is set as \(h_{n,k}^{(0)}=X_{n,k}\). The action taken by these servers in round \(k\) is represented as \(a_{k}=[a_{1,k},a_{2,k},\cdots,a_{n,k}]\). It is noteworthy that in order to reduce the size of the action space we define the server action as \(a_{n,k}=[F_{n,k},\Psi_{n,k}]\), where \(F_{n,k}\) is the selected head of the cluster where server \(n\) belongs to in the \(k\)-th round, \(1\leq F_{n,k}\leq Z_{1}\), and \(F_{n,k}\in\mathbb{N}^{+}\). Here, we take \(F_{n,k}\) to indicate the D2D pairing relationship of the servers, i.e., server \(n\) uses D2D transmission to aggregate its local model parameters to server \(F_{n,k}\) in the FL round \(k\). Moreover, we should note that action \(F_{n,k}\) is equivalent to variable \(\rho_{n,F_{n,k},k}^{\text{del}}\) in (33), whose value is set to 1. For element \(\Psi_{n,k}\) in action vector \(a_{n,k},\Psi_{n,k}=1\) indicates that server \(n\) participates in local model training in the \(k\)-th FL round, and \(\Psi_{n,k}=0\) vice versa. Subject to the number of D2D communication channels, at most \(Z_{2}\) servers in each cluster can participate in training in one FL round. When \(\Psi_{n,k}=1\), the local iteration number of server \(n\) in round \(k\), which is denoted as \(\tau_{n,k}\), can be calculated following the arithmetic sequence approach described in Section IV. The size of the action space of \(a_{k}\) is \(|2Z_{1}N|\). The reward function of this Markov process in round \(k\) is given as

\[L(\omega_{n,k}^{\prime}|\mathcal{D}_{n,k},\tau_{n,k})\leq L(\omega_{n,k}| \mathcal{D}_{n,k},\tau_{n,k})+\bigtriangledown L(\omega_{n,k}|\mathcal{D}_{n, k},\tau_{n,k})(\omega_{n,k}^{\prime}-\omega_{n,k})+\frac{\zeta}{2}||\omega_{n,k}^{ \prime}-\omega_{n,k}||^{2}\enspace. \tag{13}\]9

In the deep reinforcement learning, given state \(s\) at training time \(t\) while taking action \(a\), the \(Q\) value is

\[L(\omega_{n,k}^{*}|\mathcal{D}_{n,k},\tau_{n,k})\geq L(\omega_{n,k}|\mathcal{ D}_{n,k},\tau_{n,k})+\bigtriangledown L(\omega_{n,k}|\mathcal{D}_{n,k}, \tau_{n,k})(\omega_{n,k}^{*}-\omega_{n,k})+\frac{\mu}{2}||\omega_{n,k}^{*}- \omega_{n,k}||^{2}\enspace. \tag{14}\]0

where \(\beta\in(0,1)\) is a discount factor. To minimize \(Q^{\pi}(s,a)\) the optimal action strategy to be chosen is

\[L(\omega_{n,k}^{*}|\mathcal{D}_{n,k},\tau_{n,k})\geq L(\omega_{n,k}|\mathcal{ D}_{n,k},\tau_{n,k})+\bigtriangledown L(\omega_{n,k}|\mathcal{D}_{n,k}, \tau_{n,k})(\omega_{n,k}^{*}-\omega_{n,k})+\frac{\mu}{2}||\omega_{n,k}^{*}- \omega_{n,k}||^{2}\enspace. \tag{14}\]1

To obtain \(\pi(a|s)\) in the actor network, after all the vertices in the graph complete neighborhood information aggregation, we use a fully connected layer to obtain the action probabilities. The fully connected layer consists of the MLP function and the softmax function, and it can be written as

\[L(\omega_{n,k}^{*}|\mathcal{D}_{n,k},\tau_{n,k})\geq L(\omega_{n,k}|\mathcal{ D}_{n,k},\tau_{n,k})+\bigtriangledown L(\omega_{n,k}|\mathcal{D}_{n,k}, \tau_{n,k})(\omega_{n,k}^{*}-\omega_{n,k})+\frac{\mu}{2}||\omega_{n,k}^{*}- \omega_{n,k}||^{2}\enspace. \tag{14}\]2

In (38), \(Prob(a_{n,k})\) is the probability of taking action \(a_{n,k}\), and the action with the highest probability will be considered as the optimal choice. Based on the chosen optimal action, the state at the next step represented as \(s_{k+1}\) can be obtained. Then, the action that satisfies all the constraints in (33), the current state, the next state, and the reward form a tuple \(<s_{k},a_{k},s_{k+1},r_{k}>\), which will be stored in the replay buffer. Next, a batch of the experience will be sampled from the replay buffer and will be put into the critic network to evaluate \(Q(s_{k},a_{k})\) of action \(a_{k}\) executed in state \(s_{k}\). The PPO (proximal policy optimization) strategy is taken to train the actor network parameter \(\pi_{\theta}\) and the critic network parameter \(v_{\theta}\), where the loss function is expressed as

\[