# FL-MD3 QN-Based IoT Intelligent Access Algorithm for Smart Construction Sites
## 3 FL-MD3 QN-Based Intelligent Access
### System Model
#### 3.2.1 MD3 QN Algorithm  
The MD3 QN workflow is shown in Figure 3. During initialization, agents at edge nodes of each site collect real-time site status information. The collected data are stored in an experience replay buffer. Agents select access points \(ap_{j}\) based on current policies. After executing actions, they observe rewards \(r_{t}\) and new states \(s_{t+1}\). To comply with FL privacy constraints, raw experience data remain stored locally at edge nodes. Knowledge sharing is achieved via differential privacy mechanisms. Each experience tuple (\(s_{t},a_{t},r_{t}\), and \(s_{t+1}\)) is assigned an initial priority \(p_{t}=1\), ensuring all samples are sampled at least once. The data in the experience replay buffer correspond to indices in a priority storage array, enabling efficient prioritized experience replay. During algorithm updates, it samples batch data from the experience replay buffer based on priorities. The priority of each data point is dynamically adjusted during network parameter optimization to reflect its contribution to learning. Q-values evaluate action policies, accelerating goal achievement through efficient learning. When accumulated experiences reach a threshold, the target network adopts the latest parameters from the prediction network. This enables synchronized updates to stabilize learning outcomes. Figure 3: MD3 QN processing workflow. Figure 2: Multi-site, multi-modal, multi-user IoT intelligent access framework. In the complex, heterogeneous IoT environment of smart construction sites, the reward function must guide the agent to dynamically balance multi-objective trade-offs, including spectrum efficiency, latency constraints, and privacy protection. For differentiated services (e.g., video surveillance and environmental sensing), this paper designs a service-adaptive multi-objective reward function. The reward \(r_{t}\) for an action under state \(s_{t}\) is defined as:  
\[r(t)=aR_{sum}-bT_{avg}+c(1-BER_{avg}) \tag{11}\]  
where \(R_{sum}\) is the total system transmission rate at current time. \(T_{avg}\) is the average transmission delay at current time. \(BER_{avg}\) is the average BER at current time. The weighting coefficients \(a\), \(b\), and \(c\) are configured in real time based on NN using a lookup table-based method (see Table 2). Normalization constraint: \(a+b+c=1\). This ensures multi-objective balance. For video transmission services, the weights prioritize throughput. For text services, privacy protection is emphasized. This aligns with QoS requirements across different service scenarios. For agent decision making, the \(\epsilon\)-greedy strategy is adopted. With probability \(\epsilon\), a random action is selected. With probability \(1-\epsilon\), the action corresponding to the maximum Q-value computed by the Q-network is chosen, as shown in Equation (12):  
\[a_{t}=\left\{ \begin{aligned} a_{i},i=(0,1,2,\cdots,10,11),\text{P}= \varepsilon\\ \text{argmax}Q(s_{t},a_{t};\theta_{t}),\text{P}=1-\varepsilon\end{aligned} \right. \tag{12}\]  
In the MD3 QN algorithm, the neural network output \(Q(s,\cdot,\cdot;\theta)\) consists of the state value function V and the action advantage function A. This dual-network architecture improves policy evaluation accuracy by distinguishing state value from action advantage. The advantage function A is calculated as follows (Equation (13)):  
\[A(s,a;\theta_{t})=Q(s,a;\theta_{t})-V(s;\theta_{t}) \tag{13}\]  
where \(\theta_{t}\) represents the weight parameters of the Q-network. \(A(s,\,a;\,\theta_{t})\) denotes the advantage of action aa relative to \(V(s;\,\theta_{t})\). A higher advantage value indicates a better action choice. The state value function \(V\), which evaluates the overall expected return of probabilistic actions in future steps, is defined as (Equation (14)):  
\[V(s;\theta_{t})=\underset{n}{\max}V(s;\theta_{t}) \tag{14}\]  
The advantage function \(A(s,\,a;\,\theta_{t})\) reflects the superiority of a specific action.