## 3 System Model and Problem Formulation
### Training Model  
As there is only one type of round in this paper, we refer to a DML training round simply as a round. In round \(r\), smartphone \(n\) produces one set of local weights, denoted by \(\mathbf{w}_{n,r}\in\mathcal{W}_{r}=\{\mathbf{w}_{1,r},\mathbf{w}_{2,r},\,\cdots,\,\mathbf{w}_{ N,r}\}\). \(N\) sets of local weights are uploaded to \(M\) servers for model aggregation. After receiving \(N\) sets of local weights, server \(m\), \(m\in\mathcal{M}=\{1,2,\,\cdots,\,M\}\), executes model aggregation to obtain global weights, represented by \(\mathbf{w}_{r}\).  
Let \(d_{n,r}\in\mathcal{D}_{r}=\{d_{1,r},\,d_{2,r},\,\cdots,\,d_{N,r}\}\) and \(C_{n,r}\in\mathcal{C}_{r}=\{C_{1,r},\,C_{2,r},\,\cdots,\,C_{N,r}\}\) denote the number of data samples and the actual contribution produced by smartphone \(n\) in round \(r\), respectively. Define \(f_{n}(\mathbf{w}_{n,r})\) as the local loss function used by smartphone \(n\). Popular loss functions include cross-entropy loss and mean squared error [4]. Let \(C_{r}=\sum_{n\in\mathcal{N}}C_{n,r}\)denote the total amount of actual contributions produced by \(N\) smartphones. We define the global loss function on all distributed health data as  
\[F(\mathbf{w}_{r})=\frac{1}{C_{r}}\sum_{n\in\mathcal{N}}C_{n,r}f_{n}(\mathbf{w}_{n,r}). \tag{4}\]  
\(F(\mathbf{w}_{r})\) cannot be directly computed without sharing \(\mathcal{C}_{r}\) and \(\mathcal{W}_{r}\) among smartphones and servers. Note that FedAvg [5] uses \(\mathcal{D}_{r}\) to approximate \(\mathcal{C}_{r}\). However, this approximation may not always accurately reflect the true contribution or value of data samples to global model performance. Let \(\eta_{r}\), \(\mathbf{g}_{n,r}\) and \(d_{r}=\sum_{n=1}^{N}d_{n,r}\) denote the learning rate, gradients and total number of data samples used in round \(r\), respectively. For FedAvg, weight aggregation is described as  
\[\mathbf{w}_{r}=\sum_{n\in\mathcal{N}}\frac{d_{n,r}}{d_{r}}\mathbf{w}_{n,r}=\mathbf{w}_{r-1 }-\eta_{r}\sum_{n\in\mathcal{N}}\frac{d_{n,r}}{d_{r}}\mathbf{g}_{n,r}. \tag{5}\]  
In DML, the marginal utility of data samples diminishes as the quantity of data (i.e., \(d_{n,r}\)) increases [34]. This principle of diminishing returns suggests that each additional data sample contributes less to model improvement than previous data samples with the same quality. However, (5) does not account for the diminishing marginal utility. In Sections 4.3 and 5.1, we propose and discuss alternative methods to approximate \(C_{n,r}\) in (4) that incorporate the concept of diminishing marginal utility, aiming to reflect the true value of additional data samples more accurately.  
In this paper, the objective of DML is to determine the optimal set of global weights \(\mathbf{w}_{r}^{*}\) for round \(r\) that minimizes the global loss function \(F(\mathbf{w}_{r})\). The optimization problem is  
\[\mathbf{w}_{r}^{*}=\arg\min_{\mathbf{w}_{r}}F(\mathbf{w}_{r}). \tag{6}\]  
The above problem is often solved by mini-batch SGD [4] due to its NP-hardness [35].  
Training a DNN at the network edge is challenging due to resource constraints: IoT devices often lack the computational power required to efficiently perform SGD and backpropagation on large models. To solve this problem, we introduce a pre-trained model framework in Figure 2 for efficient feature extraction [36]. Neurons in gray color are frozen during training and inference. Only the weights of model heads are trainable, i.e., global and local weights in this paper. Instead of training and uploading the whole model, any smartphone \(n\) can train and upload the weights of the model head. Utilizing pre-trained models for feature extraction is not novel. As it can significantly reduce computational and communication overheads, we would like to highlight this method for improved QoS at the network edge.  
Figure 2: Pre-trained model for feature extraction: only parameters for the model head are trainable and exchangeable in order to reduce the computational and communication overheads at the network edge.