# Fed-DeepONet: Stochastic Gradient-Based Federated Training of Deep Operator Networks
## 4 Numerical Experiments
### Experiment 3: Fed-DeepONet and Data Heterogeneity  
To test the effect of data heterogeneity on the performance of Fed-DeepONet Algorithms 1 and 2, we designed the following two experiments.  
In the _first experiment_, we modify the amount of data heterogeneity as follows. Consider the centralized dataset \(\mathcal{D}_{\text{cent}}\) with \(N\) data triplets \(\{u_{m}^{(k)},y^{(k)},G(u_{m}^{(k)})(y^{(k)})\}\). We sort this dataset according to the target operator values \(G(u_{m}^{(k)})(y^{(k)})\) (i.e., from smallest to largest). Then, we divide the sorted dataset into \(n_{\text{shards}}\) blocks/shards (for simplicity, we let \(n_{\text{shard}}\geq C\)). We then assign to each of the clients \(n_{\text{shards}}/C\) blocks for sampling data. Notice that as the value of \(n_{\text{shards}}/C\) increases, the federated dataset becomes more independent and identically distributed. This is because the clients can get data from multiple blocks, making the data samples more diversified. As a result, the densities \(p_{i}(\{u_{m},y\},G(u_{m})(y))\) and \(p_{j}(\{u_{m},y\},G(u_{m})(y))\) become more statistically alike. On the other hand, as the value of \(n_{\text{shards}}/C\) decreases, the data heterogeneity increases. In particular, for this simple experiment, the data heterogeneity reaches its maximum when \(n_{\text{shards}}=C\).  
To verify the effect of data heterogeneity, we consider a federated training scenario with \(C=20\) clients and allow the fraction of available clients to vary at each global round. Moreover, we use \(R=20\) global synchronization rounds and \(K=200\) local updates. Figure 5 (left) (resp. Figure 5 (right)) depicts the mean \(L_{2}\) relative error for 100 in-distribution (resp. out-of-distribution) trajectories when we vary the data heterogeneity (i.e., \(n_{\text{shards}}/C\in\{1,2,\dots,10,15,20\}\)). The results show that for the most extreme data heterogeneous setting, i.e., when \(n_{\text{shards}}=C\), Algorithm 1 outperforms Algorithm 2. However, in the less extreme data heterogeneity scenarios, Algorithm 2 has an excellent performance, similar to the case when the data collected by each client is independent and identically distributed.  
In our _second experiment_, we modify the input functional distribution (that is, the input data heterogeneity) for some of the \(C\) clients. In particular, we use two distinct Gaussian Random Fields (GRF). The first GRF uses the length-scale \(\ell_{1}=0.2\) and the second GRF uses \(\ell_{2}=\ell_{1}+\Delta\). We use the control parameter \(\Delta\) to increase the heterogeneity between the two input distributions. We follow a federated training scenario similar to the one described in  
Figure 5: Fed-DeepONet \(G_{\theta^{*}}\) (Algorithms 1 and 2) performance change on the in-distribution (**left**) and out-of-distribution (**right**) test trajectories when we vary the data heterogeneity using \(n_{\text{shards}}/C\).  
the previous example and randomly assign one of the input distributions to each of the \(C\) clients.  
Figure 6 (left) (resp. Figure 6 (right)) depicts the mean \(L_{2}\) relative error for 100 in-distribution (resp. the out-of-distribution) trajectories when we vary the functional input data heterogeneity (i.e., we increase \(\Delta\)). The results illustrate that as we increase the input data heterogeneity (i.e., as we increase \(\Delta\in\{0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0\}\)), the performance of the proposed adaptive Fed-DeepONet (Algorithm 2) does not deteriorate. Such a result shows that Algorithm 2 is robust against changes in the input distribution. Note also that Algorithm 1 is also not affected by the increase in data heterogeneity. The performance of Algorithm 1, however, is not acceptable.