SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ziba Parsons CIS Department University of Michigan Dearborn, MI zibapars@umich.edu
Fei Dou  
School of Computing  
University of Georgia  
Athens, GA  
fei.dou@uga.edu
Houyi Du  
CIS Department  
University of Michigan  
Dearborn, MI  
houyidu@umich.edu
Zheng Song
CIS Department University of Michigan Dearborn, MI zhesong@umich.edu
Jin Lu  
School of Computing  
University of Georgia  
Athens, GA  
jin.lu@uqa.edu
This paper explores the challenges of implementing Federated Learning (FL) in practical scenarios featuring isolated nodes with data heterogeneity, which can only be connected to the server through wireless links in an infrastructure-less environment. To overcome these challenges, we propose a novel mobilizing personalized FL approach, which aims to facilitate mobility and resilience. Specifically, we develop a novel optimization algorithm called Random Walk Stochastic Alternating Direction Method of Multipliers (RWSADMM). RWSADMM capitalizes on the server's random movement toward clients and formulates local proximity among their adjacent clients based on hard inequality constraints rather than requiring consensus updates or introducing bias via regularization methods. To mitigate the computational burden on the clients, an efficient stochastic solver of the approximated optimization problem is designed in RWSADMM, which provably converges to the stationary point almost surely in expectation. Our theoretical and empirical results demonstrate the provable fast convergence and substantial accuracy improvements achieved by RWSADMM compared to baseline methods, along with its benefits of reduced communication costs and enhanced scalability.
Federated Learning (FL) [1, 2, 3, 4] is a distributed machine learning paradigm that enables clients to learn a shared model without sharing their private data. Unlike traditional machine learning approaches that rely on central servers for model training, FL allows clients to collaborate and train the model in a distributed manner, overcoming privacy issues related to passing data to a central server. Despite its advancements, real-world applications in environments with insufficient network support continue to face challenges. a) Maintaining consistent and reliable connections between the central server and clients becomes exceedingly challenging in environments lacking network infrastructures, e.g., natural disasters or military warzones. While intermittent connectivity may be available through satellite networks, the instability and limited capacity of such networks prevent the transmission of large data volumes, making it difficult to collect model updates from soldiers or first responders. b) The non-IID (non-independent and identically distributed) nature of clients' data, characterized by heterogeneity across the network, can hinder the generalization of the global model
37th Conference on Neural Information Processing Systems (NeurIPS 2023), New Orleans, USA.
for each client. Addressing these challenges is crucial for practical FL in such environments. In this paper, we propose RWSADMM, a novel FL scheme that uses Random Walk (RW) algorithm to enable server mobility among client clients. These dynamic approach benefits scenarios with limited internet connectivity, where clients form clusters using local short-range transmission devices.
For example, in various contexts such as robotics, emergency response, or military operations, consider a scenario where individuals or entities are equipped with integrated visual augmentation systems (IVAS) [5]. To facilitate the collection of model updates from these entities, a mobile unit equipped with a powerful computer navigates the environment [6], communicating with locations through a network, which might be satellite-based or another suitable alternative. Upon reaching an entity, the mobile unit employs short-range communication technologies such as WiFi direct, Zigbee [7], or Bluetooth to establish connections with nearby IVAS devices. Through these connections, the unit collects model updates and distributes new models as necessary. A graph-based representation is utilized to determine the order of interactions, where entities are depicted as nodes, and the edge between an entity and its neighbor indicates that the neighbor is within the communication range of the unit that reaches the entity. This graph assists the mobile unit in making informed decisions about the order in which it engages with the entities.
Various applicable examples of such constrained network situations span across different domains, including ad hoc wireless learning [8, 9, 10], wildlife tracking [11, 12, 13], Internet of Underwater Things (IoUT) [14, 15], natural disaster management [16, 17], military operations [18], or fostering a digital democracy [19, 20] which assists in overcoming restrictions imposed by regimes that prohibit internet access to civilians.
Specifically, to address challenge a), we propose an algorithmic framework called RWSADMM (Fig. 1), short for Random Walk Alternating Directional method of Multipliers, which considers a dynamic reachability graph among distributed clients using a movable vehicle as the central server. Clients are represented as nodes in the graph, with edges denoting neighborhood connections. Personal devices, referred to as local clients, establish dynamic connectivity with the server when the server is nearby. The server connects with a selected client and its neighbors while moving between locations using a non-homogeneous RW algorithm for probabilistic navigation. In each computation round, the vehicle broadcasts and gathers local model updates from residing clients, who rely on short-range communication to interact with the moving server, when it's within the communication range. The received updates are aggregated and used to update the global model iteratively.
To tackle the second challenge (b) arising from the heterogeneity of data distribution among clients, RWSADMM incorporates model personalization through local proximity among adjacent clients using hard inequality constraints, as opposed to unconstrained optimization with regularization techniques that may induce model bias. By formulating the problem with these constraints, RWSADMM reduces the computational complexity for clients, effectively mitigating the limitations of local computational power. This is achieved by designing the solver to the stochastic approximation of the minimization subproblem within the typical ADMM algorithm.
Figure 1: This illustration showcases the training process using the RWSADMM algorithm. A vehicle, serving as the mobile server, navigates between different clients using a random walk strategy. (a) In step  $k$ , the server moves to client 13, covering the clients in  $\mathcal{N}(13)$  for FL model training and completing the aggregation step. (b) In step  $k + 1$ , client 5 is selected, and the vehicle moves to client 5. The training and aggregation steps occur within the zone encompassing  $\mathcal{N}(5)$ .
Our research makes three main contributions. Firstly, our proposed RWSADMM algorithm is the first attempt to enable mobilizing FL with efficient communication and computation in an infrastructure-less setting. The RWSADMM framework involves a server dynamically moving between different regions of clients and receiving updates from one or a few clients residing in the selected zone, which reduces communication costs and enhances system flexibility. Secondly, to address the issue of data heterogeneity among clients, RWSADMM formulates local proximity among adjacent clients based on hard inequality constraints, avoiding the introduction of model bias via consensus updates. This approach provides an alternative realization of personalization, which is crucial when dealing with highly heterogeneous data distributions. Thirdly, to mitigate the computational burden on the clients, an efficient stochastic solver of the subproblem is designed in RWSADMM, which provably converges to the stationary point almost surely in expectation under mild conditions independent of the data distributions. Our theoretical and empirical results demonstrate the superiority of our proposed algorithm over state-of-the-art personalized Federated Learning (FL) algorithms, providing empirical evidence for the effectiveness of our approach.
This paper is relevant to two distinct research areas, which are reviewed in two separate sections: FL frameworks that tackle data heterogeneity and ADMM-based FL frameworks.
FL with data heterogeneity FL was initially introduced as FedAvg, a client-server-based framework that didn't allow clients to personalize the global model to their local data [1]. This led to poor convergence due to local data heterogeneity, negatively impacting the global FL model's performance on individual clients. Recent works proposed a two-stage approach to personalize the global model. In the first stage, the FL global model is trained similarly to FedAvg. However, the second stage is included to personalize the global model for each local client through additional training on their local data. [21] demonstrated that FedAvg is equivalent to Reptile, a new meta-learning algorithm, when each client collects the same amount of local data. To learn a global model that performs well for most participating clients, [22] proposed an improved version, Per-FedAvg. This new variant aims to learn a good initial global model that can adapt quickly to local heterogeneous data. An extension of Per-FedAvg, called pFedMe [23], introduced an  $\ell_2$ -norm regularization term to balance the agreement between local and global models and the empirical loss. [24] proposed Ditto, a multi-task learning-based FL framework that provides personalization while promoting fairness and robustness to byzantine attacks. Ditto uses a regularization term to encourage personalized local models to be close to the optimal global model. [25] proposed to interpolate local and global models to train local models while also contributing to the global model. However, scaling these approaches can be challenging due to high communication costs, reliance on strong assumptions about network connectivity, or the requirement to compute second-order gradients. Additionally, there is a potential for enhancing the algorithms' overall performance.
ADMM-based FL The Alternating Direction Method of Multipliers (ADMM) is a widely recognized algorithm that effectively tackles optimization problems across multiple domains. In recent studies, ADMM has been successfully employed in distributed learning, as demonstrated by several works [26, 27, 28, 29, 30, 31, 32, 33]. In Federated Learning (FL) context, researchers have proposed various methods to address specific challenges. For handling falsified data in Byzantine settings, [34] introduced a robust ADMM-based approach. To mitigate local computational burdens in FL, [35] developed an inexact ADMM-based algorithm suitable for edge learning configurations. FL itself enables local training without the need to share personal data between clients and the server. Despite the advantages of FL, there is still a concern regarding clients' privacy. Analyzing the parameter differences in the trained models uploaded by each client can compromise their privacy. To tackle this issue, [36] proposed an inexact ADMM-based federated learning algorithm that incorporates differential privacy (DP) techniques [37]. By leveraging DP, the algorithm enhances privacy protection during the FL process. These ADMM-based frameworks also have high communication costs, ranging from  $O(n)$  to  $O(n^2)$  per iteration, depending on the network's density with  $n$  clients. [38] introduced a Proximal Primal-Dual Algorithm (Prox-PDA) to enable network nodes to compute the set of first-order stationary solutions collectively. Moreover, these algorithms do not account for data heterogeneity in their framework designs, leading to performance deterioration in such scenarios. The most similar algorithm to RWSADMM is called Walkman [39]. Walkman is an ADMM-based framework utilizing the random walk technique for distributed optimization. In
Walkman, the communication and computation costs are reduced by activating only one agent at each step. Compared to other ADMM-based approaches, including Walkman, RWSADMM has several distinctive features. RWSADMM leverages stochastic approximation to reduce computation costs per iteration and enforces hard inequality constraints instead of consensus to manage heterogeneous data, resulting in increased robustness. Additionally, RWSADMM considers the dynamic graph, allowing it to adapt to changing network conditions and potentially improve communication efficiency. RWSADMM also incorporates a hard constraint parameter  $\epsilon$  to promote local proximity among clients instead of using a regularization term as Walkman does to promote client consensus. This approach better balances personalization and global optimization. Finally, while Walkman is fully distributed without server involvement, RWSADMM is a server-based approach in which the server aggregates information from a small group of clients in each computation round.
3 Random Walk Stochastic ADMM (RWSADMM)
Before delving into the specifics of the proposed algorithm, we present the key notation used throughout this research.  $\mathbf{x} \in \mathbb{R}^d$  represents a vector with length  $d$  and  $\mathbf{e}$  is defined as a vector with entries equal to 1 and  $\mathbf{X} \in \mathbb{R}^{l \times d}$  depicts a matrix with  $l$  rows and  $d$  columns.  $[\mathbf{x}]_i$  represents the  $i$ th element of vector  $\mathbf{x}$  and  $[\mathbf{X}]_{ij}$  is the  $(i,j)$ th element of matrix  $\mathbf{X}$ .  $[\mathbf{X}]_i$ ,  $[\mathbf{X}]_j$  represent the  $i$ th row and  $j$ th column of matrix  $\mathbf{X}$ , respectively.  $(\nabla f(\mathbf{x}))_j$  is used to denote the  $j$ th entry of the gradient of  $f(\mathbf{x})$ . The inner product of  $A$  and  $B$  is shown as  $\langle A,B\rangle$ .  $\mathbb{E}_t[\cdot]$  indicates the expectation given the past  $\xi_1,\ldots,\xi_{t-1}$ .  $\odot$  represents the Hadamard product/element-wise product and  $\otimes$  represents the Kronecker product between two matrices. Finally, Norm  $\mathfrak{p}$  of vector  $\mathbf{x}$  is denoted as  $\| \mathbf{x} \|_p^p = \sum_{i=1}^d |x_i|^p$ ,  $\mathbf{x} \in \mathbb{R}^d$  and Frobenius norm of matrix  $\mathbf{X}$  is written as  $\| \mathbf{X} \|_F = \sqrt{\sum_{i=1}^n \sum_{j=1}^m |x_{ij}|^2}$ .
Let us first define our Mobilizing FL problem. Mobilizing FL, which involves a mobile server, can be formulated as an optimization problem on a connected graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ . The graph comprises a set of  $n$  clients, represented as  $\mathcal{V} = v_{1},v_{2},\ldots ,v_{n}$ , and a set of  $m$  edges, denoted as  $\mathcal{E}$ . The objective is to minimize the average loss function across all clients while adhering to inequality constraints that ensure local proximity among the clients' respective local models. The optimization problem can be formulated mathematically as follows:
$$
\min  _ {\mathbf {x} _ {1: n} \in \mathbb {R} ^ {p}} \frac {1}{n} \sum_ {i = 1} ^ {n} f _ {i} \left(\mathbf {x} _ {i}\right) \quad s. t. \quad \left| \mathbf {x} _ {i} - \mathbf {x} _ {j} \right| \leq \epsilon_ {i}, \forall i \in \{1, \dots , n \}, \forall j \in \mathcal {N} (i) / v _ {i}. \tag {1}
$$
where  $f_{i}(\mathbf{x}_{i})$  represents the local loss function with the model parameter as  $\mathbf{x}_i$  for the client  $i$ , the vertex set  $\mathcal{N}(i)$  contains client  $i$  and its neighboring clients, and  $\epsilon_{i}$  is the non-consensus relaxation between local neighboring clients to replace model consensus requirement in typical FL. In our proposed FL method, we model the server's movement as a dynamic Markov Chain, introducing a dynamic element to the traditional ADMM-based approach. This work is the first to consider a dynamic mobile server within the ADMM-based FL framework. In RWSADMM, client-server communication occurs only when the server is close to a client. The sequence of client indices that are updated, denoted as  $i_k$ , evolves based on a non-homogeneous Markov Chain with a state space of  $1,\dots ,n$  [40]. To describe the transition dynamics of the Markov Chain, we employ the non-homogeneous Markovian transition matrix  $\mathbf{P}(k)$ , which represents the probabilities of transitioning between clients at time  $k$ . Specifically, the conditional probability of selecting client  $j$  as the next client, given that client  $i$  is the current client, is defined as:
$$
[ \mathbf {P} (k) ] _ {i, j} = P r \left\{i _ {k + 1} = j \mid i _ {k} = i \right\} \in [ 0, 1 ] \tag {2}
$$
Additionally, it is assumed that the server determines the probability of all possible locations for its next destination based on the transition matrix  $\mathbf{P}(k)$  at time  $k$ . This provides a probabilistic approach to server navigation, allowing it to move around the network more effectively. To guarantee convergence, RWSADMM depends on the frequency of revisiting each agent. This quality is described by the mixing time of the algorithm. An assumption for the mixing time is as follows:
Assumption 3.1. The random walk  $(i_k)_{k\geq 0},v_{i_k}\in \mathcal{V}$  forms an irreducible and aperiodic (ergodic) Markov Chain with transition probability matrix of  $\mathbf{P}(k)\in \mathbb{R}^{n\times n}$  defined in Eq. (2) and stationary distribution  $\pi$  satisfying  $\lim_{k\to \infty}\pmb{\pi}^T\mathbf{P}(k) = \pmb{\pi}^T$ . The mixing time (for a given  $\delta >0$ ) is defined as the smallest integer  $\tau (\delta)$  such that  $\forall i\in V$
$$
\left\| \left[ \mathbf {P} (k) ^ {\tau (\delta)} \right] _ {i} - \boldsymbol {\pi} ^ {\mathbf {T}} \right\| \leq \delta \boldsymbol {\pi} _ {*} \tag {3}
$$
where  $\pi_{*} \coloneqq \min_{i \in \mathcal{V}} \pi_{i}$ . This inequality states the fact that regardless of the current state  $i$  and time  $k$ , the probability of visiting each state  $j$  after  $\tau(\delta)$  steps is  $(\delta \pi_{*})$ -close to  $\pi_{j}$ , that is,  $\forall i, j \in \mathcal{V}$ ,
$$
\left\| \left[ \mathbf {P} (k) ^ {\tau (\delta)} \right] _ {i j} - \boldsymbol {\pi} _ {j} \right\| \leq \delta \boldsymbol {\pi} _ {*} \tag {4}
$$
Eq. (4) is used to prove the sufficient descent of a Lyapunov function  $L_{\beta}$  in Section 3.1. Let's also define
$$
\mathbf {P} _ {\max } = \lim  _ {k = + \infty} \left\{\mathbf {P} | [ \mathbf {P} ] _ {i j} = \max  _ {k} [ \mathbf {P} (k) ] _ {i j} \right\}, \tag {5}
$$
from which one can further obtain  $\| \mathbf{P}(k)\| \leq \| \mathbf{P}_{max}\|$  for all  $k$ . Namely, the matrix  $P_{max}$  is computed as the element-wise maximum matrix among all the matrices  $P(k)$ , for  $k = 0,\dots ,\infty$ . Therefore, the mixing time requirement in Eq. (3) is guaranteed to hold for
$$
\tau (\delta) = \left\lceil \frac {1}{1 - \sigma (\mathbf {P})} \ln \frac {\sqrt {2}}{\delta \pi_ {*}} \right\rceil \stackrel {(a)} {\leq} \left\lceil \frac {1}{1 - \sigma (\mathbf {P} _ {\max })} \ln \frac {\sqrt {2}}{\delta \pi_ {*}} \right\rceil \tag {6}
$$
where  $\sigma(\mathbf{P}) \coloneqq \sup \{\|f^T\mathbf{P}\| / \|f\| : f^T\mathbf{1} = 0, f \in \mathbb{R}^n\}$ . Using Eq. (5), we have  $\forall \mathbf{P}$ ,  $\sigma(\mathbf{P}) \leq \sigma(\mathbf{P})_{max}$  and the inequality  $(a)$  can be inferred.
In this section, we derive RWSADMM by integrating random walk and stochastic inexact approximation techniques into ADMM. Considering  $\mathbf{X} \coloneqq row(\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n) \in \mathbb{R}^{p \times n}$ ,  $F(\mathbf{X}) \coloneqq \sum_{i=1}^{n} f_i(\mathbf{x}_i)$ , where the operation  $row(.)$  refers to row-wise stacking of vectors  $\mathbf{x}_i$ 's. The mobilizing FL problem (1) can be expressed as:
$$
\min  _ {\mathbf {y} _ {1: n}, \mathbf {X}} \frac {1}{n} F (\mathbf {X}) \quad \text {s . t .} \left| \mathbf {1} \otimes \mathbf {y} _ {i} - \mathbf {X} _ {\mathcal {N} (i)} \right| \leq \mathbf {1} \otimes \boldsymbol {\epsilon} _ {i} / 2, \forall i = 1, \dots , n \tag {7}
$$
where  $\mathbf{1} = [1 1 \ldots 1] \in \mathbb{R}^{n_i}$ ,  $n_i$  denotes the volume of the vertex set  $\mathcal{N}(i)$ . The constraint implies that  $|\mathbf{x}_i - \mathbf{x}_j| \leq \epsilon_i$ ,  $\forall i = 1 \ldots n$  and  $\forall j \in \mathcal{N}(i)$  through the triangle inequality.  $\mathbf{y}_i$  stored on the server is necessarily introduced as a local proximity of  $\mathcal{N}(i)$ . We can obtain the augmented Lagrangian for problem (7)
$$
L _ {\beta} \left(\mathbf {y} _ {1: n}, \mathbf {X}, \mathbf {Z} _ {1: n}\right) = \frac {1}{n} \left[ F (\mathbf {X}) + \sum_ {i = 1} ^ {n} \left\langle \mathbf {Z} _ {i}, \left| \mathbf {1} \otimes \mathbf {y} _ {i} - \mathbf {X} _ {\mathcal {N} (i)} \right| - \varepsilon_ {i} \right\rangle + \frac {\beta}{2} \sum_ {i = 1} ^ {n} \| \left| \mathbf {1} \otimes \mathbf {y} _ {i} - \mathbf {X} _ {\mathcal {N} (i)} \right| - \varepsilon_ {i} \| _ {F} ^ {2} \right] \tag {8}
$$
where  $\varepsilon_{i} = \epsilon_{i} / 2$  and  $\mathbf{Z}_i\in \mathbb{R}^{n_i p}$  are the dual variable and  $\beta >0$  is the barrier parameter. The RWSADMM algorithm minimizes the augmented Lagrangian  $L_{\beta}(\mathbf{y}_{1:n},\mathbf{X},\mathbf{Z}_{1:n})$  in an iterative manner. At each iteration  $k$ , only a subset of clients covered by the mobilized server, the clients in  $\mathcal{N}(i_k)$ , participate in the federated update. The following updates are performed:
$$
\mathbf {x} _ {i _ {k}} = \arg \min _ {\mathbf {x} _ {i _ {k}}} L _ {\beta} (\mathbf {y} _ {i _ {k}} ^ {\prime}, \mathbf {x} _ {i _ {k}}, \mathbf {z} _ {i _ {k}} ^ {\prime}), \quad \mathbf {y} _ {i _ {k}} = \arg \min _ {\mathbf {y} _ {i _ {k}}} L _ {\beta} (\mathbf {y} _ {i _ {k}}, \mathbf {X} _ {\mathcal {N} (i _ {k})}, \mathbf {Z} _ {\mathcal {N} (i _ {k})} ^ {\prime}),
$$
where  $\mathbf{y}_{i_k}^{\prime},\mathbf{x}_{i_k}$ , and  $\mathbf{z}_{i_k}^{\prime}$  denote the groups of variables of the local parameters stored by client  $i_{k}$  at the  $(k - 1)th$  update. After solving these subproblems, we update the multiplier  $\mathbf{z}_{i_k}$  as follows:
$$
\mathbf {z} _ {i _ {k}} = \mathbf {z} _ {i _ {k}} ^ {\prime} + \beta \left(\left| \mathbf {y} _ {i _ {k}} - \mathbf {x} _ {i _ {k}} \right| - \varepsilon_ {i}\right),
$$
Next, we derive the solver of each subproblem. The three steps are noted as Updating  $\mathbf{x}_{i_k}$ , Updating  $\mathbf{y}_{i_k}$ , and Updating  $\mathbf{z}_{i_k}$ .
$$
\text {U p d a t i n g} \mathbf {x} _ {i _ {k}}: \quad \min  _ {\mathbf {x} _ {i _ {k}}} \left[ f _ {i _ {k}} \left(\mathbf {x} _ {i _ {k}}\right) + \left\langle \mathbf {z} _ {i _ {k}} ^ {\prime}, \left| \mathbf {y} _ {i _ {k}} ^ {\prime} - \mathbf {x} _ {i _ {k}} \right| - \varepsilon_ {i _ {k}} \right\rangle + \frac {\beta}{2} \| \left| \mathbf {y} _ {i _ {k}} ^ {\prime} - \mathbf {x} _ {i _ {k}} \right| - \varepsilon_ {i _ {k}} \| _ {2} ^ {2} \right] \tag {9}
$$
The Problem (9) can be solved iteratively, consuming significant computational resources for the local clients. Furthermore, the computational complexity increases as the local dataset grows, as is often true in real-world applications. By utilizing the stochasticity and first-order subgradient expansion, we arrive at a more computationally efficient approximation of the original problem in Eq. (10).
$$
\min  _ {\mathbf {x} _ {i _ {k}}} \left[ g _ {i _ {k}} \left(\mathbf {x} _ {i _ {k}} ^ {\prime}, \boldsymbol {\xi} _ {i _ {k}}\right) \left(\mathbf {x} _ {i _ {k}} - \mathbf {x} _ {i _ {k}} ^ {\prime}\right) + \left\langle \mathbf {z} _ {i _ {k}} ^ {\prime}, \left| \mathbf {y} _ {i _ {k}} ^ {\prime} - \mathbf {x} _ {i _ {k}} \right| - \varepsilon_ {i _ {k}} \right\rangle + \frac {\beta}{2} \| \left| \mathbf {y} _ {i _ {k}} ^ {\prime} - \mathbf {x} _ {i _ {k}} \right| - \varepsilon_ {i _ {k}} \| _ {2} ^ {2} \right] \tag {10}
$$
In Eq. (10),  $\pmb{\xi}_{i_k}$  denotes one or a few samples randomly selected by client  $i_k$  from its feature set and their ground truth labels in pairs at the  $k$ -th iteration. The function  $g_{i_k}(\mathbf{x}_{i_k}',\pmb{\xi}_{i_k})$  is defined as