# Global-Entropy Driven Exploration with Distributed Models under Sparsity Constraints
## 2 Signal Models for Static Process Estimation
### Distributed Signal Models
#### 2.2.1 Splitting-over-Examples  
In SOE each agent \(k\) makes its own \(M_{k}\) local measurements \(\{y[m],\mathbf{x}[m]\}\), \(m=1,\ldots,M_{k}\). The measurements of the \(k\)-th agent can therefore be arranged as \(\mathbf{y}_{k}=[y_{k}[1],\ldots,y_{k}[M_{k}]]^{T}\in\mathbb{R}^{M_{k}}\) and \(\mathbf{X}_{k}=[\mathbf{x}_{k}[1],\ldots,\mathbf{x}_{k}[M_{k}]]^{T}\). Hence, each agent \(k\) has a part of the global measurements as \(\mathbf{y}=[y_{1}^{T},\ldots,y_{K}^{T}]^{T}\) and \(\mathbf{X}_{\text{M}}=[\mathbf{X}_{1}^{T},\ldots,\mathbf{X}_{K}^{T}]^{T}\). The local number of measurements \(M_{k}\) is related to the global measurements as \(M=\sum_{k=1}^{K}M_{k}\). Prior to the estimation and data collection, the agents need to agree on the selection of the points \(\mathbf{X}_{\text{N}}\), features or support points that effectively define the discretization of the exploration environment \(\mathcal{X}\). The number of these points \(N\) has to be selected large enough to ensure that the coverage of the exploration environment is sufficient. This number is a key trade-off parameter between efficiency and computational complexity of the resulting inference and exploration procedure, as we will show later. Using local measurements, agent \(k\) defines its design matrix \(\mathbf{\Psi}_{k}=\mathbf{\Psi}(\mathbf{X}_{k},\mathbf{X}_{\text{N}})\in\mathbb{R}^{M_{k}\times N}\). Correspondingly, in SOE formulation, the model (1) is transformed as follows:  
\[\mathbf{y}_{k}=\mathbf{\Psi}_{k}\mathbf{w}+\mathbf{e}_{k},\ \forall\ k=1,\ldots,K, \tag{3}\]  
with \(\mathbf{e}_{k}\sim\mathcal{N}(0,\lambda^{-1}\mathbf{I})\). Note that within the SOE formulation, agents can experience different perturbations in \(\mathbf{e}_{k}\). Using (3), we can now recast (2) into a distributed optimization problem as:  
\[\underset{\mathbf{w}}{\text{minimize}}\sum_{k=1}^{K}\frac{\lambda}{2}\left\|\mathbf{ \Psi}_{k}\mathbf{w}-\mathbf{y}_{k}\right\|^{2}+\delta\left\|\mathbf{w}\right\|_{1} \tag{4}\]  
To use ADMM, we introduce new local variables \(z_{k}\in\mathbb{R}^{N}\ \forall k=1,\ldots,K\) and rewrite (4) as:  
\[\underset{z_{k},w}{\text{minimize}} \sum_{k=1}^{K}\frac{\lambda}{2}\left\|\mathbf{\Psi}_{k}z_{k}-\mathbf{y}_{k }\right\|^{2}+\delta\left\|\mathbf{w}\right\|_{1}\] subject to \[\mathbf{w}=z_{k}\ \forall\ k=1,\ldots,K. \tag{5}\]  
Figure 1: The colored areas represent an individual agent, and grey areas are commonly known by all agents. The matrices in these figures are arranged according to (1). (**a**) SOE learning approach: the weight vector \(\mathbf{w}\) is commonly known by all agents; (**b**) SOF learning approach: the measurement vector \(\mathbf{y}\) and noise vector \(\mathbf{e}\) are commonly known by all agents. Note that the local variables \(z_{k}\) play the role of weights \(\mathbf{w}\) in the LASSO objective function. In contrast to (2), however, the additional constraint in (5) ensures that the local weight estimates of \(\mathbf{z}_{k}\) will eventually convergence to the same fixed point. The optimization (5) is now suited for ADMM [9]. In the following, we briefly outline the ADMM algorithm for solving (5) and refer the interested reader to [9] for more details. To define the objective function, ADMM uses an augmented Lagrangian such that:  
\[\mathcal{L}_{\text{SOE}}(\mathbf{w},\mathbf{z}_{1},\ldots,\mathbf{z}_{K},\mathbf{u}_{1},\ldots,\mathbf{u}_{K})=\sum_{k=1}^{K}\frac{\lambda}{2}\left\|\mathbf{\Psi}_{k}z_{k}-\mathbf{y}_{ k}\right\|^{2}+\delta\left\|\mathbf{w}\right\|_{1}+\mathbf{u}_{k}^{T}(\mathbf{w}-\mathbf{z}_{k})+ \frac{\rho}{2}\left\|\mathbf{w}-\mathbf{z}_{k}\right\|_{2}^{2}, \tag{6}\]  
with \(\mathbf{u}_{k}\in\mathbb{R}^{N}\) being the dual variable and \(\rho>0\) defining the ADMM penalty parameter. The minimizer of the augmented Lagrangian (6) with respect to \(\mathbf{w}\), \(z_{k}\) and \(\mathbf{u}_{k}\), \(k=1,\ldots,K\) will be a solution to (5) [9].