# Global-Entropy Driven Exploration with Distributed Models under Sparsity Constraints
## 2 Signal Models for Static Process Estimation
### Distributed Signal Models
#### 2.2.2 Splitting-over-Features  
In SOF, each agent infers a local model of the observed phenomenon, while the measurement data are shared among the agents. In our case, this implies that each agent defines its own separate set of \(N_{k}\) features, which might not necessarily be the same for the agents, and \(N_{k}\) is related to the global number of features as \(N=\sum_{k=1}^{K}N_{k}\). Here, we will assume that each agent's features are defined at \(\mathbf{X}_{N_{k}}\), where an agent \(k\) has made a measurement, i.e., \(\mathbf{X}_{N_{k}}=\mathbf{X}_{M_{k}}=[\mathbf{x}_{k}[1],\ldots,\mathbf{x}_{k}[M_{k}]]\). Then, the local design matrix in the SOF case can be defined as \(\mathbf{\Phi}_{k}=\mathbf{\Psi}(\mathbf{X}_{\mathbf{M}},\mathbf{X}_{N_{k}})\in\mathbb{R}^{M \times\mathbb{N}_{k}}\), where \(\mathbf{\Phi}_{k}\) is used for a column-splitting of \(\mathbf{\Psi}\) to better distinguish SOF with SOE. Consequently, the central measurement model in (1) can be reformulated in a distributed way as:  
\[\mathbf{y}=\mathbf{\Phi}_{1}\mathbf{w}_{1}+\cdots+\mathbf{\Phi}_{K}\mathbf{w}_{K}+\mathbf{ \epsilon}=\sum_{k=1}^{K}\mathbf{\Phi}_{k}\mathbf{w}_{k}+\mathbf{\epsilon}. \tag{11}\]  
Note that in contrast to SOE, here, all agents naturally experience the same measurement noise. By inserting (11) into (2), we get a distributed LASSO as:  
\[\underset{\mathbf{w}_{k}}{\text{minimize}} \frac{\lambda}{2}\left\|\sum_{k=1}^{K}\mathbf{\Phi}_{k}\mathbf{w}_{k}- \mathbf{y}\right\|^{2}+\delta\sum_{k=1}^{K}\left\|\mathbf{w}_{k}\right\|_{1}. \tag{12}\]  
Again, we are aiming for a problem formulation, which is suited for a solution with ADMM. Thus, we define latent variables \(\mathbf{v}_{k}=\mathbf{\Phi}_{k}\mathbf{w}_{k}\), \(k=1,\ldots,K\), that capture the response of individual agents and rewrite (12) as:  
\[\underset{\mathbf{w}_{k},\mathbf{v}_{k}}{\text{minimize}} \frac{\lambda}{2}\left\|\sum_{k=1}^{K}\mathbf{v}_{k}-\mathbf{y}\right\|^{ 2}+\delta\sum_{k=1}^{K}\left\|\mathbf{w}_{k}\right\|_{1}\] (13) subject to \[\mathbf{\Phi}_{k}\mathbf{w}_{k}=\mathbf{v}_{k}\;\forall\;k=1,\ldots,K.\]  
We likewise solve the optimization (13) by using the ADMM algorithm [9]. In this case, the augmented Lagrangian for (13) can be written as:  
\[\mathcal{L}_{\text{SOF}}(\mathbf{w}_{1},\ldots,\mathbf{w}_{K},\mathbf{v}_{1}, \ldots,\mathbf{v}_{K},\mathbf{s})=\frac{\lambda}{2}\left\|\sum_{k=1}^{K}\mathbf{v}_{k}- \mathbf{y}\right\|^{2}+\delta\sum_{k=1}^{K}\left\|\mathbf{w}_{k}\right\|+\frac{K\rho} {2}\sum_{k=1}^{K}\left\|\mathbf{\Phi}_{k}\mathbf{w}_{k}-\mathbf{v}_{k}+\mathbf{s}\right\|_{2}^ {2}, \tag{14}\]  
where \(\mathbf{s}\in\mathbb{R}^{M}\) is a dual variable of the ADMM algorithm. Now, using again the scaled version of (14) (see [9] Section 3.1 for more details), we minimize \(\mathcal{L}_{\text{SOF}}\) using the following algorithm. 1. At iteration \(i=0\), initialize \(\mathbf{w}_{k}^{[0]}=0\), \(\mathbf{s}^{[0]}=0\) and \(\mathbf{v}^{[0]}=0\). 2. Compute an estimate \(\mathbf{w}_{k}^{[i+1]}\) by solving: \[\mathbf{w}_{k}^{[i+1]}=\underset{\mathbf{w}_{k}}{\text{min}}\left(\delta\left\|\mathbf{w} _{k}\right\|_{1}+\frac{\rho}{2}\left\|\mathbf{\Phi}_{k}\mathbf{w}_{k}-\mathbf{\Phi}_{k}\bm {w}_{k}^{[i]}-\mathbf{v}^{[i]}+\overline{\mathbf{\[PERSON]}\mathbf{w}}^{[i]}+\mathbf{s}^{[i]} \right\|_{2}^{2}\right).\] (15)
3. Use the averaged consensus algorithm of the network of agents to compute: \[\overline{\mathbf{\[PERSON]}\mathbf{w}}^{[i+1]}=\frac{1}{K}\sum_{k=1}^{K}\mathbf{ \[PERSON]}_{k}\mathbf{w}_{k}^{[i+1]}.\] (16)
4. Compute an estimate of the latent variable \(\mathbf{v}^{[i+1]}\) of the weight vector by solving: \[\mathbf{v}^{[i+1]}=(\lambda K+\rho)^{-1}\left(\lambda\mathbf{y}+\rho\left(\overline{ \mathbf{\Phi}\mathbf{w}}^{[i+1]}+\mathbf{s}^{[i]}\right)\right).\] (17)
5. Update the dual variable: \[\mathbf{s}^{[i+1]}=\mathbf{s}^{[i]}+\overline{\mathbf{\Phi}\mathbf{w}}^{[i+1]}-\mathbf{v}^{[i+1]}.\] (18)
6. The algorithm converges if \(\left\|\mathbf{s}^{[i+1]}-\mathbf{s}^{[i]}\right\|<t\) is below some threshold \(t\).