Title: Block Sparse Bayesian Learning: A Diversified Scheme
Venue: NeurIPS 2024
Accepted: yes
Keywords: Compressed Sensing

Diversified Block Sparse Prior

Sparse Bayesian Learning

Lagrange Dual Ascent.
Abstract: This paper introduces a novel prior called Diversified Block Sparse Prior to characterize the widespread block sparsity phenomenon in real-world data. By allowing diversification on intra-block variance and inter-block correlation matrices, we effectively address the sensitivity issue of existing block sparse  learning methods to pre-defined block information, which enables adaptive block estimation while mitigating the risk of overfitting. Based on this, a diversified block sparse Bayesian learning method (DivSBL) is proposed, utilizing EM algorithm and dual ascent method for hyperparameter estimation. Moreover, we establish the global and local optimality theory of our model. Experiments validate the advantages of DivSBL over existing algorithms.

Block Sparse Bayesian Learning: A Diversified Scheme

Abstract

This paper introduces a novel prior called Diversified Block Sparse Prior to characterize the widespread block sparsity phenomenon in real-world data. By allowing diversification on intra-block variance and inter-block correlation matrices, we effectively address the sensitivity issue of existing block sparse learning methods to pre-defined block information, which enables adaptive block estimation while mitigating the risk of overfitting. Based on this, a diversified block sparse Bayesian learning method (DivSBL) is proposed, utilizing EM algorithm and dual ascent method for hyperparameter estimation. Moreover, we establish the global and local optimality theory of our model. Experiments validate the advantages of DivSBL over existing algorithms.

1 Introduction

Sparse recovery through Compressed Sensing (CS), with its powerful theoretical foundation and broad practical applications, has received much attention [1]. The basic model is considered as

where y ∈ R M × 1 is the measurement (or response) vector and Φ ∈ R M × N is a known design matrix, satisfying the Unique Representation Property (URP) condition [2]. x ∈ R N × 1 ( N glyph[greatermuch] M ) is the sparse vector to be recovered. In practice, x often exhibits transform sparsity, becoming sparse in a transform domain such as Wavelet, Fourier, etc. Once the signal is compressible in a linear basis Ψ , in other words, x = Ψw where w exhibits sparsity, and ΦΨ satisfies Restricted Isometry Constants (RIP) [3], then we can simply replace x by Ψw in (1) and solve it in the same way. Classic algorithms for compressive sensing and sparse regression include Lasso [4], Sparse Bayesian Learning (SBL) [5], Basis Pursuit (BP) [6], Orthogonal Matching Pursuit(OMP) [7], etc. Recently, there have been approaches that involve solving CS problems through deep learning [8; 9; 10].

However, deeper research into sparse learning has shown that relying solely on the sparsity of x is insufficient, especially with limited samples [11; 12]. Widely encountered real-world data, such as image and audio, often exhibit clustered sparsity in transformed domains [13]. This phenomenon, known as block sparsity, means the sparse non-zero entries of x appear in blocks [11]. Recent years, block sparse models have gained attention in machine learning, including sparse training [14], adversarial learning [15], image restoration [16; 14], (audio) signal processing [17; 18] and many other areas. Generally, the block structure of x with g blocks is defined by

where d i ( i = 1 ...g ) represent the size of each block, which are not necessarily identical. Suppose only k ( k glyph[lessmuch] g ) blocks are non-zero, indicating that x is block sparse. Up to now, several methods have been proposed to recover block sparse signals. They are mainly divided into two categories.

Block-based Classical algorithms for processing block sparse scenarios include Group-Lasso [19; 20; 21], Group Basis Pursuit [22], Block-OMP [11]. Blocks are assumed to be static with a fixed preset size. Furthermore, Temporally-SBL (TSBL) [23] and Block-SBL (BSBL) [24; 25], based on Bayesian models, provide refined estimation of correlation matrices within blocks. However, they assume elements within one block tend to be either zero or non-zero simultaneously. Although they can estimate intra-block correlation with high accuracy in block-level recovery, they require preset choices of suitable block sizes and patterns, which are too rigid for many practical applications.

Pattern-based StructOMP [26] is a pattern-based greedy algorithm allowing structures, which is a generalization of group sparsity. Another classic model named Pattern-Coupled SBL (PC-SBL) [27; 28], does not have a predefined requirement for block size as well. It utilizes a Bayesian model to couple the signal variances. Building upon PC-SBL, Burst PC-SBL, proposed in [29], is employed for the estimation of mMIMO channels. While pattern-based algorithms address the issue of explicitly specifying block patterns in block-based algorithms, these models provide a coarse characterization of the intra-block correlation, leading to a loss of structural information within the blocks.

In this paper, we introduce a diversified block sparse Bayesian framework that incorporates diversity in both variance within the same block and intra-block correlation among different blocks. Our model not only inherits the advantages of block-based methods on block-level estimation, but also addresses the longstanding issues associated with such algorithms: the diversified scheme reduces sensitivity to a predefined block size or specified block location, hence accommodates general block sparse data. Based on this model, we develop the DivSBL algorithm, and also analyze both the global minimum and local minima of the constrained cost function (likelihood). The subsequent experiments illustrate the superiority of proposed diversified scheme when applied to real-life block sparse data.

2 Diversified block sparse Bayesian model

We consider the block sparse signal recovery, or compressive sensing question in the noisy case

where n ∼ N (0 , β -1 I ) represents the measurement noise, and β is the precise scalar. Other symbols have the same interpretations as (1). The signal x exhibits block-sparse structure in (2), yet the block partition is unknown. For clarity in description, we assume that all blocks have equal size L , with the total dimension denoted as N = gL . Henceforth, we presume that the signal x follows the structure:

In Sections 2.1.1 and 5.2, we clarify that this assumption is made without loss of generality. In fact, our algorithm can automatically adjust L to an appropriate size, expanding or contracting as needed.

2.1 Diversified block sparse prior

The Diversified Block Sparse prior is proposed in the following scheme. Each block x i ∈ R L × 1 is assumed to follow a multivariate Gaussian prior

in which, G i represents the Diversified Variance matrix, and B i represents the Diversified Correlation matrix, with detailed formulations in Sections 2.1.1 and 2.1.2. Therefore, the prior distribution of the entire signal x is denoted as

where Σ 0 = diag { G 1 B 1 G 1 , G 2 B 2 G 2 , · · · , G g B g G g } . The dependency in this hierarchical model is shown in Figure.1.

2.1.1 Diversified intra-block variance

We first execute diversification on variance. In (5), G i is defined as

and B i ∈ R L × L is a positive definite matrix capturing the correlation within the i -th block. According to the definition of Pearson correlation, the covariance term G i B i G i in (5) can be specified as

:Measurements

:Sparse vector to be recovered

：Covariance hyperparameters

X11

:Noise level

G1B1G1

Pre-defined Block

XIL

y1

True Block

xg1

yM

GgBgGg

xgL

DemonstrationRationale

Left:When thetrueblockislocatedwithin the presetblock.

Right: When the true block is located across the presetblock.

The variances Yi. corresponding to the non-zero (~0) or zero positions in x. will be automatically learnedasnon-zeroor zerovalues through posteriorinference,hence refine tothe trueblock.

L

LT

xi-

L

L

L

0

~0

0

0

~0

Yi.

√ √ √ √ L 1 L 2       , where ρ i sk ( ∀ s, k = 1 · · · L ) are the elements in correlation matrix B i , serving as a visualization of covariance with displayed structural information.

Now it is evident why assuming equal block sizes L is insensitive. For the sake of clarity, we denote the true size of the i -th block x i as L i T . As illustrated in Figure 2, when the true block falls within the preset block, the variances γ i · corresponding to the non-zero positions in x i will be learned as non-zero values through posterior inference, while the variances at zero positions will automatically be learned as zero. When the true block is positioned across the preset block, several blocks of the predefined size L covered by the actual block will be updated together, and likewise, variances will be learned as either zero or non-zero. In this way, both of the size and location of the blocks will be automatically learned through posterior inference on the variances.

2.1.2 Diversified inter-block correlation

Due to limited data and excessive parameters in intra-block correlation matrices B i ( ∀ i ) , previous works correct their estimation by imposing strong correlated constraints B i = B ( ∀ i ) to overcome overfitting [24]. Recognizing that correlation matrices among different blocks should be diverse yet still exhibit some correlation, we apply a weak-correlated constraint to diversify B i in the model.

Here we introduce novel weak constraints on B i , specifically,

where ψ : R L 2 → R is the weak constraint function and B is obtained from the strong constraints B i = B ( ∀ i ) , as detailed in Section 3.2. Weak constraints (8) not only capture the distinct correlation structure but also avoid overfitting issue arising from the complete independence among different B i .

Furthermore, the constraints imposed here not only maintain the global minimum property of our algorithm, as substantiated in Section 4, but also effectively enhance the convergence rate of the algorithm. There are actually gL ( L +1) / 2 constraints in the strong correlated constraints B i = B ( ∀ i ) , while with (8), the number of constraints significantly decreases to g , yielding acceleration on the convergence rate. The experimental result is shown in Appendix A.

In summary, the prior based on (5), (6), (7) and (8) is defined as diversified block sparse prior .

2.1.3 Connections to classical models

Note that the classical Sparse Bayesian Learning models, Relevance Vector Machine (RVM) [5] and Block Sparse Bayesian Learning (BSBL) [23], are special cases of our model.

Connection to RVM Taking B i as identity matrix, diversified block sparse prior (6) immediately degenerates to RVM model

which means ignoring the correlation structure.

Connection to BSBL When G i is scalar matrix √ γ i I , the formulation (5) becomes

which is exactly BSBL model. In this case, all elements within a block share common variance γ i .

2.2 Posterior estimation

By observation model (3), the Gaussian likelihood is

With prior (6) and likelihood (11), the diversified block sparse posterior distribution of x can be derived based on Bayes' theorem as

After estimating all hyperparameters in (12), i.e, ˆ Θ = { { ˆ G i } g i =1 , { ˆ B i } g i =1 , ˆ β } , as described in Section 3, the Maximum A Posterior (MAP) estimation of x is formulated as

3 Bayesian inference: DivSBL algorithm

3.1 EMformulation

To estimate Θ = {{ G i } g i =1 , { B i } g i =1 , β } , either Type-II Maximum Likelihood [30] or ExpectationMaximization (EM) formulation [31] can be employed. Following EM procedure, our goal is to maximize p ( y ; Θ) , or equivalently log p ( y ; Θ) . Defining objective function as L (Θ) , the problem can be expressed as

where Σ y = β -1 I + ΦΣ 0 Φ T . Then, treating x as hidden variable in E-step, we have Q function as

where Θ t -1 denotes the parameter estimated in the latest iteration. As indicated in (17), we have divided Q function into two parts: Q ( β ) glyph[defines] E x | y ;Θ t -1 [log p ( y | x ; β )] depends solely on β , and Q ( { G i } g i =1 , { B i } g i =1 ) glyph[defines] E x | y ;Θ t -1 [log p ( x ; { G i } g i =1 , { B i } g i =1 )] only on { G i } g i =1 and { B i } g i =1 . Therefore, the parameters of these two Q functions can be updated separately.

In M-step, we need to maximize the above Q functions to obtain the estimation of Θ . As shown in Appendix B, the updating formula for γ ij , B i can be obtained as follows 2 :

where

where T ij and A ij are expressed as T ij = [ ( B -1 i ) j · glyph[circledot] diag( W i -j ) -1 ] · ( Σ i + µ i ( µ i ) T ) · j , A ij = ( B -1 i ) jj · ( Σ i + µ i ( µ i ) T ) jj , and W i -j = diag { √ γ i 1 , · · · , √ γ i,j -1 , 0 , √ γ i,j +1 , · · · , √ γ iL } . The update formula of β is derived in the same way as [23]. The learning rule is given by

3.2 Diversified correlation matrices by dual ascent

Now we propose the algorithm for solving the correlation matrix estimation problem satisfying (8). As mentioned in Section 2.1.2, previous studies have employed strong constraints B i = B ( ∀ i ) , i.e.,

In diversified scheme, we apply weak-correlated constraints (8) to diversify B i . Therefore, the problem of maximizing the Q function with respect to B i becomes

which is equivalent to (in the sense that both share the same optimal solution)

where B is already derived in (21). Therefore, by solving (P) , we will obtain diversified solution for correlation matrices B i , ∀ i . The constraint function ψ can be further categorized into two cases: explicit constraints and hidden constraints.

Explicit constraints with complete dual ascent Explicit functions such as the Frobenius norm, the logarithm of the determinant, etc., are good choices for ψ . An efficient way to solve this constrained optimization is to solve its dual problem (mostly refers to Lagrange dual [32]). Choosing ψ ( · ) as log det( · ) , the detailed solution process is outlined in Appendix C. And the update formulas for B i and multiplier λ i (dual variable) are given by

in which α k i represents the step size in the k -th iteration for updating the multiplier λ i ( i = 1 , · · · , g ). Convergence is only guaranteed if the step size satisfies ∑ ∞ k =1 α k i = ∞ and ∑ ∞ k =1 ( α k i ) 2 < ∞ [32]. In our experiment, we choose a diminishing step size 1 /k to ensure the convergence of the algorithm. The procedure, using dual ascent to diversify B i , is summarized in Algorithm 2 in Appendix C.

Hidden constraints with one-step dual ascent The weak correlated function ψ can also be chosen as hidden constraint without an explicit expression. Specifically, the solution to sub-problem (22) equipped with hidden constraints ψ corresponds exactly to one-step dual ascent in (23)(24). We summarize the proposition as follows:

Proposition 3.1. Define an explicit weak constraint function ζ : R n 2 → R . For the constrained optimization problem:

the stationary point ( { B k +1 i } g i =1 , { λ k i } g i =1 ) of the Lagrange function under given multipliers { λ k i } g i =1 satisfies:

Then there exists a constrained optimization problem with hidden weak constraint ψ : R n 2 → R :

such that ( { B k +1 i } g i =1 , { λ k i } g i =1 ) is a KKT pair of the above optimization problem.

Compared to explicit formulation, hidden weak constraints, while ensuring diversification on correlation, significantly accelerate the algorithm's speed by requiring only one-step dual ascent for updating. Here, we set ζ ( · ) as log det( · ) , actually solving the optimization problem under corresponding hidden constraint ψ . The comparison of computation time between explicit and hidden constraints, proof of Proposition 3.1 and further explanations on hidden constraints are provided in Appendix D.

Considering that it's sufficient to model elements of a block as a first order Auto-Regression (AR) process [24] in which the intra-block correlation matrix is a Toeplitz matrix, we employ this strategy for B i . After estimating B i by dual ascent, we then apply Toeplitz correction to B i as

where r glyph[defines] m 1 m 0 is the approximate AR coefficient, m 0 represents the average of elements along the main diagonal of B i , and m 1 represents the average of elements along the main sub-diagonal of B i .

In conclusion, the Diversified SBL (DivSBL) algorithm is summarized as Algorithm 1 below.

Algorithm 1 DivSBL Algorithm

1: Input: Measurement matrix Φ , response y , initialized variance γ , prior's covariance Σ 0 , noise's variance 0

4 Global minimum and local minima

For the sake of simplicity, we denote the true signal as x true, which is the sparsest among all feasible solutions. The block sparsity of the true signal is denoted as K 0 , indicating the presence of K 0 blocks. Let ˜ G glyph[defines] diag ( √ γ 11 , · · · , √ γ gL ) , ˜ B glyph[defines] diag ( B 1 , · · · , B g ) , thus Σ 0 = ˜ G ˜ B ˜ G . Additionally, we assume that the measurement matrix Φ satisfies the URP condition [2]. We employed various techniques to overcome highly non-linear structure of γ in diversified block sparse prior (5), resulting in following global and local optimality theorems.

4.1 Analysis of global minimum

By introducing a negative sign to the cost function (16), we have the following result on the property of global minimum and the threshold for block sparsity K 0 .

(a)

(b)

(c)

(d)

0.12

Reconstruction NMSE

0.1

++ +|

Reconstruction CORR

0.99

--

0.08

860

-==…- **+

0.06

60

0.04

++

0.96

= = -

i

0.02

0.95

自

F -

0

0.94

DivSBL

SBL

GLasso

GBPDN

DivSBL

BSBL

SBL

GLasso

GBPDN

0.15

Reconstruction NMSE

+++|-

Reconstruction CORR

--

0.98

++ +** -

0.1

白

++++  = =

0.96

-→ +

+ ++++++ +

0.05

-

L

0.94

+ +下

----

---

0.92

0

BSBL

PC-SBL

SBL

GLasso

GBPDN

StrOMP

DivSBL

BSBL

PC-SBL

SBL

GLasso

GBPDN

StrOMP

DivSBL

Theorem 4.1. As β →∞ and K 0 < ( M +1) / 2 L , the unique global minimum ̂ γ glyph[defines] ( ̂ γ 11 , . . . , ̂ γ gL ) T yields a recovery ˆ x by (13) that is equal to x true, regardless of the estimated ̂ B i ( ∀ i ).

The proof draws inspiration from [23] and is detailed in Appendix E. Theorem 4.1 shows that, in noiseless case, achieving the global minimum of variance enables exact recovery of the true signal, provided the block sparsity of the signal adheres to the given upper bound.

4.2 Analysis of local minima

We provide two lemmas firstly. The proofs are detailed in Appendices F and G.

Lemma 4.2. For any semi-definite positive symmetric matrix Z ∈ R M × M , the constraint Z glyph[followsequal] ΦΣ 0 Φ T + β -1 I is convex with respect to Z and ( √ γ ⊗ √ γ ) .

Lemma 4.3. y T Σ -1 y y = C ⇔ P ( √ γ ⊗ √ γ ) = b for any constant C , where b glyph[defines] y -β -1 u , P glyph[defines] [ ( u T Φ ) ⊗ Φ ] diag ( vec ( ˜ B ) ) , and u is a vector satisfying y T u = C .

It's clear that P is a full row rank matrix, i.e, r ( P ) = M . Given the above lemmas, we arrive at the following result, which is proven in Appendix H.

Theorem 4.4. Every local minimum of the cost function (16) with respect to γ satisfies || ˆ γ || 0 ≤ √ M , irrespective of the estimated ̂ B i ( ∀ i ) and β .

Theorem 4.4 establishes an upper bound on the sparsity level of any local minimum for the cost function in terms of the parameter γ . Therefore, together with Theorem 4.1, these results ensure the sparsity of the final solution obtained.

5 Experiments

In this section, we compare DivSBL with the following six algorithms: 4 1. Block-based algorithms: (1) BSBL, (2) Group Lasso, (3) Group BPDN. 2. Pattern-based algorithms: (4) PC-SBL, (5) StructOMP. 3. Sparse learning (without structural information): (6) SBL. Results are averaged over 100 or 500 random runs (based on computational scale), with SNR ranging from 15-25 dB

StrOMP

StrOMP

except the test for varied noise levels. 'Normalized Mean Squared Error (NMSE)', defined as || ˆ x -x true || 2 2 / || x true || 2 2 , and 'Correlation (Corr)' (cosine similarity) are used to compare algorithms. 5

5.1 Synthetic signal data

Weinitially test on synthetic signal data, including homoscedastic (provided by [24]) and heteroscedastic data, where block size, location, non-zero quantity, and signal variance are randomly generated, mimicking real-world data patterns. The reconstruction results are provide in Appendix I.1. Table 1 shows that DivSBL achieves the lowest NMSE and the highest Correlation on both scenarios. To more intuitively demonstrate the statistically significant improvements of the conclusion, we provide box plots of the experimental results on both homoscedastic and heteroscedastic data in Figure 3 .

Unlike many frequentist approaches that require more complex debiasing methods to construct confidence intervals, the Bayesian approach offers a straightforward way to obtain credible intervals for point estimates. For more results related to Bayesian methods, please refer to Appendix I.2.

5.2 The robustness of pre-defined block sizes

As mentioned in Section 1, block-based algorithms require presetting block sizes, and their performance is sensitive to these parameters, posing challenges in practice. This experiment assesses the robustness of block-based algorithms with predefined block sizes. The test setup is shown in Figure 4. We vary preset block sizes and conduct 100 experiments for all algorithms. Confidence intervals in Figure 4 depict reconstruction error for statistical significance.

0.6

Environment:Three blocks with sizes of 100,40,and 30, respectively. The block sizes differ significantly, with some being relatively large, and their positions are random.

0.5

0.4

DivSBL

BSBL

.GroupLasso

NMSE

.GroupBPDN

0.3

DemonstrationRationale

(1)DivSBL:The optimalblock sizefor DivSBLis in the range of 20to50,which alignsmorecloselywiththeactualblock size of the signalcompared tootherblock-based algorithms.

0.2

0.1

(2)Otherblock-based algorithms:They tend tohave smaller (morebiased)optimal blocksizes.Evenat their optimalblock sizes,theresultlagsbehindDivSBL.Theirimprovement over element-wiserecoveryis alsolimited.

0

0

50

100

150

200

250

300

350

400

450

500

BlockSize

Resolves the longstanding sensitivity issue of block-based algorithms. DivSBL demonstrates strong robustness to the preset block sizes, effectively addressing the sensitivity issue that block-based algorithms commonly encounter with respect to block sizes.

Figure 5 visualizes the posterior variance learning on the signal to demonstrate DivSBL's ability to adaptively identify the true blocks. The algorithms are tested with preset block sizes of 20 (small), 50 (medium), and 125 (large), respectively, to show how each algorithm learns the blocks when block structure is misspecified. As expected in Section 2.1 and Figure 2 , DivSBL is able to adaptively find the true block through diversification learning and remains robust to the preset block size.

Exhibits enhanced recovery capability in challenging scenarios. The optimal block size for DivSBL is around 20-50, which is more consistent with the true block sizes. This indicates that when true block sizes are large and varied, DivSBL can effectively capture richer information within each block by setting larger block sizes, thereby significantly improving the recovery performance. In contrast, other algorithms do not perform as well as DivSBL, even at their optimal block sizes.

(a)

L

= 20

(b)

L

= 50

(c)

L

= 125

(d) PC-SBL & SBL

Original Signal (x)

Original Signal (x)

200

30

400

500

DivSBL(Variance)

200

30

400

500

L=50

10

200

30

400

500

200

300

400

500

BSBL(Variance)

BSBL(Variance )

L=50

100

120

200

300

340

400

500

100

200

250

300

350

400

500

Original Signal (x)

Original Signal (x)

DivSBL(Variance)

200

30

400

00S

PC-SBL(Variance)

200

3d

400

500

L=125

0.5

0

20.0

BSBL(Variance)

300

400

500

201

SBL(Variance)

400

00

L=125

.5

100

002

300

375400

500

100

200

300

5.3 1D audioSet

As shown in Figure 14, audio signals exhibit block sparse structures in discrete cosine transform (DCT) basis, which is well-suited for assessing block sparse algorithms. In this subsection, we carry out experiments on real-world audios, which are randomly chosen in AudioSet [34]. The reconstruction results are present in Appendix J. In the main text, we focus on analyzing DivSBL's sensitivity to sample rate, evaluating its performance across different noise levels, and investigating its phase transition properties.

The sensitivity of sample rate The algorithms are tested on audio sets to investigate the sensitivity of sample rate ( M/N ) varied from 0.25 to 0.55. The result is visualized in Figure 16. Notably, DivSBL emerges as the top performer across diverse sampling rates, showing a consistent 1 dB enhancement in NMSE compared to the best-performing algorithm among others.

The performance under various noise levels We assess each algorithm as Signal-to-Noise Ratio (SNR) varied from 10 to 50 and include the standard deviation from 100 random experiments on audio sets. Here, we present the performance under the minimum sampling rate 0.25 tested before, which represents a challenging recovery scenario. As shown in Table 2, the performance of all algorithms improves with higher SNR. Notably, DivSBL consistently leads across all SNR levels.

Phase Transition This audio data contains approximately 90 non-zero elements in DCT domain ( K = 90 ), which constitutes about 20% of the total dimensionality ( N = 480 ). Therefore, we start the test with a sampling rate of a same 20%. In this scenario, M/K is roughly 1 and increases with the sampling rate. Concurrently, the signal-to-noise ratio (SNR) varies gradually from 10 to 50.

The phase transition diagram in Figure 6 shows that DivSBL performs well at more extreme sampling rates and is better suited for lower SNR conditions.

AudioinDCTdomain

DivSBL

BSBL

PC-SBL

SBL

GroupBPDN

StructOMP

Nonzero-elements K=90

353025

40

40

35

40

40

40

35

40

35

40

0.7

32

30

30

30

SNR

NS

HNS

HNS

HNS

NS

25

HNS

25

0.4

20

20

20

20

20

20

20

15

15

15

15

15

15

15

0

100

200

300

400

50

10

02

M/N

67

10

M/N

10

M/N

10

M/N

10

M/N

10

M/N

10

M/N

5.4 2D image reconstruction

In 2D image experiments, we utilize a standard set of grayscale images compiled from two sources 6 . As depicted in Figure 17, the images exhibit block sparsity in discrete wavelet domain. In Section 5.3, we've shown DivSBL's leading performance across diverse sampling rates. Here, we use a 0.5 sample rate and the reconstruction errors are in Table 3 and Appendix K. DivSBL's reconstructions surpass others, with an average improvement of 9.8% on various images.

(a) Original Signal

(b) DivSBL

(c) BSBL

(d) PC-SBL

(e) SBL

(g) Group BPDN

(h) StructOMP

In Figure 7, we display the final reconstructions of Parrot, Monarch and House images as examples. DivSBL is capable of preserving the finer features of the parrot, such as cheek, eye, etc., and recovering the background smoothly with minimal error stripes. As for Monarch and House images, nearly every reconstruction introduces undesirable artifacts and stripes, while images restored by DivSBL show the least amount of noise patterns, demonstrating the most effective restoration.

6 Conclusions

This paper established a new Bayesian learning model by introducing diversified block sparse prior, to effectively capture the prevalent block sparsity observed in real-world data. The novel Bayesian model effectively solved the sensitivity issue in existing block sparse learning methods, allowing for adaptive block estimation and reducing the risk of overfitting. The proposed algorithm DivSBL, based on this model, enjoyed solid theoretical guarantees on both convergence and sparsity theory. Experimental results demonstrated its state-of-the-art performance on multimodal data. Future works include exploration on more effective weak constraints for correlation matrices, and applications on supervised learning tasks such as regression and classification.

Acknowledgments and Disclosure of Funding

This research was supported by National Key R&D Program of China under grant 2021YFA1003300.

The authors would like to thank all the reviewers for their helpful comments. Their suggestions have helped us enhance our experiments to present a more comprehensive demonstration of the effectiveness of DivSBL.

References

David L Donoho. Compressed sensing. IEEE Transactions on Information Theory , 52(4):1289-1306, 2006.

Irina F Gorodnitsky and Bhaskar D Rao. Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions on Signal Processing , 45(3):600-616, 1997.

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE Transactions on Information Theory , 51(12):4203-4215, 2005.

Robert Tibshirani. Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society Series B: Statistical Methodology , 58(1):267-288, 1996.

Michael E Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research , 1(Jun):211-244, 2001.

Scott Shaobing Chen, David L Donoho, and Michael A Saunders. Atomic decomposition by basis pursuit. SIAM review , 43(1):129-159, 2001.

Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krisnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers , 40-44. IEEE, 1993.

Pei Peng, Shirin Jalali, and Xin Yuan. Auto-encoders for compressed sensing. In NeurIPS 2019 Workshop on Solving Inverse Problems with Deep Networks , 2019.

Ajil Jalal, Liu Liu, Alexandros G Dimakis, and Constantine Caramanis. Robust compressed sensing using generative models. Advances in Neural Information Processing Systems , 33:713-727, 2020.

Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G Dimakis, and Jon Tamir. Robust compressed sensing MRI with deep generative priors. Advances in Neural Information Processing Systems , 34:14938-14954, 2021.

Yonina C Eldar, Patrick Kuppinger, and Helmut Bolcskei. Block-sparse signals: Uncertainty relations and efficient recovery. IEEE Transactions on Signal Processing , 58(6):3042-3054, 2010.

David L Donoho, Iain Johnstone, and Andrea Montanari. Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising. IEEE Transactions on Information Theory , 59(6):3396-3433, 2013.

Charlie Nash, Jacob Menick, Sander Dieleman, and Peter Battaglia. Generating images with sparse representations. In International Conference on Machine Learning , 7958-7968. PMLR, 2021.

Peng Jiang, Lihan Hu, and Shihui Song. Exposing and exploiting fine-grained block structures for fast and accurate sparse training. Advances in Neural Information Processing Systems , 35:38345-38357, 2022.

Darshan Thaker, Paris Giampouras, and René Vidal. Reverse engineering l p attacks: A block-sparse optimization approach with recovery guarantees. In International Conference on Machine Learning , 21253-21271. PMLR, 2022.

Yuchen Fan, Jiahui Yu, Yiqun Mei, Yulun Zhang, Yun Fu, Ding Liu, and Thomas S Huang. Neural sparse representation for image restoration. Advances in Neural Information Processing Systems , 33:15394-15404, 2020.

Mehdi Korki, Jingxin Zhang, Cishen Zhang, and Hadi Zayyani. Block-sparse impulsive noise reduction in OFDMsystems-A novel iterative Bayesian approach. IEEE Transactions on Communications , 64(1):271284, 2015.

Aditya Sant, Markus Leinonen, and Bhaskar D Rao. Block-sparse signal recovery via general total variation regularized sparse Bayesian learning. IEEE Transactions on Signal Processing , 70:1056-1071, 2022.

Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology , 68(1):49-67, 2006.

Sahand Negahban and Martin J Wainwright. Joint support recovery under high-dimensional scaling: Benefits and perils of l 1 , ∞ -regularization. Advances in Neural Information Processing Systems , 21:11611168, 2008.

Yasutoshi Ida, Yasuhiro Fujiwara, and Hisashi Kashima. Fast sparse group LASSO. Advances in Neural Information Processing Systems , 32, 2019.

Ewout Van den Berg and Michael P Friedlander. Sparse optimization with least-squares constraints. SIAM Journal on Optimization , 21(4):1201-1229, 2011.

Zhilin Zhang and Bhaskar D Rao. Sparse signal recovery with temporally correlated source vectors using sparse Bayesian learning. IEEE Journal of Selected Topics in Signal Processing , 5(5):912-926, 2011.

Zhilin Zhang and Bhaskar D Rao. Extension of SBL algorithms for the recovery of block sparse signals with intra-block correlation. IEEE Transactions on Signal Processing , 61(8):2009-2015, 2013.

Zhilin Zhang and Bhaskar D Rao. Recovery of block sparse signals using the framework of block sparse Bayesian learning. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 3345-3348. IEEE, 2012.

Junzhou Huang, Tong Zhang, and Dimitris Metaxas. Learning with structured sparsity. In Proceedings of the 26th Annual International Conference on Machine Learning , 417-424, 2009.

Jun Fang, Yanning Shen, Hongbin Li, and Pu Wang. Pattern-coupled sparse Bayesian learning for recovery of block-sparse signals. IEEE Transactions on Signal Processing , 63(2):360-372, 2014.

Lu Wang, Lifan Zhao, Susanto Rahardja, and Guoan Bi. Alternative to extended block sparse Bayesian learning and its relation to pattern-coupled sparse Bayesian learning. IEEE Transactions on Signal Processing , 66(10):2759-2771, 2018.

Jisheng Dai, An Liu, and Hing Cheung So. Non-uniform burst-sparsity learning for massive MIMO channel estimation. IEEE Transactions on Signal Processing , 67(4):1075-1087, 2018.

David JC MacKay. Bayesian interpolation. Neural Computation , 4(3):415-447, 1992.

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society: series B (methodological) , 39(1):1-22, 1977.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge University Press, 2004.

Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in Neural Information Processing Systems , 31, 2018.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , 776-780. IEEE, 2017.

Timothy CY Chan, Rafid Mahmood, and Ian Yihang Zhu. Inverse optimization: Theory and applications. Operations Research , 2023. https://doi.org/10.1287/opre.2022.0382.

Xinyang Yi and Constantine Caramanis. Regularized EM algorithms: A unified framework and statistical guarantees. Advances in Neural Information Processing Systems , 28, 2015.

David P Wipf, Julia P Owen, Hagai T Attias, Kensuke Sekihara, and Srikantan S Nagarajan. Robust Bayesian estimation of the location, orientation, and time course of multiple correlated neural sources using MEG. NeuroImage , 49(1):641-655, 2010.

Shane F Cotter, Bhaskar D Rao, Kjersti Engan, and Kenneth Kreutz-Delgado. Sparse solutions to linear inverse problems with multiple measurement vectors. IEEE Transactions on Signal Processing , 53(7):2477-2488, 2005.

Ralph Tyrell Rockafellar. Convex Analysis . Princeton University Press, 1970.

David G Luenberger and Yinyu Ye. Linear and nonlinear programming , volume 116. Springer, 2008.

Carlos M Carvalho, Nicholas G Polson, and James G Scott. Handling sparsity via the Horseshoe. In Artificial Intelligence and Statistics , 73-80. PMLR, 2009.

Veronika Roˇ cková and Edward I George. The spike-and-slab LASSO. Journal of the American Statistical Association , 113(521):431-444, 2018.

Daniela Calvetti, Erkki Somersalo, and A Strang. Hierachical Bayesian models and sparsity: l 2 -magic. Inverse Problems , 35(3):035003, 2019.

A Experimental result of diversifying correlation matrices

0.6

0.6

BSBL

...BSBL

Diff-BSBL

Diff-BSBL

0.5

DivSBL(withoutdiversified correlation)

0.5

DivSBL(withoutdiversified correlation)

DivSBL

DivSBL

0.4

0.4

NMSE

NMSE

0.3

0.3

0.2

0.2

0.1

0.1

0

0

100

200

300

400

500

0

0

10

20

30

40

50

60

Iteration

Iteration

(a) A complete iteration record

(b) The first 60 iterations

Below, we will explain the necessity of the three steps for updating B in DivSBL:

Firstly, the purpose of the first step, estimating B under a strong constraint, is to avoid overfitting. As shown by the green line (Diff-BSBL) in Figure 8 , algorithm without strong constraints tend to worsen NMSE after several iterations due to overfitting.

Since the strong constraint in (1) forces all blocks to have the same correlation, it leads to the loss of specificity in correlation matrices within different blocks. Therefore, the motivation for the second step, applying weak constraints, is to make the correlations within blocks similar to some extent while preserving their individual specificities. As the black line in Figure 8 (DivSBL-without diversified correlation) shows, DivSBL without weak constraints fails to capture the specificity within blocks, resulting in a loss of accuracy and slower speed compare to DivSBL.

The third step, Toeplitzization, inherits the advantages of BSBL, allowing the correlation matrices to have a reasonable structure.

Overall, while BSBL utilizes a single variance and a shared correlation matrix within each block, DivSBL incorporates diversification into both variance and correlation matrix modeling, making it more flexible and enhancing its performance.

B Derivation of hyperparameters updating formulas

In this paper, lowercase and uppercase bold symbols are employed to represent vectors and matrices, respectively. det( A ) denotes the determinant of matrix A . tr( A ) means the trace of A . Matrix diag ( A 1 , . . . , A g ) represents the block diagonal matrix with the matrices { A i } g i =1 placed along the main diagonal, and Diag( A ) represents the extraction of the diagonal elements from matrix A to create a vector. We observe that

in which Σ 0 can be reformulated as

where

and the Kronecker delta function here, denoted by δ ij , is defined as

Hence, Σ 0 is split into two components in (27), with the second term of the summation only depending on G i and B i , and the first part D -i being entirely unrelated to them. Then, we can update each B i and G i independently, allowing us to learn diverse B i and G i for different blocks.

The gradient of (26) with respect to √ γ ij can be expressed as

The first term results in

and the second term yields

where µ i ∈ R L × 1 represents the i -th block in µ , and Σ i ∈ R L × L denotes the i -th block in Σ 7 . Using A 1 ( M + N ) A 2 = A 1 MA 2 + A 1 NA 2 , the formula above can be further transformed into

in which, T ij and A ij are independent of √ γ ij , and their expressions are

Thus, the derivative of Q (Θ) with respect to √ γ ij reads as

It is important to note that the variance should be non-negative. So by setting (28) equal to zero, we obtain the update formulation of γ ij as

As for the gradient of (26) with respect to B i , we have

Setting it equal to zero, the learning rule for B i is given by

C The procedure for solving the constrained optimization problem

To clarify the dual problem of (P), we firstly express (P)'s Lagrange function as

Since the constraints in (P) are equalities, we do not impose any requirements on the multipliers { λ i } g i =1 . The primal and dual problems in terms of L are given by

respectively. Although the objective here is non-convex, dual ascent method takes advantage of the fact that the dual problem is always convex [32], and we can get a lower bound of (P) by solving (D). Specifically, dual ascent method employs gradient ascent on the dual variables. As long as the step sizes are chosen properly, the algorithm would converge to a local maximum. We will demonstrate how to choose the step sizes in the following paragraph.

According to this framework, we first solve the inner minimization problem of the dual problem (D). Keeping the multipliers { λ i } g i =1 fixed, the inner problem is

where the superscript k implies the k -th iteration. According to the first-order optimality condition, the primal solution for (32) is as follows:

Subsequently, the outer maximization problem for the multiplier λ i (dual variable) can be addressed using the gradient ascent method. The update formulation is obtained by

in which, α k i represents the step size in the k -th iteration for updating the multiplier λ i ( i = 1 ...g ). Convergence is only guaranteed if the step size satisfies ∑ ∞ k =1 α k i = ∞ and ∑ ∞ k =1 ( α k i ) 2 < ∞ [32]. Therefore, we choose a diminishing step size 1 /k to ensure the convergence. The procedure, using dual ascent method to diversify B i , is summarized in Algorithm 2 as follows:

Algorithm 2 Diversifying B i

1: Input: Initialized λ 0 i ∈ R , ε > 0 , the common intra-block correlation B obtained from (21).

2: Output: B i , i = 1 , · · · , g .

3: Set iteration count k = 1 .

4: repeat

5: Set step size α k i = 1 /k .

6: Update B k +1 i using (33).

7: Update λ k +1 i using (34).

8: k := k +1 ;

9: until | log det B k i -log det B | ≤ ε .

D The computing time and the explanation of one-step dual ascent

Computing time As our algorithm is based on Bayesian learning and involves covariance structures estimation, it is slower compared to heuristic algorithm StructOMP and solvers like CVX and SPGL1

(group Lasso, group BPDN). Here, we provide curves of NMSE versus CPU time for DivSBL, DivSBL (with complete dual ascent), and BSBL to better visualize the speed of the algorithms. In Figure 9, DivSBL shows larger NMSE reductions at each time step compared to both BSBL and fully iterative DivSBL.

One-step dual ascent It has been observed that imposing strong constraints B i = B leads to slow convergence, while not applying any constraints results in overfitting [24].

DivSBL

DivSBL(withcompletedualascent)

0.8

BSBL

E

0.6

NMSE

0.4

0.2

0

0

0.15

0.3

0.45

0.6

0.75

time(s)

We consider providing further theoretical and intuitive explanations. From another perspective, the tuple ( B k +1 i , λ k i ) satisfying the update formula (23)(24) in the text is, in fact, a KKT pair of a certain constrained optimization problem, which is summarized in Proposition 3.1.

Proof of Proposition 3.1

Proof. Construct a weak constraint function ψ : R n 2 → R

Such a function ψ always exists. In fact, we can always construct a polynomial function ψ that satisfies the given conditions (Hermitte interpolation polynomial). The finite conditions of function values and derivatives correspond to a system of linear equations for the coefficients of the polynomial function. Since the degree of the polynomial function is arbitrary, we can always ensure that the system of linear equations has a solution by increasing the degree.

From the proposition above, it can be observed that the tuple ( B k +1 i , λ k i ) satisfying equation (23)(24) is, in fact, a KKT point of a certain weakly constrained optimization problem. Therefore, although we do not precisely solve the constrained optimization problem under the weak constraint ζ ( · ) (i.e., log det( · ) in the main text), we accurately solve the constrained optimization problem under the weak constraint ψ ( · ) . Even though this weak constraint ψ ( · ) is unknown, it certainly exists and may be better than the original weak constraint ζ ( · ) . This perspective aligns with the concept of inverse optimization [35]. Interestingly, the technique here provides an application case for inverse optimization. From this viewpoint, we are still optimizing Q function with the weak constraint ψ ( · ) , and the subproblems under the weak constraint ψ ( · ) are accurately solved. Therefore, similar to the constrained EM algorithm, it can still generate a sequence of solutions, which explains the experimental results mentioned above.

Meanwhile, the above method can also be viewed as a regularized EM algorithm [36]. The update formula (23) corresponds to the exact solution of the regularized Q function Q ( { B i } g i =1 , { G i } g i =1 ) -

Therefore, by establishing weak constraints and employing dual ascent to solve the constrained optimization problem (P), we achieve diversification on intra-block correlation matrices, which are more aligned with real-life modeling. Since the convergence speed of dual ascent is fastest in the initial few steps and sub-problems are unnecessary to be solved accurately, our initial motivation was to consider allowing it to iterate only once, which yielded promising experimental results, as shown in the Figure 9.

∑ i λ k i ( ζ ( B i ) -ζ ( B )) , while equation (24) utilizes gradient descent to update the regularization parameters.

E Proof of Theorem 4.1

Proof. The posterior estimation of the block sparse signal is given by ˆ x = β ˆ ΣΦ T y = ( β -1 ˆ Σ -1 0 + Φ T Φ ) -1 Φ T y , where ˆ Σ 0 = diag { ˆ G 1 ˆ B 1 ˆ G 1 , ˆ G 2 ˆ B 2 ˆ G 2 , · · · , ˆ G g ˆ B g ˆ G g } , and ˆ G i = diag { √ ˆ γ i 1 , · · · , √ ˆ γ iL } .

Let ˆ γ = (ˆ γ 11 , · · · , ˆ γ 1 L , · · · , ˆ γ g 1 , · · · , ˆ γ gL ) T , which is obtained by globally minimizing (35) for a given ˆ B i ( ∀ i ).

Inspired by [37], we can rewrite the first summation term as

Then (35) is equivalent to

So when β →∞ , (35) is equivalent to minimizing the following problem,

Let g ( x ) = min γ ( x T Σ -1 0 x +logdet Σ y ) , then according to Lemma 1 in [37], g ( x ) satisfies

where K represents the estimated number of blocks and β -1 → 0 (noiseless). Therefore, when g ( x ) achieves its minimum value by (36), K will achieve its minimum value simultaneously.

The results in [38] demonstrate that if K 0 < M +1 2 L , then no other solution exists such that y = Φx with K < M +1 2 L . Therefore, we have K ≥ K 0 , and when K reaches its minimum value K 0 , the estimated signal ˆ x = x true .

F Proof of Lemma 4.2

Proof. We can equivalently transform the constraint Z glyph[followsequal] ΦΣ 0 Φ T + β -1 I into

The LHS ω T Z ω is linear with respect to Z . And for the RHS, ω T ΦΣ 0 Φ T ω + β -1 ω T ω = q T Σ 0 q + β -1 ω T ω , where q = Φ T ω , and Σ 0 can bu reformulated as

Therefore, RHS = q 2 1 ˜ B 11 γ 11 + . . . + q 1 q N ˜ B 1 N √ γ gL √ γ 11 + . . . + q N q 1 ˜ B N 1 √ γ 11 √ γ gL + q 2 N ˜ B NN γ gL + β -1 ω T ω = vec ( qq T glyph[circledot] ˜ B ) T ( √ γ ⊗ √ γ ) + β -1 ω T ω which is linear with respect to √ γ ⊗ √ γ . In conclusion, Z glyph[followsequal] ΦΣ 0 Φ T + β -1 I is convex with respect to Z and √ γ ⊗ √ γ ,

G Proof of Lemma 4.3

Proof. " = ⇒ " Given y T Σ -1 y y = C and u satisfying y T u = C , without loss of generality, we choose u glyph[defines] Σ -1 y y , i.e., y = Σ y u . Then b can be rewritten as

Applying the vectorization operation to both sides of the equation, (37) results in

" ⇐ = " Vice versa.

H Proof of Theorem 4.4

Proof. Based on Lemma 4.3, we consider the following optimization problem:

where Σ y = β -1 I + ΦΣ 0 Φ T , P and b are already defined in Lemma 4.3. In order to analyze the property of the minimization problem (38), we introduce a symmetric matrix Z ∈ R M × M here. Therefore, the problem with respect to Z and γ becomes

It is evident that problem (38) and problem (a) are equivalent. Denote the solution of (a) as ( Z ∗ , γ ∗ ) , so γ ∗ here is also the solution of (38). Thus, we will analysis the minimization problem (a) instead in the following paragraph.

We first demonstrate the concavity of (a). Obviously, with respect to Z and ( √ γ ⊗ √ γ ) , the objective function log det Z is concave, and (a.3) is convex. According to Lemma 4.2, (a.2) is convex as well. Hence, we only need to show the convexity of (a.4) with respect to Z and ( √ γ ⊗ √ γ ) .

It is observed that

where h i glyph[defines] ( δ i 1 , · · · , δ i,gL ) T , H ∈ R gL × ( gL ) 2 . Based on the convexity-preserving property of linear transformation [39], the constraint γ glyph[followsequal] 0 exhibits convexity with respect to ( √ γ ⊗ √ γ ) . Therefore, (a) is concave with respect to ( √ γ ⊗ √ γ ) and Z . So we can rewrite (a) as

The minimum of (b) will achieve at an extreme point. According to the equivalence between extreme point and basic feasible solution (BFS) [40], the extreme point of (b) is a BFS to

,

which concludes || √ γ ⊗ √ γ || 0 ≤ r ( P ) = M , equivalently || γ || 0 ≤ √ M . This result implies that every local minimum (also a BFS to the convex polytope) must be attained at a sparse solution.

I The experiment of 1D signals with block sparsity

I.1 The reconstruction results

As mentioned in Section 2.1.3, homoscedasticity can be seen as a special case of our model. Therefore, we test our algorithm on homoscedastic data provided in [24]. In this dataset, each block shares the same size L = 6 , and the amplitudes within each block follow a homoscedastic normal distribution.

The reconstructed results are shown in Figure 10, Figure 3 and the first part of Table 1. DivSBL demonstrates a significant improvement compared to other algorithms.

(a) Original Signal

(b) DivSBL

(c) BSBL

(d) PC-SBL

(e) SBL

(g) Group BPDN

(h) StructOMP

0

0

0

0

-1

-1

-1

-1

0

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

1

1

0

D

0

-1

-1

-1

-1

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

Furthermore, we consider heteroscedastic signal, which better reflects the characteristics of real-world data. The recovery results are presented in Figure 11, Figure 3 and the second part of Table 1.

(a) Original Signal

(b) DivSBL

(c) BSBL

(d) PC-SBL

(e) SBL

(g) Group BPDN

(h) StructOMP

20

20

20

20

0

0

10

0

10

-20

0

20

40

60

80

100

120

140

160

-20

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

-10

0

20

40

60

80

100

120

140

160

20

20

20

20

10

10

0

0

0

0

-20

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

0

20

40

60

80

100

120

140

160

-20

0

20

40

60

80

100

120

140

160

I.2 Reconstructed by Bayesian methods with credible intervals for point estimation

Here we provide comparative experiments on heteroscedastic data with three classic Bayesian sparse regression methods: Horseshoe model [41], spike-and-slab Lasso [42] and hierarchical normal-gamma method [43] in Figure 12. Regarding the natural advantage of Bayesian methods in quantifying uncertainties for point estimates, we further include the posterior confidence intervals from Bayesian methods. As shown in Figure 13, DivSBL offers more stable and accurate posterior confidence intervals.

0.99

Reconstruction NMSE

0.1

Reconstruction CORR

0.98

0.97

0.96

0.96

DivSBL

0.05

0.94

BSBL

PC-SBL

SBL

0.93

Group BPDN

0.92

Horseshoe

StructOMP

Spike and Slab Lasso

0.91

(a) DivSBL

(b) BSBL

(c) PC-SBL

(d) SBL

(e) Horseshoe

(f) Normal-Gamma

30

30

30

20

20

20

Value

10

Value

10

Value

10

0

-10

-10

0

-20

0

20

40

60

Index

80

100

120

140

160

-20

0

20

40

60

Index

80

100

120

140

160

-10

0

20

40

60

Index

80

100

120

140

160

30

30

30

20

20

20

Value

10

10

e

10

0

0

-10

-10

-20

0

20

40

60

Index

80

100

120

140

160

-20

0

20

40

60

Index

80

100

120

140

160

-10

0

20

40

60

Index

80

100

120

140

160

J The experiment of audio signals

Audio signals display block sparse structures in the discrete cosine transform (DCT) basis. As illustrated in Figure 14, the original audio signal (a) transforms into a block sparse structure (b) after DCT transformation.

(a) Original Audio Signal

(b) Sparse Representation

0.6

0.4

0.2

2

-0.2

-0.4

-0.6

0

50

100

150

200

250

300

350

400

450

500

3

50

100

150

200

250

300

350

400

450

500

(a) Original Signal

(b) DivSBL

(c) BSBL

(d) PC-SBL

(e) SBL

(g) Group BPDN

(h) StructOMP

4

4

4

4

2

2

2

2

0

0

0

0

-2

-2

-2

-2

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

4

4

女

+

2

CN

2

CN

0

0

0

0

-2

-2

-2

-2

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

0

50

100

150

200

250

300

350

400

450

The sensitivity of sample rate As demonstrate in Section 5.3, we tested on audio sets to investigate the sensitivity of sample rate ( M/N ) varied from 0.25 to 0.55. DivSBL emerges as the top performer across diverse sampling rates, exhibiting a consistent 1 dB improvement in NMSE relative to the bestperforming algorithm, as depicted in Figure 16.

-10

DivSBL

....BSBL-EM

-12

=-PC-SBL

SBL

GroupLasso

NMSE (dB)

-14

.p..s

StructOMP

Group BPDN

-16

-18

-20

0.25

0.35

0.45

0.55

M/N

K The experiment of image reconstruction

As depicted in Figure 17, the images exhibit block sparsity in discrete wavelet domain. We've created box plots for NMSE and correlation reconstruction results for each image undergoing restoration,

We carry out experiments on realworld audio signal, which is randomly chosen in AudioSet [34]. The reconstruction results for audio signals are present in Figure 15. It is noteworthy that DivSBL exhibits an improvement of over 24.2% compared to other algorithms.

as depicted in Figures 18-25. It's evident that the DivSBL exhibits significant advantages in image reconstruction tasks.

(b) House Image's Sparse Domain

800

600

400

200

0

-200

20

40

60

80

100

120

600

500

400

300

200

100

0

-100

20

40

60

80

100

120

L The sensitivity to initialization

According to our algorithm, given the variance γ ij , the prior covariance matrix can be obtained as Σ 0 = diag ( √ γ 11 , · · · , √ γ gL ) ˜ B diag ( √ γ 11 , · · · , √ γ gL ) . In the absence of any structural infor-

mation, the initial correlation matrix ˜ B is set to the identity matrix. Consequently, the mean and covariance matrix for the first iteration can also be determined. Since other variables are derived from the variance, we only need to test the sensitivity to the initial values of variances γ .

We test the sensitivity of DivSBL to initialization on the heteroscedastic signal from Section 5.1. Initial variances are set to γ = η · ones ( gL, 1) and γ = η · rand ( gL, 1) with the scale parameter η ranging from 1 × 10 -1 to 1 × 10 4 . The result in Figure 26 shows that while initialization could affect the convergence speed to some extent, the algorithm's overall convergence is assured.

(a)

γ

=

η

·

ones

(

gL,

1)

(b)

γ

=

η

·

rand

(

gL,

1)

=0.1

=0.1

0.9

=0.5

0.9

=0.5

0.8

=1

0.8

=1

=10

=10

0.7

0.7

1

=50

1

=50

NMSE

0.6

=100

NMSE

0.6

1

=100

0.5

=1000

0.5

←=1000

+=10000

+=10000

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1

10

20

30

40

50

60

70

80

90

100

10

20

30

40

50

60

70

80

90

100

Iteration

Iteration

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims present in abstract and introduction (Section 1) have reflected the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Theorem 4.1 and Theorem 4.4, we conclude the proof under a noiseless assumption. And we conjecture that similar conclusion hold even in the noise case. In Section 5.3, we evaluate the algorithm's robustness at different signal-to-noise ratios (SNR). The experimental results consistently show leading performance of the proposed algorithm across various noise environments.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the theorems and assumptions are clearly stated in Section 4. Appendix E-H present the proof of them.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: For reproducibility, experiment details are provided in each subsection of Section 5. Additionally, the code is available in the Supplementary Material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: For reproducibility, experiment code is available in Supplementary Material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full details are provided with code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results are accompanied by confidence intervals, as shown in Section 5. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper have provided compute time in Section 5 and Appendix D. Details on compute worker are provided in code (README.md).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed. The paper aims to explore new technical methods or algorithms without delving into topics related to societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The data and code used in the paper have all been properly credited , and citations are provided in the text.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a prior named Diversified Block Sparse Prior, which can be viewed as a generalization of priors for existing sparse Bayesian learning methods. It utilizes EM algorithm and dual ascent to obtain parameter estimates and convergence of estimates to the true parameter in the $\\beta$ limit has been established.\n\nSTRENGTHS:\nThe structure of the paper is well organized for readers to follow. Generalization of the existing sparse Bayesian learning methods would provide more flexible framework in sparse regression problems.\n\nWEAKNESSES:\n1. The paper does not mention any of the existing well-known Bayesian sparse regression methods: for instance, horse-shoe prior [1], spike-slab LASSO [2], hierarchical normal-gamma hyperpriors [3], which have been very successful in sparse signal estimation problems. Furthermore, the strength of the Bayesian approach in this problem is in quantifying uncertainties associated with the point estimate. Unlike many frequentist approaches where the construction of confidence intervals requires a more sophisticated debiasing approach, the Bayesian approach provides a natural way to obtain credible intervals for the point estimate. The paper seems to largely ignore such strength of the Bayesian approach.\n\n[1] Carvalho, Carlos M., Nicholas G. Polson, and James G. Scott. \"Handling sparsity via the horseshoe.\" Artificial intelligence and statistics. PMLR, 2009. \n\n[2] Ročková, Veronika, and Edward I. George. \"The spike-and-slab lasso.\" Journal of the American Statistical Association 113.521 (2018): 431-444\n\n[3] Calvetti, Daniela, Erkki Somersalo, and A. Strang. \"Hierachical Bayesian models and sparsity: ℓ2-magic.\" Inverse Problems 35.3 (2019): 035003.\n\n2. The paper establishes theoretical results on local minima in Section 4.2, but this section doesn't seem to add much information about the algorithm. Perhaps the authors should provide more qualitative statements from the established theoretical results, which the current manuscript misses.\n\n3. The current theoretical guarantee only considers the noiseless setting. Is there anything further one could say in the noisy setting?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNot Applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper introduces block sparse bayesian learning for block sparse settings, as motivated by compressed sensing theory. The method relies on a “diversified scheme” which allows for inference that is robust to block choices by modeling intra-block covariance and inter-block correlation. The authors derive an algorithm for learning the model parameters and provide some simple but solid convergence results. They demonstrate their method’s effectiveness on a variety of applications, showing it is more robust than multiple existing and established methods to the prior specification of block information.

STRENGTHS:
The method introduced is intuitive. The authors provide a clear description of the EM algorithm and dual ascent method, backing the use of both with theoretical justification. Impressive ground truth and computational results, a technically sound paper overall.

WEAKNESSES:
Since EM is used for fitting, method may be sensitive to initialization.

QUESTIONS:
In practice, when the block structure is misspecified, how well does the proposed method learn the zero variance terms described in e.g. figure 2? How sensitive is the method to initialization?

LIMITATIONS:
The authors evaluate the limitations of their work with extensive experiments, as outlined in the checklist. However, the authors could do a better job of addressing the limitations of their work (for example, potential sensitivity to initialization) directly within their paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a hierarchical bayesian model for sparse inverse problems where sparsity is structured in blocks. The authors propose a diversified block sparse prior using a structured covariance taking into account both intra block and block-to-block correlations. They propose an EM algorithm to solve the problem, showcase some thereotical properties in noiseless settings and illustrate the performance of their model with synthetic and real datasets.\n\nSTRENGTHS:\n- Self contained paper\n- Novel and simple model with both theoretical and applied contributions\n- The illustration used to provide the intuition are not easy to understand\n- The theoretical findings are not well motivated / explained and seem out of place / space fillers.\n- The presentation of the paper could be significantly improved (figures hard to read, tiny fonts)\n\nWEAKNESSES:\n- The illustration used to provide the intuition are not easy to understand\n- The theoretical findings are not well motivated / explained and seem out of place / space fillers.\n- The presentation of the paper could be significantly improved (figures hard to read, tiny fonts)\n\nQUESTIONS:\n1. Paragraph 2.1.1 should be explained better. I fail to see how the proposed structured covariance would lead to the diversified sparsity (intra block/extra block) regardless of the predefined blocks. But if the method is robust to the chosen blocks, what is their effect then ? Why not let the method discover the blocks by itself ?\n\n2. I fail to see how the proposed method can recover accurate sparsity structures regardless of the predefined blocks. I do not find fig2 helpful\nin this regard. In both conditions (white/pink) the recovered gamma_i are close to 0 ? \n\n3. I believe in  L95 there should be gL*(L+1)/2 constraints ?\n\n4. Is there an intuition behind selecting the constraint function psi ? \n\n5. Where does the toeplitz correction come from ? What is the impact of this step on the convergence of the EM algoritm ? Is this step necessary ?\n\n6. Since the main contribution of the paper is the structured sparsity in groups, in the experiments I would expect the group lasso to perform very well when the groups are known and fixed. Which is not the case here. Is that due to a poor hyperparameter setting ?\n\n- typo: L137 a diversified solution\n\n- typo: L176 the diversified block sparse prior, the following global\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a novel prior called Diversified Block Sparse Prior  towards a new framework to address the problem of recovery of block sparse signals. They provide theoretical and experimental justification as a proof of the efficacy of their work.\n\nSTRENGTHS:\nThe authors propose a novel diversified block sparse prior which allows for a diversified variance and a diversified correlation. Such a prior can be used to encode/learn the knowledge of DAGs. An EM based solution is derived.\n\nWEAKNESSES:\nThis area of research is quite old, and hence, any result that comes about tends to have flavors of several previous works. While the results here are relevant for a journal, I do not find anything exciting in the work which is worth publishing at Neurips. Overall the impact of the work is poor. However, in a stand-alone manner, these are some of the weaknesses:\n1. The length of the block is assumed constant across the blocks. Although the authors claim that the corresponding entries will be zero or non-zero, the model will be forced towards solutions that have a constant non-zero block size. \n2. Lack of sample complexity results: when we introduce new variables into any existing set-up (as compared to B-SBL, the variance here is G_iB_iG_i, which is twice as many parameters) we expect the number of measurements to be larger. Noise also impacts sample complexity. Hence, it is essential to analyse their joint effect on the sample complexity.\n3. Local and global minima results hold in no-noise scenarios: The noisy scenario is presented only as a conjecture. \n4. Although EM iterations are used, a closed form solution for B_i is not available. Authors propose the ascent approach for estimating B_i. This impacts the estimation process, but it has not been clarified. \n5. It is not clear from the experimental settings as to what leads to better results of the proposed algorithm. The algorithm displays better performance in spite of having to estimate additional parameters in G and B. This is possible only if the number of measurements increases. More experimental results based on the ratio, m/p where m denotes the number of observations and p denotes the sparsisity is required.\n\nQUESTIONS:\n1. What is the additional burden due to the introduction of diversified prior. Plots of m/p to substantiate the above. \n2. How should the proofs change to include noise?\n3. How does your method compare with PC-SBL, to be added into 2.1.3.\n4. What is the error in estimating B_i in the synthetic case?\n\nLIMITATIONS:\nThis method is effective only with signals that have constant block-size. No sample complexity results are available.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}