Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors generalize the univariate Bayesian horseshoe shrinkage prior to the multivariate setting and propose a new prior that detects sparsity and encourages smoothness within clusters of variables. The paper illustrates the application of the prior in Bayesian regression and present an MCMC algorithm. Numerical experiments with synthetic data and results with one real dataset illustrate the potential advantages over alternative methods.
main_review: The paper is well-written. The proposed prior is novel and the authors develop an efficient reversible jump MCMC sampler. The discussion on clustering effects provides valuable insights into the nature of the proposed prior and the authors provide a posterior contraction result.
The weaker parts of the paper are the discussion of related work and the simulation experiments. 

(1) The proposed method is compared against only two baselines, with one dating from 2005. Although fused lasso should definitely be included as a classical baseline, I think there are many alternatives for problems of this kind. Are the authors suggesting that there has been little interest in the analyzed problem and that there has only been one suitable competitive method proposed in the past decade? I would think that network lasso [R1] and its derivatives, the graph-based regularization of [R2], OWL and OSCAR regularization, and Bayesian graph Laplacian regularization [R3] would be candidate alternative methods. 

(2) The synthetic experiments only explore a scenario of two clusters. The problem would seem to be much more complex when there are multiple clusters in the data. The algorithm appears to perform well for the real data-set (where it appears to identify more clusters). Results for synthetic experiments with more clusters would provide insight into the behaviour – it’s not clear if many more iterations are required for a satisfactory exploration.

[R1] Hallac, David, Jure Leskovec, and Stephen Boyd. "Network lasso: Clustering and optimization in large graphs." Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015.

[R2] Li, Yuan, et al. "Graph-Based Regularization for Regression Problems with Alignment and Highly Correlated Designs." SIAM J. Mathematics of Data Science 2.2 (2020): 480-504.

[R3] Liu, Fei, et al. "Bayesian regularization via graph Laplacian." Bayesian Analysis 9.2 (2014): 449-474.

** AFTER AUTHOR RESPONSE

My main concern with the paper was the inadequate comparison to other methods for addressing the same problem. In their response, the authors have indicated that they will include comparisons to several more recent baseline methods, and they have included results for preliminary experiments, which indicate that the proposed method provides a meaningful improvement. If the paper is revised to include more extensive experimental comparisons (which constitutes a relatively minor change), then I would consider it a valuable technical contribution. I have modified my recommendation to "good paper, accept"

limitations_and_societal_impact: None
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose a new prior referred to as Tree-based Low-rank Horseshoe (T-Loho), which can incorporate relationship among variables as a graph.
Theoretical characteristics of the clustering effects and posterior concentrations are investigated.
In computational experiments based on both synthetic and real-world data, T-Loho turned out to be superior to fused lasso and soft-thresholded Gaussian process in terms of both regression performance and support recovery.
main_review: The paper is presented with in-depth technical details, and therefore not easy to read from time to time.
limitations_and_societal_impact: In computational experiments, comparison to Bayesian fused lasso is missing. 
More experiments demonstrating the power of tree-based graph partition prior are expected.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a multi-dimensional horseshoe prior for Bayesian inference that enforces clustering of the (non)zero parameters. The main contribution of the paper is to make this approach more flexible, feasible and computationally efficient.
main_review: strengths:
Imposing structured sparsity can have many practical applications and making this computationally feasible may have a great impact on the community. The inclusion of the scheme in a Bayesian inference setup has many advantages from the theoretical perspective. The paper is technically sound and the proposed method comes with posterior consistency guarantees.

weaknesses:
The possibly unconvincing side of the paper is the value of its net contribution compared with the state-of-the-art. It is not clear if similar graph-structured priors have never been used in Bayesian learning and if their use to enforce parameter structured smoothness is new. A technical novelty of the paper is to go beyond the piecewise constant assumption of existing shrinkage approaches. The authors could have explained more explicitly what is the technical challenge of this extension. From what the authors write in Section 2, it looks like the key contribution relies on the inclusion of a new covariance matrix in the horseshoe prior (as the partition approximation scheme seems to be taken from previous work). The authors should explain better how this leads to the claimed computational advantages.

questions:
- One of the main motivations of this work is the fact that "fixed chain or tree orders may not be compatible with the true order with respect to its context G". How is the order-within-a-context defined?
- Are there examples of feasible or quasi-feasible regularization schemes that take into account the whole graph structure? 
- How are the theoretical results listed at the end of Section 1, e.g. posterior consistency, different from standard guarantees that can be obtained with other Bayesian methods? 
- What is a practical situation where a multi-dimensional parameter needs to show some sort of graph-neighbour continuity?
- The proposed covariance matrix is claimed to capture at the same time fusion and sparsity. Is this obtained by including a tunable diagonal factor?
- The goal of the new graph-clustering prior is to allow non-connected components in the graph. How does this come from the proposed form of the covariance matrix? And why is this important from an applications point of view?
limitations_and_societal_impact: In the paper, I could not find any comment about the potential negative societal impact of the results. On the other hand, I can hardly think about social issues related to this work, except for possible unfair effects on the obtained models coming from a naive graph-based selection process of the model parameters.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new prior for high dimensional parameters with graphical relations. It can be regarded as an extension of univariate horseshoe shrinkage prior to multivariate setting by borrowing ideas from studies on homogenous sparse priors and tree-based graph partition prior. The paper describes posterior sampling strategy with efficient implementation details and theoretical guarantees. In the end, the paper demonstrates its methods via both simulated and real data experiments.
main_review: The paper extends the widely used univariate horseshoe prior to multivariate case. The task itself is not new and some of the ideas have their roots in previous literatures, however, the paper combines them in a non-trivial way. By introducing the low-rank structure for prior distribution, the paper naturally incorporates cluster information into prior. Then the paper uses the random minimum spanning tree approach introduced by [1] to model the partition prior. The remaining computation and theoretical proof techniques are borrowed from previous literatures. But the way they are combined are novel.

The paper is technically sound. The methods are clearly stated, and details of efficient implementation are included. The theoretical results are rigorously proved. In addition, the author uses simulation studies to demonstrate the benefits of their proposed methods compared with other widely used approaches. The simulation results are adequately interpreted. Besides, the author(s) also demonstrate(s) the method via real data application.

The paper is clearly written and well-written. Many informative figures are much appreciated. The details of method and experiments are described clearly. With that being said, the section 3.2 is a bit hard to follow. I think it will be better if the author(s) can give more high-level intuitive explanation.

Given the task of the paper is widely attractive, the Bayesian approach offered by the paper will be interesting to many researchers and practitioners. The method given by the paper is new and the benefit of it is shown by well-designed experiments. With that being said, most of results and techniques seem to be borrowed from somewhere else, and thus may not be surprising.

[1] Zhao Tang Luo, Huiyan Sang, and Bani Mallick. A Bayesian contiguous partitioning method for learning clustered latent variables. Journal ofMachine Learning Research, 22(37):1–52, 2021.

limitations_and_societal_impact: Yes, the authors adequately addressed the potential negative societal impact of their work.
review_title: null

Meta-review: The reviewers all agree that the paper makes novel and valuable contributions. One the main concerns of the reviewers was that the paper was missing a sufficiently compelling discussion of how the method proposed in the paper compares with the state of the art, and an experimental comparison with more alternative methods. The responses to the different reviewers addresses well these concerns. The authors are strongly encouraged to include the new experimental results and the elements of discussion provided in the final version of the paper. 
Decision: Accept (Poster)