Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:

This paper proposes the VarIational STructured Attention networks (VISTA-Net), which improves pervious SOTA models for dense pixel-wise prediction tasks. The proposed VISTA-Net is featured by two aspects: 1) A new structured attention is proposed, which is able to jointly model spatial-level and channel-level dependencies; 2) It incorporates the proposed structured attention with a CRF-like inference framework, which allows the probabilistic inference. Experimental studies are conducted on monocular depth estimation and semantic image segmentation, showing improved performances of VISTA-Net consistently.

##########################################################################

Reasons for score:

Overall, I vote for rejection. My major concerns lie in three aspects, as detailed in Cons below: 1) This work is highly similar to Xu et al. (2017a) in terms of both methods and presentations. The difference is not significant; 2) While the presentation mainly follows Xu et al. (2017a), it needs some improvement; 3) The experimental studies lack more detailed analysis on the proposed method.

##########################################################################

Pros: 

1. The work is well-motivated. The aim of the proposed method sounds natural to me.

2. I like the ablation studies. But they could be performed on at least one more dataset.

##########################################################################

Cons: 

1. This work is highly similar to Xu et al. (2017a) in terms of both methods and presentations. The difference is not significant.
Method-wise, as discussed in Related Work, the difference only lies in that VISTA-Net takes channel-level dependencies into consideration. First, this means that "Moreover, we integrate the estimation of the attention within a probabilistic framework" (quoted from abstract) is not a novel contribution. Second, considering channel-level dependencies in attention has limited novelty. As discussed in Related Work, multiple studies have explored several ways. In addition, a key step in the proposed method is Equation (1), where the tensor multiplication operator is not explained. In my understanding, it should be the outer product, or more generally, Kronecker product. Missing the clear definition of this operator hinders the clarity in describing the proposed method.
Presentation-wise, the similarity is even higher. The entire section 2 follows the exact organization of section 2 in Xu et al. (2017a). By comparing the equations and presentations, it's more convincing that the novelty of this work is quite limited.

2. While the presentation mainly follows Xu et al. (2017a), it needs some improvement. First, the same notations as in Xu et al. (2017a) are used. However, some key things are not well explained. For example, the set of hidden variables $z_s$ corresponding to $f_s$ comes out from nowhere. I have to resort to Xu et al. (2017a) to know why we need $z_s$. Second, as mentioned above, the key steps like Equation (1) lack clear explanations.

3. The experimental studies lack more detailed analysis on the proposed method. It would be more meaningful to visualize those attention maps/gates instead of dense prediction results.

##########################################################################

Questions during rebuttal period: 

Please address and clarify the cons above

##########################################################################

Comments after the rebuttal period:

Pros:

First, it is totally acceptable to follow the notations and organization of Xu et al. (2017a), as long as the statements are clear and self-contained. The original submission failed on providing key details. The authors have made revisions to address this concern. Thanks!

Second, I appreciate the extra experimental results and visualizations.

Cons:

However, the authors' responses do not fully address my concerns about the novelty, especially method-wise. I will raise my score to 5, but still recommend rejection.
review_title: Recommendation to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a unified method to combine spatial and channel attention in a probabilistic framework, so that spatial and channel attention weights and probabilistic variables can be jointly optimized. The proposed method is incoporated in the proposed VISTA-Net to achieve state of the art performance in two dense pixel-wise prediction tasks: monocular depth prediction, semantic segmentation.

Overall this work is well motivated and organized in a good shape, so that I am on the positive side. I have some concerns as listed below.

1. My main concern is that this paper combines ideas of (Fu et al., 2019) and (Xu etal., 2017a), i.e., combining spatial and channel attention from (Fu et al., 2019) and Attention-Gated CRFs from (Xu etal., 2017a). 

2. In Eq.1, the attention tensor is limited to T rank. However, it is not clear to me how this can faciliate the inference in Eq.9.

3. In Eq.2, this paper proposes to additionally model CRF kernels. However, it is not explained why this is necessary. This is important as it is listed as a difference to existing method (Xu etal., 2017a). 

4. The approximation used in Eq.3 needs more details or reference to support.


Minor issues.
1. There are a lot of symbols in section 2. It would be better to draw a figure to include these symbols, while illustrating how the CRF formulation is incorporated in the attention mechanism.
2. The joint learning in section 2.2 can be formed in a algorithm procedure, so that others could see it more clearly.
review_title: review 2893

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview

This paper carefully designs a structured attention network incorporating with a variational solution, to benefit the dense pixel-wise prediction via inferring the latent attention gate between spatial and channel-level features. Based on the suppose of T rank-1 attention tensors, the proposed structured attention network performs a CRF formulation with latent gating variables. 

Strengths

1.	Experiments are conducted on various dataset KITTI and NYU-v2 for depth estimation, PASCAL and Cityscapes for semantic segmentation, ScanNet for surface normal prediction. All the results demonstrate the outperformance.

2.	The motivation is clear. Jointly estimating spatial attention maps and channel attention vectors with a probabilistic framework.

Weakness

1.	The contribution is limited. Just a combination of two existing works, e.g., CRF-based models for multi-scale attention estimation and DANet for spatial and channel-wise attention. Why the T rank attention tensors are advanced than full-rank ones? How to guarantee the T channels optimal than the possible C?

2.	The generation of structured latent attention gate is confused. Compared with the work (Xu et al., 2017a), which illustrates the attention-gated CRFs with a clear graph, Figure 1 confuses me. Where is the gate? How to produce a probabilistically enhanced feature map? 

3.	The model is complex with four sets of latent variables (Z, M, V, and K) to be inferred. It is hard to be reproduced.

Questions:

1.	How to choose the parameter T? Ablation study might be added to demonstrate the difference with different T.

2.	In the inference stage, there are Z-step, M-step, V-step, and K-step. How to deal with so many variables to obtain the optimal performance? Is there any dependence on data size?

3.	For experiments of depth estimation, why the proposed method obtains optimal metrics but rel?



review_title: a probabilistic solution to jointly estimate spatial attention maps and channel attention vectors

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel structure attention mechanism with a probabilistic CRF-like inference framework for the pixel-wise prediction, which not only models the spatial-wise dependencies but also considers the channel-level dependencies using an attention tensor. This attention mechanism is obviously different from the existing attention mechanism. Additionally, many quantitative and qualitative experimental results on three pixel-wise prediction tasks have demonstrated the superiority of the proposed method. The proposed attention mechanism is useful and flexible, which could be integrated into any pixel-wise prediction frameworks in theory. However, I think there exist some disadvantages:

1. There are some unclear statements in the paper. For instance, the authors say “the attention tensor is nothing but the sum of T rank-1 tensors”, but how are these rank-1 tensors generated by the spatial attention map and the channel-wise attention vector? Is each rank-1 tensor associated with the special regions of different objects? If the authors visualize the attention map, it would be better to understand. Moreover, the overall architecture of the proposed method is not clear. The authors do not specify the details of the multi-scale features, including the dimensions of them, where they come from, etc. The dimensions of the most important variables are also not specified, which makes the paper is hard to follow.

2. In the ablation study, when the model is ‘no structure’ on a single scale, does it denote the model does not integrate any attention mechanism? If so, when only considering the spatial or channel attention, why does it outperform DANet that includes dual attentions? If not, I think the comparison is not fair. 

3. Although the performance of the proposed attention mechanism outperforms most existing methods, the authors do not report the runtimes and parameters. Moreover, whether does it significantly burden the complexity of the model with the increase of T?

4. There are some grammar mistakes and typos. For example, “Since the exact a posteriori distribution is not computationally tractable, ...”, “Again, our method not outperforms …”
 
review_title: There some unclear statements in this paper. However, generally speaking, this paper proposes a novel structured attention mechanism for the pixel-wise prediction tasks. Extensive experiments have been conducted to demonstrate the effectiveness and superiority of the proposed method. Therefore, I think this paper may be considered to accept.  

Meta-review: Reviewers were concerned with the novelty, although appreciated sota results in extensive experiments.
Decision: Reject