Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method based on generative vision transformer with latent variables 
following an informative energy-based prior for salient object detection.
Both the vision transformer network and the energy based prior model are jointly trained 
via Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation.

Extensive experiments reveal both high performance model and meaningful confidence map scores,
as well as estimation of uncertainty maps.
main_review: 
Not assuming the prior of the GAN to be Gaussian, and instead using an energy based NL function is a good start in the model.
Most the analytics is meaningful and well drafted.

The proposed model of learning by MCMC-based maximum likelihood where the Langevin sampling is used to evaluate the intractable posterior and prior distributions of the latent variables for calculating the gradients of the log-likelihood w.r.t. the model parameters, is quite interesting. The authors however do not claim it to be novel, wondering if a variant exists somewhere in literature ?

The tables 1-5 with performance measures, may also be put as bar-charts in Supple material, as its often hard to compare enhancement of your performance wrt nearest SOA, from decimal places in mind.

limitations_and_societal_impact: 
How do you validate the estimated uncertainty maps, in the absence of GTs for the same ?

Wonder what is the expansion of the acronym "D_CBase"  used ? What does it mean ?

What is MAE in fig. 4 ? Not defined anywhere in main doc. - got it in Supple though.

Fig. 9 of Supple material - how do you validate these complexity scores ? Even the GT or human evaluation is not provided.
What are the typical ranges of complexity values you get by the entropy based measure ?
How do you thus claim it to be consistent with human visual perception ?

Some typos:
line 169: ......local minimum, it solve the following estimating.... - change to : 
it solves the....

Some more references may be added as: 
1. Bayesian Learning via Stochastic Gradient Langevin Dynamics; Max Welling, Yee Whye Teh, ICML 2011;

2. Variance Reduction in Stochastic Gradient Langevin Dynamics; Avinava Dubey, Sashank J. Reddi, Barnabas P´oczos, Alexander J. Smola, Eric P. Xing, NIPS 2016.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The submission proposes a method for saliency detection based on a generative adversary learning framework, which uses a transformer architecture to replace the backbone of the framework and is jointly trained with a Markov chain Monte Carlo-based maximum likelihood estimation to improve the performance. The performance of the proposed method is evaluated on both RGBD- and RGB-based saliency prediction datasets. The paper is generally written well and easy to follow.


main_review: Although the overall performance of the proposed method is promising, the motivation, the contribution, and the effectiveness of the method are not very clearly illustrated and demonstrated. The reviewer has some detailed comments below:

(i)	The motivation of using a GAN-based method for saliency detection is not that clear. Does it really outperform the convolutional-based discriminative models for the task? From the results, the reviewer also cannot see any performance comparison to show the advantages of using a GAN-based method as a baseline. 

(ii)	It is not clear that if the contribution is an improvement of GAN-based model with a transformer, or a method that advances saliency detection. If the focus of the paper is a new GAN-based method with transformer, and saliency detection is just an application, the authors should illustrate more differences to existing works that using GAN with transformers. 

(iii)	The effectiveness of energy-based prior distribution seems not demonstrated in the experimental results. The whole ablation study seems to be very limited. 

(iv)	The authors claimed state-of-the-art performance on saliency detection. However, the comparison with previous sota methods is significantly not enough. To support that claim, the authors need to compare more methods on more challenging datasets.  

(v)	For the presentation, an overall framework depiction could help the readers to understand the whole thing more easily. It is also important to elaborate more, probably with a figure as well, on how the proposed energy-based strategy can be jointly optimized with the network.

(vi)	The authors showed some visualization of the learned uncertainties, but the reviewer is not clear that why the uncertainty is critical in the method. More explanation on the motivation of the uncertainty is definitely helpful, and corresponding experimental results to show the benefits of the uncertainty are also needed. 

limitations_and_societal_impact: The motivation and contribution are not clear, and the experimental results and ablation study are not sufficient. Please check the main review part for more detailed comments.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel* supervised generative model for detecting salient image regions, where saliency is defined as containing object. This is a latent-variable model with an energy-based prior and a transformer decoder that maps the latent variable to a saliency map. The prior and posterior inference is done via SGLD. The model is well-evaluated on several datasets, with several metrics, and compared across a range of state-of-the-art baselines. It not only outperforms every single baseline (setting new SOTA), but also has the added benefit of providing high-quality uncertainty estimates. Detailed ablation studies show the relative importance of various model components.

\* I think it is novel, but difficult to say as I am not expert in the field and there is very information in the paper about prior art.

# Update
I'm happy with the author response and am increasing my score to 7. I still have some doubts about the motivation of this work, but the theory is sound and the the results are very good.
main_review: Originality: 
- The task is well-established (silent object detection, or SOD for short. This is at least as far as I can tell, given the number of baselines.
- It is not clear how this work differs from prior art. Even though there are numerous baselines in the experiments, they are not discussed. The related work section lists a huge number of related works but does not discuss how these differ from the current work. Or rather, it does so only implicitly, stating only very shortly what some other methods do. Since I am not an expert in SOD, it is difficult for me to understand the differences to the current method with so little information.
- This work is a combination of well-known techniques: energy-based generative modelling for the prior, SGLD for prior and posterior inference, a transformer for decoding. It is this combination and its application to SOD that, as far as I can tell (I am not an expert in the field), is novel.

Quality:
- The presented method and results are very good quality, but not very surprising: it is clear that a generative model can provide good uncertainty estimates. 
- The paper needs some writing work (more in clarity).

Good:
- The submission is technically sound. The details of the method are well-explained and seem correct.
- An extensive experimental evaluation is provided. The method outperforms numerous baselines on several datasets.
- The authors provide a short-but-sufficient analysis of the learning algorithm (with prior/posterior SGLD sampling).
- There are numerous ablation studies evaluating importance of model components.
- I really like the "hyperparameter analysis" section in L263-L276. It gathers and describes all hparams of the method and says how they were tuned and if the model was sensitive. Nice!

Bad:
- Even though the model can provide some uncertainty estimates, it cannot estimate the uncertainty in the model parameters. The authors do not comment on that.
- One of the main motivations of an EBM prior is that a correlated prior/posterior should do better than (conditionally-) independent ones as in a GAN or a VAE. While "obvious" on the surface, we have to remember that an independent prior + a single MLP layer (not to mention more) can lead to a highly-correlated prior, cf. normalizing flows. Therefore, I would say that his motivation is invalid---despite the fact that the authors instantiate GAN and VAE variants of their model and show they underperform the EBM. 
- I am not convinced that Eq. 4 is fully correct. I tried deriving it, but I only managed to get the expectation of the gradient with respect to the model distribution of (latent, saliency), but training is done with ground-truth saliency and posterior-sampled latents. Could you please walk me through the derivation?
- In L198 you say that your method is more memory-efficient than e.g. VAE because there is no encoder. However, for SGLD, you need to evaluate the prior and the decoder multiple times. Do you backprop through SGLD? If yes, then you need to store activations of every prior/decoder evaluation, which renders your method *very* memory-inefficient. Care to elaborate?

Clarity: 
- This submission would benefit from some more writing work. It is quite unclear in places, see comments below. 
- There is a sufficient level of detail to reimplement the model, provided that the reader can understand the paper.
- There are a lot of typos, e.g. L44 none->non, L75 modal->modality, L226 & L278 analysis -> analyse.
- "SOD" is first used in L65 but not explained; I needed to infer that it means "salient object detection". Similar "saliency prediction problem" appears in L103 without definition. The paper does not state until very far into the text that it tackles supervised learning.
- The related works section is almost unreadable. It lists tens of different works without giving any differences between them, nor specifying how they relate to the current work.
- Since prior/posterior samples are defined with + and - signs, it would be nice to define the related hyperparams in Alg 1 using + and - as well (instead of 0 and 1).
- L192 "and other newly added layers randomly initialized" <- I've no idea what this refers to.
- It is very unclear that Tab. 3 lists deterministic baselines; perhaps add that in the caption? and also mention in Tab. 4 caption that a deterministic variant of "ours" is available in Tab. 3. The paragraph describing the deterministic backbone study (L227-238) is very difficult to read and the fact that the model is now deterministic is highly obscured.
- Unclear what "scales" in Fig. 4 means.
- Unclear what 0/1 as supervision in L251 means.
- Fig. 5 is difficult to read; please provide an interpretation.
- Please add a ylabel to Fig. 6.


Significance: 
- While it is nice to see an application of energy-based generative modelling to supervised learning, I am not sure how significant this is. I am not an expert in this field, and I have no idea where SOD is used and what are the benefits as compared to e.g. object detection or instance/panoptic segmentation.
- Given that there is a number of SOD baselines in the paper, I can be convinced that SOD is an important research area. Since the presented results are quite a bit better than the baselines, and the method is simple, it is likely that it will be built upon. 

I would be happy to increase my score, potentially quite significantly, if the authors address my concerns above and clearly delineate themselves from prior art.
limitations_and_societal_impact: Adequate discussion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The author construct a generative model for salient object detection in the form of top-down conditional latent variable model.
A generative vision transformer is used by adding latent variables into the traditional deterministic transformer. Experimental results on RGB and RGB-D image salient object detection show that the generative framework equipped with the EBM prior and the transformer-based non-linear mapping is powerful in representing the conditional distribution of object saliency given an image.
main_review: This paper exhibits a generative vision transformer for salient object detection. The idea using generative vision transformer to address the salient object detection is creative. The extensive experiments also show that it achieves the SOTA performance. 

My concerns and suggestions about this paper is that many details are missing.
1. Figure 1 presents the uncertainty map, but there is no introduction presented as it is important in this paper. what is the uncertainty map, how to get it?
2. In line 51, EBM occurs at the first time, but there is no introduction about what is EBM in the whole paper. 
3. In line 122, ''energy-based prior, which is presented in Eq.(1)'', which one term is the energy based prior? In addition, in some places, energy-based prior is mentioned or energy-based prior model. Does the two mean the same?
4. In line 162-166, the energy-based prior model is introduced, but it is not shown in Fig2. How does the transformer and the prior model work together? it should be explained in more detail.
5. For equation 7 and 8, it uses z_t and z_(t+1), but in the algorithm 1, step 4, 5, z_{t}^{-} and z_{t}^{+} are used. It's ambiguous to differentiate them.
6. In line 194-198, computation and memory cost are introduced. But there is no detailed comparison given in the paper. I think the comparison between the proposed and previous SOTA for computation time and memory should be detailed.
7. The performance comparison should include transformer based method, like recent work "P2T: Pyramid Pooling Transformer
for Scene Understanding".

limitations_and_societal_impact: The suggestions are presented in Main Review.
review_title: null

Meta-review: This paper describes a novel transformer-based latent variable model for salient object detection. Reviewers were generally positive on the approach and results, and the work was deemed to be technically sound. The reviews and authors had a productive discussion which we hope will lead to some clarifications that will enhance the paper. One reviewer provided a score below the threshold for acceptance, but this reviewer appears to have had some confusion about the nature of the method and did not reengage with authors when they sought to clarify; as such, this review was down-weighted in the overall evaluation. All in all, this is an interesting method producing state of the art results, with solid theoretical grounding.
Decision: Accept (Poster)