Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction
Authors: Jing Zhang
Jianwen Xie
Nick Barnes
Ping Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Salient object detection
Generative vision transformer
Energy-based model
Langevin dynamics

Abstract: Vision transformer networks have shown superiority in many computer vision tasks. In this paper, we take a step further by proposing a novel generative vision transformer with latent variables following an informative energy-based prior for salient object detection. Both the vision transformer network and the energy-based prior model are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables are performed by Langevin dynamics. Further, with the generative vision transformer, we can easily obtain a pixel-wise uncertainty map from an image, which indicates the model confidence in predicting saliency from the image. Different from the existing generative models which define the prior distribution of the latent variables as a simple isotropic Gaussian distribution, our model uses an energy-based informative prior which can be more expressive to capture the latent space of the data. We apply the proposed framework to both RGB and RGB-D salient object detection tasks. Extensive experimental results show that our framework can achieve not only accurate saliency predictions but also meaningful uncertainty maps that are consistent with the human perception. 

Learning Generative Vision Transformer with Energy-Basee Latent Space for Saliency Prediction
Jing Zhang', Jianwen Xie', Nick Barnes", Ping Li'
Cognitive Computing Lab, Baidu Research
2 The Australian National University
(zjnwpu, jiawennkknn pingli98) @gmail.com, nick. @ @ww.edu.uuuaaaa..
Abstract
Vision transformer networks have shown superiority in many computer vision tasks. In this paper, we take step further by proposing a novel generative vision transformer with latent variables following informative energy- energy-based prior for salient object detection. Both the vision transformer network and the energy-based prior model jointly trained via Markoy chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables performed by Langevin dynamics. Further, with the generative vision transformer, we can easily obtain pixel-wise uncertainty map from image, which indicates the model confidence in predicting saliency from the image. Different from the existing generative models which define the prior distribution of the latent variables our model uses an energy-based informative prior which can be more expressive capture the latent space of the data. We apply the proposed framework to both RGB and RGB-D salient object detection tasks. Extensive experimental results show that our framework can achieve not only accurate saliency predictions but also meaningful uncertainty maps that are consistent with the human perception.
simple isotropic Gaussian distribution,
1 Introduction
In the field of computer vision, salient object detection [64, 65, 16, 17, 89] (SOD) or visual saliency prediction, which aims at highlighting objects more attentive than the surrounding areas in images, has achieved significant performance improvement with the deep convolutional neural network revolution. Given a set of training images along with their saliency annotations, the conventional SOD models seek to learn a deterministic one-to-one mapping from image domain to saliency domain.
Two main issues exist in the above conventional deep saliency prediction framework: (i) the convolu- tion operation based on sliding window makes the deep saliency prediction model less effective in modeling the global contrast of the image, which is essential for salient object detection [7]; (ii) the one-to-one deterministic mapping mechanism makes the current framework not only impossible to represent the pixel-wise uncertainty in predicting salient objects, but also hard to handle incomplete data in a weakly supervised scenario [89]. Besides, given an image, the saliency output of a human subjective, therefore, a stochastic generative model is more natural than deterministic model for representing saliency prediction. Although [85] introduces a conditional variational autoencoder (CVAB) [56] for RGB-D salient object detection, the potential posterior collapse problem [23] makes the stochastic predictions less effective in generating meaningful uncertainty estimation.
Motivated by the above issues, propose novel framework, the generative vision transformer, for salient object detection, where a vision transformer structure [40] is used as a backbone and latent variables introduced in designing generative framework. On the one hand, transformers [60] have proven to be very effective in long-range dependency modeling, and are capable of modeling
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

various scopes of object context information with the multi-head self-attention module. With such an architecture, we can achieve global context modeling for effective salient object detection. On the other hand, the latent variables account for randomness and uncertainty in modeling the mapping from image domain saliency domain, and also enable the model to produce stochastic saliency predictions for uncertainty estimation. Therefore, the proposed model is latent variable transformer.
Nowadays, there are two types of generative models that have been widely used, namely the varia- tional autoencoder (VAE) [31] and the generative adversarial network (GAN) [20], which correspond to two different generative learning strategies to train latent variable models. To train a top-down latent variable generator. the VAE introduces an extra encoder to approximate the intractable pos- terior distribution of the latent variables, and trains the generator via a perturbation of maximum likelihood; while the GAN introduces discriminator to distinguish between generated samples and real data, and trains the generator fool the discriminator. [22, 69] present the third learning strategy, namely alternating back preppprppaattoon (ABP), to train the generator with latent variables being directly sampled from the true posterior distribution by using a gradient-bar baaeed Markov chain Monte Carlo (MCMC) [38], e.g., Langevin dynamics [43, 66, 12]. All the three generative models define the prior distribution of the latent variables as a simple non-informative isotropic Gaussian distribution, which is less expressive in capturing meaningful latent representation of the data.
In this paper, we investigate generative modeling and learning of the vision transformer. We construct generative model for salient object detection in the form of top- conditional latent variable model. Specifically, we propose generative vision transformer by adding latent variables into the traditional deterministic vision transformer, and assume the latent variables follow an informative trainable energy-ban -base prior distribution [47, 48]. Following [72], we parameterize the energy function of the energy-based model (EBM) by a deep net. Instead of using variational learning or adversarial learning, we jointly train the parameters the EBM prior and the transformer network by maximum likelihood estimation (MLE). The MLE algorithm relies on MCMC sampling to evaluate the intractable prior and posterior distributions the latent variables.
Experimental results on RGB and RGB-D salient object detections [64, 68. 85, 16] show that the generative framework equipped with the EBM prior and the transformer-basse non-lineat mapping powerful in representing the conditional distribution of object saliency given an image, leading to more reasonable uncertainty estimation as shown in Figure where stochastic saliency prediction provided by learned model and the visualization of the pixel-wise uncertainty is presented.
Image
GT
Predictions by sampling
Uncertainty
Figure 1: An illustration of the stochastic saliency prediction obtained by the proposed generative vision transformer with an EBM prior, as well as the corresponding pixel-wise uncertainty map.
We summarize our main contributions and novelties as follows: (i) we propose a novel top-down generative vision transformer network with an energy-based prior distribution defined on latent space for salient object detection; (ii) we jointly train the vision transformer network and the energy-based prior model by an MCMC-based maximum likelihood estimation, without relying any extra assisting network for adversarial learning or variational learning: (iii) we achieve new benchmark results for both RGB and RGB-D salient object detections, and obtain meaningful uncertainty maps that are highly consistent with human perception for saliency prediction.
Related Work
Salient object detection:
The main goal of the existing deep fully-supervised RGB amage-based
salient object detection models [67, 41, 68, 64, 61, 87, 65] achieve structure preserving saliency prediction, by either sophisticated feature aggregation [67, 61]. auxiliary edge detection [68, 52, 65], or resorting to structure-aware loss functions [41, 64). With extra depth information, RGB-D salient object detection models [53, 85, 5, 27, 93, 90, 16, 28, 51, 92, 59, 86] mainly focus effective multi-modal modeling. Our paper solves the same problems, i.e., RGB and RGB-D salient object detection, by developing new generative transformer-basee framework.

---

Vision transformers: The breakthroughs of the Transformer networks [60] in natural language processing (NLP) domain have sparked the interest of the computer vision community in develop- ing vision transformers for different computer vision tasks, such as image classification [10, 40], object detection [4, 63, 6, 40], image segmentation [96, 54, 63, 40], object tracking [80, 81], pose estimation [42, 58), etc. Among them, DPT [54] adopts a U-shape structure and uses ViT [10] as an encoder to perform semantic segmentation and monocular depth estimation. Swin [40] presents a hierarchical transformer with shifted windowing scheme to achieve efficient transformer network with high resolution images as input. Different from the above vision transformers that mainly focus on discriminative modeling and learning, our paper emphasizes generative modeling and learning of the vision transformer by involving latent variables and MCMC inference.
Dense prediction with generative models: VAEs have been successfully applied to image segmenta- tion [3, 32]. For saliency prediction, [34] adopts a VAE to model the image background, and separates salient objects from the background through reconstruction residuals. [85, 84] design CVAEs to model the subjective nature of saliency. GAN-based methods can be divided into two categories, namely fully-supervised and semi-supervised settings. The former [21, 33] uses the discriminator to distinguish model predictions from ground truths, while the latter [57, 26] uses the GAN to explore the contribution of unlabeled data. [88] uses cooperative learning framework [71, 74] for generative saliency prediction. [84] trains a single top-down generator in the ABP framework for RGB-D saliency prediction. Our model generalizes [84] by replacing the simple Gaussian prior by learnable EBM prior and adopting a vision transformer- based generator for salient object prediction.
Energy-based models: Recent works have shown strong performance of data space EBMs [72, 44] in modeling high-dimensional complex dependencies, such as images [97, 95, 18, 11, 19], videos [78, 791, 3D shapes [75, 76], and point clouds [73], and also demonstrated the effectiveness of latent space EBMs [47] in improving the model expressivity for text [48], image [47], and trajectory [49] generation. Our paper also learns a latent space EBM as the prior model but builds the EBM on top of a vision transformer generator for image-conditioned saliency map prediction.
3 Generative Vision Transformer with Energy-Based Latent Space
3.1 Model
We formulate the supervised saliency prediction problem as a conditional generative learning problem. Let I E Rxx be an observed RGB image, s E hxx be the saliency map, and E IT g1x1xd be the d-dimensional vector of latent variables, where w >> d. Consider the following generative model to predict a saliency map s from an image I,
(I) where T's is the non-linear mapping process from [z, 1] to s with parameters 0, pa (z) is the prior distribution with parameters a, and ~ NO0 o²() is the observation residual of saliency with o, being given. Due to the stochasticity of the latent variables 2, given an image its saliency map also stochastic. Such a probabilistic model is in accord with the uncertainty of the image saliency.
= (I, + 2~ P=(;;, E E~ (
Following [47], the prior Pa (2) is not assumed to be simple isotropic Gaussian distribution as GAN [20]. VAB [31, 56) or ABP [22]. Specifically, it is in the form of the energy-based correction or exponential tilting [72] of an isotropic Gaussian reference distribution po(=) NN o I),, i.e.,
Pa (2) ((a) exp (-Ua(=))
Po)=) exp UU)(=)
|
20
where (2) = (z) 207 ||= is the energy function that maps the latent variables z to a scalar, and (z) is parameterized by multi-layer perceptron (MLP) with trainable parameters The standard deviation o₂ is a hyperparameter. Z (a) = expl-Ua (=)[po (=))po(2)d2 is the intractable normalizing constant that resolves the requirement for probability distribution to have total probability equal to one. Pa (2) is an informative prior distribution in our model and its parameters a need to be estimated along with the non-linear mapping function To from the training data.
The mapping function To is parameterized by a vision transformer [40] with self-attention mechanism, which encodes the input image I and then decodes it along with the vector of latent variables to the saliency map 8, thus, po po(s], 2) (T, 2), T). The resulting generative model is conditional directed graphical model that combines the EBM prior [47] and the vision transformer [40].
3

---

3.2 Learning
The generative transformer with an energy- based prior, which presented in Eq. (I), can be trained via maximum likelihood estimation. For notation simplicity, let 3 = (0, a). For the training examples ((I, st), I,....,,, the observed-data log ligeelklioood function is defined as
((3)= = Σ logpo(s. ==1
Σ log 1=1
|PPo(s, 2, 2,1
= Σ log
[f Paales)oossl (....,))) 2,)dz
==1
Maximizing ((3) is equivalent to minimizing the Kullback-Leibler (KL) divergence between the model pa (s(I and the data distribution Paata (s|I). The gradient of ((3) can be computed based on
V log ( I)) = Eps(= a.I) VV a log Pa (8, =|1)] = E) 2 Paa (2(2) + log po I],] (3)
where the posterior distribution ps(2]s (z|s, I) pa(s, = (s)pp(ss)) Pa (z) po (s (s 2)/p>(ss)))) I).
The learning gradient in Eq. (3) can be decomposed into two parts, i.e., the gradient for the energy- based model
(=)ss.) V a log Pa(]]] (p. VVa (2) (p (aU
(4)
and the gradient for the transformer 0
E ( Vo lo (s|I, =)] = (alo) 2 To (o)
(5)
(z) in Eq. (4) and (L. 2) in Eq. (5) can be efficiently computed via back- soeckpropagation Both Eq. (4) and Eq. (5) include intractable expectation terms (-), which can be approximated by MCMC samples. To specific, we can use a gradient-based MCMC, e-g.. Langevin dynamies, which is initialized with Gaussian noise distribution po. to draw samples from the energy-based basey prior model Pa (2) x cxp - Ea (z) by iterating
2++1 = 2t oV Ea (zt) + V25c,, 2u ~ po(2), et ~ N),,
(6)
and draw samples from the posterior distribution pa(2|s, ss I) by iterating
o? 1 (s (L V.) 2)) + 28e,, 20 po(2)) ~ I).,
2411 24 8 V (21)
5 is the Langevin step size and can be specified independently in Eq. (6) and Eq. (7). We use (2 and {2, } to denote, respectively, the samples from the posterior distribution (=(=|s, 1) and the prior distribution (=). The gradients of and can be computed with (J,,8:)) {2t) and (,, by
1 [aUa(
Va V.U.(25)
(8)
n2
V8
(s. To TL,,, VoT
(9)
We can update the parameters with Va and V0 via the Adam optimizer [30]. We present the full learning and sampling algorithm of our model in Algorithm 1.
Algorithm Maximum likelihood learning algorithm for generative vision transformer with energy- based latent space for saliency prediction
Input: (1) Training images (I,) with associated saliency maps (s,) (2) Number of learning iterations M; (3) Numbers of Langevin steps for prior and posterior (K" K²); (4) Langevin step sizes prior and posterior {8-,8" 8+ 3; (5) Learning rates for energy-hased prior and transformer {E.,, {o (6) batch size n'.
Output: Parameters 0 for the transformer and for the energy-based prior model
1: Initialize 0 and a
2: for m 4 1 to MM do
3: Sample observed image-saliency pairs {(L,,
4: For each s:), sample the prior 2, ~ Por.. (z) using K Langevin steps in Eq. (6) with a step size
8-
5: For each s;), sample the posterior 27th ~ Ps.. (2|8,, I.) using K* Langevin steps Eq. (7) with
step size 5+.
6:
Update energy-based prior by Adam with the gradient Va computed in Eq. (8) and learning rate E..
7: Update transformer by Adam with the gradient 40 computed in Eq. (9) and a learning rate Ep.
8: end for

---

3.3 Network
Generative vision transformer: We design the generative vision transformer using the Swin transformer [40] backbone as shown in Figure 2, which takes a three-channel image I and the latent variables as input and outputs a one-channel saliency map (1 2). Two main modules are included in our generative vision transformer, including a "Transformer Encoder" module and a "Feature Aggregation" module. The former takes I as input and produces a set of fea- ture maps {fi]== of channel sizes 128, 256, 512, 1024 and 1024, respectively, while the latter takes { and the vector of latent variables z as input to generate the saliency prediction s.
Specifically, we first feed each fi to a 3 x 3 convolutional layer to reduce the channel dimension to 32, and ob- tain new set of feature maps { after channel reduction. Then, we repli- cate the vector 2 spatially and perform a channel- -wiee concatenation with f/+ followed by a 3 x 3 convolutional layer that seeks to produce a feature map Fs with same number of channels as that of fz Finally, we sequentially concatenate feature maps from high level to low level via feature aggregation, i.e., from = to 1, we compute Convaxa (M [M(Concat(f]- Fi+1, .... Fss)), where Convax3 (-) is 3 3 convolutional layer that reduces the channel dimension to 32, M(-) is the channel attention module [91], and Concat(-) is the channel-wise concatenation operation. Note that, upsample the higher level feature map the same spatial size as that of the lower level feature map hefore the concatenation operation. We feed F1 to 3 x 3 convolutional layer to obtain the one-channel saliency map To (I)
Transformer Encoder
Encoder
Feature
Aggragation
Figure 2: Generative latent variable vision transformer
Energy- Gnasey-baseed prior model: We design an energy-based model for the latent variables z by param- eterizing the function (z) via a multilayer perceptron (MLP), which uses three fully connected layers to map the latent variables z to scalar. The sizes of the feature maps of different layers of the MLP are Ce, C. and 1, respectively. We will simply Ce to represent the size of the EBM prior and set Ce = 60 in our experiment. GELU [25] activation is used after each layer except the last one.
3.4 Analysis
Theoretically, when the Adam optimizer of 3 = (0, a) in the learning algorithm converges a local minimum, it solves the following estimating equations
Convergence:
v
E Ls (aU - (p) [aUa(z)]
(10)
1 (st (st To (1, 21
V0 = 0
(=
= 0,
(11)
which are the maximum likelihood estimating equations. However, in practise, the Langevin dynamics Eq. (8) and Eq. (9) might not converge to the target distributions due the use of small number of Langevin steps (i.e., short-run MCMC), the estimating equations in Eq. (10) and Eq. (11) will correspond perturbation of the MLE estimating equation according to [44, 45, 47]. The learning algorithm can be justified as a Robbins- Monro [55] algorithm, whose convergence is theoretically sound. Our model can also be trained with an extra encoder as an amortized inference network [31] for (=]2|s, I) and an extra generator as an amortized sampling network [70. 71, 77] for Pz) In this work, we prefer to keep our training algorithm simple in order to avoid extra efforts for the design of the auxiliary network architectures. We will study the joint training strategy in our future work.
Accuracy: Compared with the GAN-based generative framework, our model likelihood-baäd based generative framework that will not suffer from mode collapse [2]. In comparison with VAE-based gen- erative framework, whose training is also based on likelihood, our MCMC-based maximum likelihood learning algorithm will not encounter the posterior collapse issue that is caused by amortized inference. On the other hand, the variational inference typically relies on an extra inference network for efficient inference of the latent variables given an image and saliency pair, however, the approximate inference might not be able to take the full place the posterior distribution in practise. To be specific, we use qs (2|s s.s I) to denote the tractable approximate inference network with parameters ø. The variational
5

---

Figure 3: Visual comparison of our model and the stateof-the-aa saliency prediction model, the BBSNet [16]. From top to bottom: images, ground truth saliency maps, results of the BBSNet [16] and results obtained by our model.
inference seeks to optimize ming ming KL (z)))) D)lpp( s(D), which can be further decomposed into ming ming KL(Pamal [spD))ppp( [ (L (2)s. [[lps (,,, I)). That is, the varia- tional inference maximizes the conditional data likelihood plus KL-divergence between the approxi- mate inference network and the posterior distribution. Only when the (L(go(2|a, D) D)) + O, the variational inference will lead to the MLE solution, which is exactly the objective of our model. However, there might exist gap between them in practise due to the improper design of the inference network. Our learning algorithm is much simpler and more accurate than amortized inference.
Computational and memory costs analysis: From the learning perspective, due to the iterative Langevin sampling for the posterior and prior distributions of latent variables, model is more time-consuming for training than generative models with amortized inference, such as VAE, which is roughly 2.4 times faster than VAE, the inference model is parameterized by another set of parameters, which need to be updated by pack-propagation. In our model, the Langevin dynamics is not treated the top-down generator is updated in each iteration, the posterior distribution can be derived from the generator. With the posterior distribution, the Langevin sampling is just an optimization-like process to find fair samples in the latent space defined by the posterior distribution. Without relying on an extra inference network, framework efficient in memory and friendly for network design.
RGB image-based salient object detection. However, for
model because once
Experimental Results
4.1 Setup
Datasets: For RGB SOD, train models on DUTS training set [62], and test them on five benchmark datasets, including DUTS testing set, ECSSD [82], DUT [83], HKU-IS [35] and PASCAL-S [36]. For RGB-D SOD, we follow the conventional training setting, in which the training set is combination of 1,485 images from NJU2K dataset [29] and 700 images from NLPR dataset [50]. We test the trained models NJU2K testing set, NLPR testing set, DES [8], SSB [46] and SIP [15) testing set.
Evaluation Metrics: We adopt four evaluation metrics to measure the performance, including Mean Absolute Error M, Mean F-measure (Fz), Mean E-measure (Eş) [14] and S-measure (S.) [13].
Implementation Details: Our generative vision transformer is built upon the Swin transformer [40] and uses it as the backbone of the encoder in our framework. The Swin backbone can be initialized with the parameters pretrained on the ImageNet-1K [9] dataset for image classification, and the other parameters of the newly added components, including the decoder part and the MLP of the energy based prior model, will be randomly initialized from the Gaussian distribution A (0 0.01). Empirically we set the number of dimensions d of the latent variables as d = 32. We set Eq. (2). We resize all the images and the saliency maps to the resolution of 384 x 384 pixels to fit the Swin transformer. The maximum epoch is 50. The initial learning rates
1 in
Eq. (1) and a=

---

are 2.5 X 10 The whole training takes 9 hours with a batch size n' = 10 on one NVIDIA GTX 2080Ti GPU for each model. During testing, our model can process 15 images per second.
4.2
Performance Comparison
We compare our framework with the sfate-of-thh-arr RGB SOD models and RGB-D SOD models, and show a comparison of performance in Table 1 and Table 2 respectively, where VST [39] is the only transformer-base saliency detection model. We observe consistently better performance of the proposed frameworks. Especially for PASCAL-S dataset [36], which contains more than 40% examples with large-sized salient objects, the significant performance gap between our method and the existing solutions demonstrates the effectiveness of our generative framework for saliency prediction. We further show qualitative comparison between our RGB-D saliency prediction model and the BBSNet [16] in Figure As we can see, our transformer- baased framework, with an effective global context modeling, is superior to its competitor in detecting various sizes of salient objects.
Table 1: Performance comparison with benchmark RGB salient object detection models.
DUTS [62]
ECSSD [82]
DUT [83]
IIKU-IS [35]
PASCAL S [36]
Method
S. t Fa MM
S..
1
Fg
JM
S.. 1
Fg 1
AM
S.. t Fp 1 Eg I
S., Fg 1 Egg M.
CPD [67]
869
.821
1913
.909
.937
.742
.847
938
.819 .882
SCRN [68]
.833
.00
.920
-910
.749
.847
.116
935
.892
PoolNet [37]
887
.840
.910
.037
919
.913
.938
.038
.81
.788
.848
.054
.919
993
.945
.00
8665
.335
.896
.065
BASNet [52]
876
882
.896
.48
911
913
9938
.000
836
.767
.865
.057
.909
903
.943
.032
838
818
.879
.076
EGNet [94]
878
824
.98
.043
914
906
993
.03
840
7755
.855
.054
917
9000
943
.031
852
823
.881
.074
F3Nel [64]
888
852
.900
.035
.919
921
.943
.036
.339
.766
.864
.053
.977
-910
.922
.028
.661
.835
.898
.062
ITSD [98]
886
841
917
.039
.920
916
.943
.037
842
.767
.867
.056
.911
906
.55
.030
860
.300
.894
.066
SCNet [88]
.902
870
.936
.032
.928
933
9955
.030
8447
.778
.879
.053
.927
.917
.900
.026
.83
8446
.909
.058
LDF [65]
892
861
.925
.034
.919
922
.943
.036
839
.770
.865
.052
.920
.933
.953
.028
860
885
.901
.063
VST
896
842
918
-037
9332
.911
.943
.034
850
.771
.869
.058
928
903
950
.030
873
.832
.900
.67
Ours
.988
.855
.942
.029
.935
.935
.962
855
.797
.892
.90
.922
964
885
.915
Table 2: Performance comparison with benehmark RGB-D salient object detection models.
NJU2K [29)
SSB
[46]
DES [8]
NLPR
[50]
SIP [15)
Method
S, f
Fa
Eg
.M
5,
7
Fa
Eg
JM
S.
F3 t Eg
1
S, t
F2 t Eg
PM
S. t Fa f Eg 1 IM
BBSNet [16]
921
-902
938
.035
908
.883
.928
.041
.933
-910
.49
.021
930
889
.950
.023
.879
868
.906
.055
BiaNet
.915
-903
9004
.879
.911
.910
8944
.988
.913
CoNet [28]
911
.00
944
.877
.939
906
.80
.999
900
.59
937
855
9155
UCNet
897
886
930
.04
903
884
.38
.09
934
919
.967
0099
920
891
951
875
867
.14
JLDCF
902
.855
935
.041
903
.83
.936
.40
931
9907
.959
.011
925
894
.955
002
880
8873
.918
.049
DSA2F [59]
900
9001
923
.03
9004
898
.933
.036
920
.896
.92
.021
918
897
.50
.024
VST [39)
.922
898
939
.03
913
877
.937
.33
943
.20
965
.017
932
.99
.91
.024
.04
.894
.93
.040
Ours
.929
924
956
.028
911
898
.950
.032
945
.928
.971
.016
.988
.921
966
.018
.906
9988
.40
.037
4.3
Backbone Analysis
To test the performance of the transformer structure proposed in Section 3.3, we compare it with a conventional convolutional backbone for salient object detection. We create the baseline by replacing the Swin encoder in our transformer structure by the ResNet50 [24] encoder and keeping the decoder part (i.e., the "Feature Aggregation" module shown in Figure 2) unchanged for fair comparison.
We first test them without involving latent variables, which means that we need to train them in a discriminative manner. We use "Ours-Swin-D" to denote the model using the Swin encoder and "Ours- Res50-D" to denote the one using the ResNet50 encoder. Model performance in the task of RGB-D saliency detection are shown in Table 3. We observe that "Ours-Swin-d outperforms "Ours-Res50 D", which clearly indicates the effectiveness of transformer backbone for salient object detection.
Table 3: Analysis of different backbones without involving latent variables.
NJU2K [29]
SSB [46]
DES [8]
NLPR [50]
[IS]
Method
S. Fa 1 Eg 1
S. 1
F3 1
Eg 1
1
S. 1
Fa t E, 1 1MM
5,
1
Fa
E, 1 M L
S. 1
F3 1 Eg 1 J
Ours-Ros50-D
.908
.896
991
.36
906
889
927
.07
918
.907
.48
.022
921
893
.951
.024
874
.856
.917
.049
Ours DD
.919
.933
.947
.032
914
.97
943
.033
931
.919
.599
.022
.933
.912
.951
.022
.897
.899
.931
.041
7

---

Table 4: Performance of different backbones within our model for RGB saliency prediction.
DUTS [62]
ECSSD [82]
DUT D8T
IS [35]
PASCAL [136]
Method
Su 1 Fg 1 Eg 1 MM
Su
1
Fg 1 E
.M
4
Sta
Fa
1 Bg
M.M
S.
Fg
1 Bg
1 .M
Su
F. 1 Eg 1 .M
Ours Rex50
.890
.850
.927
.055
918
914
944
.356
.337
.762
867
.033
917
.906
.952
.029
.559
830
.896
.063
Ours-DPT
.899
.874
.40
.031
.24
.933
.956
.031
.84
.792
.80
.044
.92
.920
.960
.026
.870
8844
.911
.055
Ours-Swin
.988 .875
942
.029
.955
9935
.92
.026
.858
.797
892
.011
930
.92
.964
.023
.877
.855
.955
.054
Table 5: Performance of different backbones within our model for RGB-D saliency prediction.
NJU2K [29]
SSB
[46]
DES [8]
NLPR [50]
SIP [15]
Method
S. 1 F3 t Eg tM
S. 1 Fa Eg
MMM
S. 1 Fg
Bg
1
8. t Fg
Eg
MM
US. 1 Fa Eg TMM
Ours-Res50
.199
946
906
882
9937
.38
925
974 017
920
949
.25
.882
.822 .918
.049
Ours.DPT
924
950
915
892
.344
921
968
935
964
901
.933
.038
Ours-Swin
.999 .96
.966
.898
.032
.988
.016
.988
.96
.906 .988
.037
We further study the influence of different encoder backbones in the context of the proposed generative saliency prediction framework with an EBM prior. Table 4 and Table 5, respectively, depict the performance comparisons of our frameworks using different backbones in the tasks of RGB and RGB- D salient object detections. We compare the ResNet50 encoder backbone and the Swin transformer encoder backhone. We also modify the DPT [54] backhone such that it can adapt to the tasks of RGB and RGB-D salient object detections with our framework. The DPT built on the ViT transformer [10] originally designed for semantic segmentation and depth estimation. The comparison results verify the effectiveness of the vision transformer used in our generative framework. In particular, our current solution with the Swin transformer encoder backbone can achieve the best performance for saliency prediction. Further, the performance gap between "Ours-Swin-D" in Table and "Ours-Swin" in Table 5 indicates the effectiveness of the generative learning with an expressive latent space.
4.4 Generative Learning Analysis
We compare our framework with other alternative generative solutions in Table 6. The results are reported in the task of RGB-D saliency detection. For fair comparison, we implement an ABP-based model [22], a GAN-based model [20], and VAE-based model [31, 56] using the same transformer- based generator as ours. The latent variables in these models are still assumed to follow the isotropic Gaussian distribution, as in their original algorithms. To be specific, for the ABP-based model, we use MCMC- -baee inference while training. and sample the latent variables directly from the Gaussian distribution during testing. For the GAN-based alternative, discriminator [26] that consists of five 3 x 3 convolutional layers with a stride of in each layer. The discriminator takes the concatenation of an image and a saliency map input, and is trained to distinguish between the predicted saliency map and the ground truth given an image. The numbers of the output channels of the discriminator introduce an extra encoder as an approximate Gaussian inference model via the reparameterization trick. The encoder consists of four 4 convolutional layers with a stride of 2 in each layer and maps the concatenation of an image and a saliency map to feature maps of channel sizes 64, 64, 64 and 64 sequentially. After that, two fully connected layers are adopted to produce the mean and the standard deviation of the inference model. For all the three alternative generative models, we set the numbers of the dimension of the latent space the same as that in our model, which is d 32.
design fully convolutional
64, 64, 64, 64 and 1. For the VAE-based alternative,
Table 6: Performance of different generative models with transformer backbones for SOD.
NJUZK [29]
SSB [46]
DES [8]
NLPR [50]
SIP [15]
Methou 5. t Fa 1 Eg 1MM
8.
Fy
Eg
.M 1
S.
F2 1 Eg JM
S,,
Fg
Egg 1 M
4
S. 1 Fa
IM
ABP
915
.951
.910
.890
935
920
962
020
.900
.898
GAN
992
954
.93
9441
003
940
.944
969
934
.61
021
.901
.904
VAE
.988
.911
.955
.02
.944
.944
.947
.03 .94
.92
970
.017
.944
.914
.961
.220
.904
.906
.935
.038
Ours
.999
.924
.96
.028
916
.98
.95
.032
945
928
.91
.016
.988
.921
.966
.018
.966
998
.40
.037
As shown in Table 6, compared with the deterministic baseline "Ours-Swin-D" (in Table 3). the three generative frameworks in achieve better or comparable performance. Especially in the DES dataset [8], they achieve large performance improvements. Our model outperforms all these alternative generative solutions, showing the effectiveness of the informative EBM prior distribution used in our model.
8

---

4.5 Hyperparameter Analysis
The main hyperparameters in our framework include the number of Langevin steps K, the Langevin step size 6, the number of dimensions of the latent space d, and the size of EBM prior C.. We have two sets of hyperparameters (o_ } and {ō+ K*) } of the I Langevin dynamics for sampling from the prior distribution and the posterior distribution, respectively. As the Langevin step size, find stable model performance with 8 E [0.2, 0.6] and 8+ E [0.05, 0.3), and we set 8 0.4 and 8+ 0.1 in our paper. For the number of Langevin steps, we empirically set K = K 5 to achieve a trade-off' between the training efficiency and the model performance, as more Langevin steps will lead longer training time but more convergent inference results. Additionally, we investigate the influence of the number of latent dimensions by varying d = {8, 16, 32, 64), and observe comparable performance among different choices of d. We set d = 32 our paper. We also investigate the influence of the EBM size by varying Ce = {200, 60, 100), and show the results in Table 7, in which we find C. 60 can provide optimal saliency prediction performance.
Table 7: Influence of the size of the EBM prior model
NJU2K [29]
SSB [46]
DES [8]
NLPR [50]
S1P[15]
Method
S. 1 F4 g5 1 M
S. 1 Fg 1 Eg 1 MM
9.
Fg
1
Eg
M
5m 1
Fg 1 Egg
1M
4
S,
Fig Egg TM
C. 20
.930
.913
.957
.06
.917
.898
950
.030
.951
.91
970
.016
.977
913
.950
.022
901
.892
.931
.037
C. 100
.926
.904
.950
.031
.916
.891 .952 .01
.942
.930
.90
.016
.939
.93
.960
.021
900
.895
.934
.036
Ours (C. 60)
929
.924
.956
022
.966
.898
950
.03
945
928
.91
.016
938
.911
.966
.018
.906
.908
.940
.037
4.6 Explainability Analysis
As a generative model, our framework is capable of obtaining a meaningful uncertainty map that summarizes the stochastic behavior of the model in performing saliency prediction. The uncertainty in our paper refers to the stochastic property of generating the saliency prediction from an input image, which is captured by the probabilistic model pa (=[I). In general, generative saliency prediction method can provide not only accurate predictions but also reasonable uncertainty maps that represent the "subjective nature" of the human visual saliency. In this section, we propose to use the uncertainty map to help qualitatively evaluate different generative saliency prediction frameworks. Specifically, we propose to compute the uncertainty map as the variance of multiple saliency predictions produced from the learned probabilistic model. In the experiment, for each input image, we first output ten saliency maps by using the Langevin sampling from the learned conditional distribution, and then compute the variance map (uncertainty) based on the generated saliency predictions.
To quantify the complexity of color image, we calculate a complexity score by the following way: (i) we first over-segment the image with the SLIC [1] to obtain 200 superpixels, (ii) then we compute the similarity of each superpixel with the others using handcrafted features from [99]. which gives us a 200 300-dimensional contrast vector, representing an overall contrast of the image, (iii) we define the complexity of the image as the mean entropy of the contrast vector. The higher the score, the more complicated the image. In general, reasonable generative saliency prediction model should have more confident predictions on images with simpler backgrounds (i.e., lower complexity scores), and less confident predictions on images with complicated backgrounds (i.e., higher complexity scores).
In Figure 4, we compare the uncertainty maps of different generative models that we have presented in Section 4.4. For each row of Figure 4, we display an input image, which is tagged with an image complexity score, and the associated ground truth saliency map, followed by the predicted saliency maps and the uncertainty maps obtained by the ABP-hased model, the GAN-based model, the VAE-based model, and our model, respectively. As shown in Figure 4, the complexity score of each image is reasonable in the sense that it is consistent with the human perception. We observe that the uncertainty maps obtained by the other generative models fail to reflect the difficulties of the images for saliency prediction. For example, the image shown in the first row of Figure 4 has a structured foreground (i.e.. a temple) and textured background (i.e., a forest), which leads to a large ambiguity of the boundary between the foreground and the background. The uncertainty map obtained by our model indicates a big variance around the boundary region, which demonstrates the explainability and the reasonability of our model. Note that only generative saliency prediction frameworks can provide such an explainability analysis based on the uncertainty maps. This might motivate us to develop generative frameworks for explainable saliency prediction in the future.

---

0.20
0.17
0.14
0.07
0.04
0.02
Image
GT
ABP
GAN
VAE
Ours
Figure 4: A comparison of uncertainty maps obtained by different generative saliency prediction frameworks for explainability analysis. Each row represents one example, in which we display an image tagged with a complexity score, the corresponding ground truth saliency map, as well as the predicted saliency maps and the uncertainty maps obtained by different generative frameworks.
Conclusion and Discussion
In this paper, we study the generative modeling and learning of vision transformer in the context of RGB and RGB-D salient object detections. We start from defining conditional probability distribution of saliency map given an input image by a top-down latent variable generative framework. in which the non-linear mapping from image domain to saliency domain is parameterized by proposed vision transformer network and the prior distribution of the low-dimensional latent space is represented by a trainable masee model. Instead of using amortized inference and sampling strategies, we learn the model by the MCMC-based maximum likelihood, where the Langevin sampling is used to evaluate the intractable posterior and prior distributions of the latent variables calculating the learning gradients of the model parameters. With the informative energy-based prior and the expressive top-down vision transformer network, our model can achieve both accurate predictions and meaningful uncertainty maps that are consistent with the human perception.
From the machine learning perspective, our model is a likelihood based top-down deep conditional generative model, which neither CGAN-based nor CVAE-based frameworks, and it is trained without relying on any assisting network. The learning algorithm derived from the proposed model based on MCMC inference for the posterior and MCMC sampling for the prior, which makes our framework more natural, principled, and statistically rigorous than others. The MCMC-based inference immediately available the sense that there is nothing to worry about the non non-trivial design and training of separate inference model in VAEs. Such framework is not only useful for saliency prediction (Though this is what we target in this paper) but also applicable to a vast of conditional learning scenarios, such as semantic segmentation, image-to-ing to-image translation, etc. Thus, the proposed generative model and the learning algorithm are generic and universal.
From the computer vision perspective, our model with a special network design to handle saliency prediction new member of the family of saliency prediction methods. In comparison with the traditional discriminative saliency prediction methods, our generative method is natural and reasonable because it models the saliency prediction as a conditional probability distribution. Moreover, demonstrates impressive performance over all RGB and RGB-D SOD benchmarks. As we know, the computer vision community has started to develop vision transformer networks for various computer vision tasks, such as image classification, segmentation, detection, generation, etc. Our paper is the first one to present a generative vision transformer for both RGB and RGB-D saliency predictions. Thus, the proposed framework is significantly important for the computer vision community.
10

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurélien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels. Technical report, EPFL, 06 2010.
[2] Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. In International Conference on Learning Representations (ICLR), 2018.
[3] Christian F. Baumgartner, Kerem Can Tezcan, Krishna Chaitanya, Andreas M. Hötker, Urs J. Muehlematter, Khoschy Schawkat, Anton S. Becker, Olivio Donati, and Ender Konukoglu. Phiseg:
Capturing uncertainty in medical image segmentation. In The 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), pages
119-127, 2019.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision (ECCV), pages 213-229, 2020.
[5] Hao Chen and Youfu Li. Progressively complementarity aware fusion network for RGB-D salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3051-3066 2018.
[6] Shuo Chen, Tan Yu, and Ping Li. Mvt: Multi-view vision transformer for 3d object recognition. In British Machine Vision Conference (BMVC), 2021.
[7] Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, and Shi-Min Hu. Global contrast based salient region detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 37(3):569-555 582, 2015.
[8] Yupeng Cheng, Huazhu Fu, Xingxing Wei, Jiangjian Xiao, and Xiaochun Cao. Depth enhanced saliency detection method. In Proceedings of International Conference on Internet Multimedia Computing and Service, pages 23-27, 2014.
[9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 248-255,,20 2009.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[11] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Conference on Neural Information Processing Systems (NeurIPS), pages 3603-3613, 2019.
[12] Kumar Avinava Dubey, Sashank J. Reddi, Sinead A. Williamson, Barnabás Póczos, Alexander J. Smola, and Eric P. Xing. Variance reduction in stochastic gradient langevin dynamics. In Conference on Neural Information Processing Systems (NIPS), pages 1154-1162, 2016.
[13] Deng-Ping Fan, Ming-Ming Cheng, Yun Liu, Tao Li, and Ali Borji. Structure-measure: A new way to evaluate foreground maps. In IEEE International Conference Computer Vision (ICCV), pages 4554-4557, 2017.
[14] Deng-Ping Fan, Cheng Gong, Yang Cao, Bo Ren, Ming-Ming Cheng, and Ali Borji. Enhanced- alignment measure for binary foreground map evaluation. In International Joint Conference Artificial Intelligence (IJCAT), pages 698-704, 2018.
[15] Deng- Deng-Ping Fan, Zheng Lin, Zhao Zhang, Menglong Zhu, and Ming-Ming Cheng. Rethinking RGB-D Salient Object Detection: Models. Datasets, and Large-Scale Benchmarks. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2020.
[16) Deng-Ping Fan, Yingje Zhai, Ali Borji, Jufeng Yang, and Ling Shao. BBS-Net: RGB-D salient object detection with a bifurcated backbone strategy network. In European Conference on Computer Vision (ECCV), pages 275-292, 2020.

---

[17] Keren Fu, Deng- Deng-Ping Fan, Ge-Peng Ji, and Qijun Zhao. JL-DCF: Joint learning and densely- cooperative fusion framework for RGB-D salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3052-3062, 2020.
[18] Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative convnets via multi-grid modeling and sampling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9155-9164 2018.
[19] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning energy- based models by diffusion recovery likelihood. In International Conference Learning Representations (ICLR), 2021.
[20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Conference Neural Information Processing Systems (NIPS), pages 2672-2680, 2014.
[21] Rick Groenendijk, Sezer Karaoglu, Theo Gevers, and Thomas Mensink. On the benefit of adversarial training for monocular depth estimation. Computer Vision and Image Understanding, 190: 102848, 2020.
[22] Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating back-propagation for generator network. In AAAI Conference on Artificial Intelligence (AAAT), page 1976-1984, 2017.
[23] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatricke Lagging inference networks and posterior collapse in variational autoencoders. In International Conference on Learning Representations (ICLR), 2019.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEÈE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[25] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 160iv 08415, 2016.
[26] Wei-Chih Hung. Yi-Hsuan Tsai, Yan-Ting Liou, Yen- Yu Lin, and Ming- Husun Yang. Adversar- ial learning for semi-supervised semantic segmentation. In British Machine Vision Conference (BMVC), 2018.
[27] Wei Ji, Jingjing Li. Shuang Yu, Miao Zhang, Yongri Piao, Shunyu Yao, Qi Bi, Kai Ma, Yefeng Zheng, Huchuan Lu, and Li Cheng. Calibrated rgb-d salient object detection. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 9471-9481, 2021.
[28] Wei Ji, Jingjing Li, Miao Zhang, Yongri Piao, and Huchuan Lu. Accurate RGB-D salient object detection via collaborative learning. In European Conference on Computer Vision (ECCV), 2020.
[29] Ran Ju, Yang Liu, Tongwei Ren, Ling Ge, and Gangshan Wu. Depth-aware salient object detec- tion using anisotropic sentruudu difference. Signal Processing: Image Communication, 38:1 15-115-- 126, 2015.
[30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[31] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
[32] Simon Kohl, Bernardino Comeee-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R. Ledsam, Klaus Maier-Hein, S. M. Ali Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In Conference on Neural Information Processing Systems (NeurIPS), pages 6965-6975. 2018.
[33] Quang H. Le, Kamal Youcef-Toumi, Dzmitry Tsetserukou, and Ali Jahanian. Gan mask r-cnncinstance semantic segmentation benefits from generative adversarial networks. arXiv preprint 27X1v:2010.13757, 2020.
12

---

[34] Bo Li, Zhengxing Sun, and Yuqi Guo. Supervae: Superpixelwise variational autoencoder for salient object detection. In AAAI Conference on Artificial Intelligence (AAAI), pages 8569-8576, 2019.
[35] Guanbin Li and Yizhou Yu. Visual saliency based on multiscale deep features. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5455-5463, 2015.
[36] Yin Li, Xiaodi Hou, Christof Koch, James M Rehg, and Alan L Yuille. The secrets of salient object segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 280-287, 2014.
[37] liang-Jiang Liu, Qibin Hou, Ming-Ming Cheng, Jiashi Feng, and Jianmin Jiang. simple pooling- -ased design for real-time salient object detection. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[38] Jun Liu. Monte Carlo strategies in scientific computing. Springer Science & Business Media, 2008.
[39] Nian Liu, Ni Zhang, Kaiyuan Wan, Ling Shao, and Junwei Han. Visual saliency transformer. In IEEE International Conference on Computer Vision (ICCV), pages 4722-4732, 2021.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE International Conference on Computer Vision (ICCV), 2021.
[41] Zhiming Luo, Akshaya Mishra, Andrew Achkar, Justin Eichel, Shaozi Li, and Pierre-Mare Jodoin. Non-local deep features for salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[42] Weian Mao, Yongtao Ge, Chunhua Shen, Zhi Tian, Xinlong Wang, and Zhibin Wang. Tfpose: Direct human pose estimation with transformers. arXiv preprint arXiv:2103. 153200 2021.
[43] Radford M Neal. MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(1 ():22 2011.
[44] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Conference on Neural Information Processing Systems (NeurIPS), pages 5233-5243, 2019.
[45] Erik Nijkamp, Bo Pang. Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning multi-layer latent variable model via variational optimization of short run meme for approximate inference. In European Conference on Computer Vision (ECCV), pages 361-378. Springer, 2020.
[46] Yuzhen Niu, Yujie Geng, Xueqing Li, and Feng Liu. Leveraging stereopsis for saliency analysis. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 454 461, 2012.
[47] Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-base based prior model. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[48] Bo Pang and Ying Nian Wu. Latent space energy- model of symbol-vector coupling for text generation and classification. In International Conference on Machine Learning (ICML), pages 8359-8370. 2021.
[49] Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 11814 1824, 2021.
[50] Houwen Peng, Bing Li, Weihua Xiong, Weiming Hu, and Rongrong Ji. RGBD salient object detection: A benchmark and algorithms. In European Conference on Computer Vision (ECCV), pages 92- 2014.
13

---

[51] Yongri Piao, Wei Ji, Jingjing Li, Miao Zhang, and Huchuan Lu. Depth-induce multi-scale recurrent attention network for saliency detection. In IEEE International Conference on Computer Vision (ICCV), pages 7254-7263, 2019.
[52] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary-aware salient object detection. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 7479-7489, 2019.
[53] Liangqiong Qu, Shengfeng He, Jiawei Zhang, Jiandong Tian, Yandong Tang, and Qingxiong Yang. RGBD salient object detection via deep fusion. IEEE Transactions on Image Processing (T1)), 2285, 2017.
[54] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE International Conference on Computer Vision (ICCV), pages 22179-12188) 2021.
[55] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[56] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Conference on Neural Information Processing Systems (NIPS), pages 3483-3491, 2015.
[57] Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. In IEEE International Conference on Computer Vision (ICCV), pages 5689-5697, 2017.
[58] Lucas Stoffl, Maxime Vidal, and Alexander Mathis. End-to-end trainable multi-instance pose estimation with transformers. arXiv preprint arXiv:2103./2115 2021.
[59] Peng Sun, Wenhu Zhang, Huanyu Wang, Songyuan Li, and Xi Li. Deep rgb-d saliency detection with depth-sensitive attention and automatic multi-modal fusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1407-1417, 2021.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems (NIPS), pages 5998-6008, 2017.
[61] Bo Wang, Quan Chen, Min Zhou, Zhiqiang Zhang, Xiaogang Jin, and Kun Gai. Progressive fea- ture polishing network for salient object detection. In AAAI Conference on Artificial Intelligence (AAAI), pages 12135- 2020.
[62] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 136-145, 2017.
[63] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In IEEE International Conference on Computer Vision (ICCV), pages 568 168-578,22 2021.
[64] Jun Wei, Shuhui Wang, and Qingming Huang. Fnet: Fusion, feedback and focus for salient object detection. In AAAI Conference on Artificial Intelligence (AAAI), pages 12321-12328, 2020.
[65] Jun Wei, Shuhui Wang, Zhe Wu, Chi Su, Qingming Huang, and Qi Tian. Label decoupling framework for salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13025-13034, 2020.
[66] Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML), pages 681 688- 2011.
[67] Zhe Wu, Li Su, and Qingming Huang. Cascaded partial decoder for fast and accurate salient object detection. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3902-3911, 2019.
14

---

[68] Zhe Wu, Li Su, and Qingming Huang. Stacked cross refinmmnt network for edge-aware salient object detection. In IEEE International Conference on Computer Vision (ICCV), pages 2019.
[69] Jianwen Xie, Ruiqi Gao, Zilong Zheng. Song- Chun Zhu, and Ying Nian Wu. Learning dynamic generator model by alternating back- paok-propagation through time. In AAAI Conference on Artificial Intelligence (AAAI), pages (498-5507 2019.
[70] Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. In AAAI Conference on Artificial Intelligence (AAAI), pages 4292-4301 2018.
[71] Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of descriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 42(1):27 2018.
[72] Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning (ICML), pages 2635 2644, 2016.
[73] Jianwen Xie, Yifei Xu, Zilong Zheng. Song-Chun Chgn Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-basee based learning on unordered point sets for 3d generation, reconstruction and classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14985, 2021.
[74] Jianwen Xie, Zilong Zheng, Xiaolin Fang, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of fast thinking initializer and slow thinking solver for conditional learning. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
[75] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song- Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 8629-8638, 2018.
[76] Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang. Song- Long-Chun Zhu, and Ying Nian Wu. Generative voxelnet: Learning energy-based models for 3d shape synthesis and analysis. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2020.
[77] Jianwen Xie, Zilong Zheng, and Ping Li. Learning energy- based model with variational auto- encoder as amortized sampler. In AAAI Conference on Artificial Intelligence (AAAI), pages 10441-10451, 2021.
[78] Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1061-1069, 2017.
[79] Jianwen Xie, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based spatial-temporal generative convnets for dynamic patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 43(2):516-531, 2021.
[80] Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Alameda- Pineda. Transcenter: Transformers with dense queries for multiple-object tracking. arXiv preprint arXiv:2103. 15145, 2021.
[81] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In IEEE International Conference on Computer Vision (ICCV), pages 10448-10457, 2021.
[82] Qiong Yan, Li Xu, Jianping Shi, and Jiaya Jia. Hierarchical saliency detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1155-1162, 2013.
[83] Chuan Yang, Lihe Zhang, Huchuan Lu, Xiang Ruan, and Ming-Hsuan Yang. Saliency detection via graph-based manifold ranking. IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3166-3173, 2013.
15

---

[84] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Sadat Saleh, Sadegh Aliak- barian, and Nick Barnes. Uncertainty inspired rgb-d saliency detection. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 2021.
[85] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Saeed Anwar, Fatemeh Sadat Saleh, Tong Zhang, and Nick Barnes. Uc-net: Uncertainty inspired rgb-d saliency detection via conditional variational autoencoders. In IEEE Conference on Computer Vision and Pultern Recognition (CVPR), pages 8582 8591, 2020.
[86] Jing Zhang, Deng-Ping Fan, Yuchao Dai, Xin Yu, Yiran Zhong, Nick Barnes, and Ling Shao. Rgb-d saliency detection via cascaded mutual information minimization. In IEEE International Conference on Computer Vision (ICCV), pages 4338-43477 2021.
[87] Jing Zhang, Jianwen Xie, and Nick Barnes. Learning noise-aware encoder-decoder from noisy labels by alternating back-propagation for saliency detection. In European Conference Computer Vision (ECCV), pages 349-366, 2020.
[88] Jing Zhang, Jianwen Xie, Zilong Zheng, and Nick Barnes. Energy-based generative cooperative saliency prediction. arXiv preprint
13389, 2021.
[89] Jing Zhang, Xin Yu, Aixuan Li. Peipei Song. Bowen Liu, and Yuchao Dai. Weakly-supervine salient object detection via scribble annotations. In IEEE Conference on Computer Vision and Pattemn Recognition (CVPR), pages 12543-12552 2020.
[90] Miao Zhang, Weisong Ren, Yongri Piao, Zhengkun Rong, and Huchuan Lu. Select, supplement and focus for RGB-D saliency detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2020.
[91] Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In European Conference on Computer Vision (ECCV), pages 286-301,2011
[92] Zhao Zhang, Zheng Lin, Jun Xu, Wenda Jin, Shao-Ping Lu, and Deng-Ping Fan. Bilateral attention network for RGB-D salient object detection. IEEE Transactions on Image Processing (T1P), 30:19999911. 2021.
[93] Jia-Xing Zhao, Yang Cao, Deng-Ping Fan, Ming-Ming Cheng, Xuan-Yi Li, and Le Zhang. Contrast prior and fluid pyramid integration for RGBD salient object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3927-3936, 2019.
[94] Jia-Xing Zhao, Jiang-Jiang Liu, Deng-Ping Fan, Yang Cao, Jufeng Yang, and Ming-Ming Cheng. Egneteddge guidance network for salient object detection. In IEEE International Conference on Computer Vision (ICCV), pages 8778-8787. 2019.
[95] Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based generative models via coarse-to- fine expanding and sampling. In International Conference on Learning Representations (ICLR), 2021.
[96] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, and Li Zhang. Rethinking semantic segmentation from
perspective with transformers. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6881-6890, 2021.
[97] Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy-based models from a single natural image for internal learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2961-2970, 2021.
[98] Huajun Zhou, Xiaohua Xie, Jian-Huang Lai, Zixuan Chen, and Lingxiao Yang. Interactive two-stream decoder for accurate and fast saliency detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1151-915,, 2020.
[99] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2814-2821, 2014.
16


Reviews:
[{"review_id": "xzb2u2WmxV", "reviewer": "Reviewer_d7vK", "summary": "The submission proposes a method for saliency detection based on a generative adversary learning framework, which uses a transformer architecture to replace the backbone of the framework and is jointly trained with a Markov chain Monte Carlo-based maximum likelihood estimation to improve the performance. The performance of the proposed method is evaluated on both RGBD- and RGB-based saliency prediction datasets. The paper is generally written well and easy to follow.\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the overall performance of the proposed method is promising, the motivation, the contribution, and the effectiveness of the method are not very clearly illustrated and demonstrated. The reviewer has some detailed comments below:\n\n(i)\tThe motivation of using a GAN-based method for saliency detection is not that clear. Does it really outperform the convolutional-based discriminative models for the task? From the results, the reviewer also cannot see any performance comparison to show the advantages of using a GAN-based method as a baseline. \n\n(ii)\tIt is not clear that if the contribution is an improvement of GAN-based model with a transformer, or a method that advances saliency detection. If the focus of the paper is a new GAN-based method with transformer, and saliency detection is just an application, the authors should illustrate more differences to existing works that using GAN with transformers. \n\n(iii)\tThe effectiveness of energy-based prior distribution seems not demonstrated in the experimental results. The whole ablation study seems to be very limited. \n\n(iv)\tThe authors claimed state-of-the-art performance on saliency detection. However, the comparison with previous sota methods is significantly not enough. To support that claim, the authors need to compare more methods on more challenging datasets.  \n\n(v)\tFor the presentation, an overall framework depiction could help the readers to understand the whole thing more easily. It is also important to elaborate more, probably with a figure as well, on how the proposed energy-based strategy can be jointly optimized with the network.\n\n(vi)\tThe authors showed some visualization of the learned uncertainties, but the reviewer is not clear that why the uncertainty is critical in the method. More explanation on the motivation of the uncertainty is definitely helpful, and corresponding experimental results to show the benefits of the uncertainty are also needed. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The motivation and contribution are not clear, and the experimental results and ablation study are not sufficient. Please check the main review part for more detailed comments. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "jjgWYyByRlR", "reviewer": "Reviewer_xC44", "summary": "The author construct a generative model for salient object detection in the form of top-down conditional latent variable model.\nA generative vision transformer is used by adding latent variables into the traditional deterministic transformer. Experimental results on RGB and RGB-D image salient object detection show that the generative framework equipped with the EBM prior and the transformer-based non-linear mapping is powerful in representing the conditional distribution of object saliency given an image.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper exhibits a generative vision transformer for salient object detection. The idea using generative vision transformer to address the salient object detection is creative. The extensive experiments also show that it achieves the SOTA performance. \n\nMy concerns and suggestions about this paper is that many details are missing.\n1. Figure 1 presents the uncertainty map, but there is no introduction presented as it is important in this paper. what is the uncertainty map, how to get it?\n2. In line 51, EBM occurs at the first time, but there is no introduction about what is EBM in the whole paper. \n3. In line 122, ''energy-based prior, which is presented in Eq.(1)'', which one term is the energy based prior? In addition, in some places, energy-based prior is mentioned or energy-based prior model. Does the two mean the same?\n4. In line 162-166, the energy-based prior model is introduced, but it is not shown in Fig2. How does the transformer and the prior model work together? it should be explained in more detail.\n5. For equation 7 and 8, it uses z_t and z_(t+1), but in the algorithm 1, step 4, 5, z_{t}^{-} and z_{t}^{+} are used. It's ambiguous to differentiate them.\n6. In line 194-198, computation and memory cost are introduced. But there is no detailed comparison given in the paper. I think the comparison between the proposed and previous SOTA for computation time and memory should be detailed.\n7. The performance comparison should include transformer based method, like recent work \"P2T: Pyramid Pooling Transformer\nfor Scene Understanding\".\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The suggestions are presented in Main Review.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical conerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "W472DApi-dh", "reviewer": "Reviewer_Re5H", "summary": "This paper proposes a method based on generative vision transformer with latent variables \nfollowing an informative energy-based prior for salient object detection.\nBoth the vision transformer network and the energy based prior model are jointly trained \nvia Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation.\n\nExtensive experiments reveal both high performance model and meaningful confidence map scores,\nas well as estimation of uncertainty maps.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nNot assuming the prior of the GAN to be Gaussian, and instead using an energy based NL function is a good start in the model.\nMost the analytics is meaningful and well drafted.\n\nThe proposed model of learning by MCMC-based maximum likelihood where the Langevin sampling is used to evaluate the intractable posterior and prior distributions of the latent variables for calculating the gradients of the log-likelihood w.r.t. the model parameters, is quite interesting. The authors however do not claim it to be novel, wondering if a variant exists somewhere in literature ?\n\nThe tables 1-5 with performance measures, may also be put as bar-charts in Supple material, as its often hard to compare enhancement of your performance wrt nearest SOA, from decimal places in mind.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nHow do you validate the estimated uncertainty maps, in the absence of GTs for the same ?\n\nWonder what is the expansion of the acronym \"D_CBase\"  used ? What does it mean ?\n\nWhat is MAE in fig. 4 ? Not defined anywhere in main doc. - got it in Supple though.\n\nFig. 9 of Supple material - how do you validate these complexity scores ? Even the GT or human evaluation is not provided.\nWhat are the typical ranges of complexity values you get by the entropy based measure ?\nHow do you thus claim it to be consistent with human visual perception ?\n\nSome typos:\nline 169: ......local minimum, it solve the following estimating.... - change to : \nit solves the....\n\nSome more references may be added as: \n1. Bayesian Learning via Stochastic Gradient Langevin Dynamics; Max Welling, Yee Whye Teh, ICML 2011;\n\n2. Variance Reduction in Stochastic Gradient Langevin Dynamics; Avinava Dubey, Sashank J. Reddi, Barnabas P´oczos, Alexander J. Smola, Eric P. Xing, NIPS 2016.\n\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10-12 Hrs"}, "extra_text": {}}, {"review_id": "T2OmCPQN32E", "reviewer": "Reviewer_6hfU", "summary": "This paper introduces a novel* supervised generative model for detecting salient image regions, where saliency is defined as containing object. This is a latent-variable model with an energy-based prior and a transformer decoder that maps the latent variable to a saliency map. The prior and posterior inference is done via SGLD. The model is well-evaluated on several datasets, with several metrics, and compared across a range of state-of-the-art baselines. It not only outperforms every single baseline (setting new SOTA), but also has the added benefit of providing high-quality uncertainty estimates. Detailed ablation studies show the relative importance of various model components.\n\n\\* I think it is novel, but difficult to say as I am not expert in the field and there is very information in the paper about prior art.\n\n# Update\nI'm happy with the author response and am increasing my score to 7. I still have some doubts about the motivation of this work, but the theory is sound and the the results are very good.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- The task is well-established (silent object detection, or SOD for short. This is at least as far as I can tell, given the number of baselines.\n- It is not clear how this work differs from prior art. Even though there are numerous baselines in the experiments, they are not discussed. The related work section lists a huge number of related works but does not discuss how these differ from the current work. Or rather, it does so only implicitly, stating only very shortly what some other methods do. Since I am not an expert in SOD, it is difficult for me to understand the differences to the current method with so little information.\n- This work is a combination of well-known techniques: energy-based generative modelling for the prior, SGLD for prior and posterior inference, a transformer for decoding. It is this combination and its application to SOD that, as far as I can tell (I am not an expert in the field), is novel.\n\nQuality:\n- The presented method and results are very good quality, but not very surprising: it is clear that a generative model can provide good uncertainty estimates. \n- The paper needs some writing work (more in clarity).\n\nGood:\n- The submission is technically sound. The details of the method are well-explained and seem correct.\n- An extensive experimental evaluation is provided. The method outperforms numerous baselines on several datasets.\n- The authors provide a short-but-sufficient analysis of the learning algorithm (with prior/posterior SGLD sampling).\n- There are numerous ablation studies evaluating importance of model components.\n- I really like the \"hyperparameter analysis\" section in L263-L276. It gathers and describes all hparams of the method and says how they were tuned and if the model was sensitive. Nice!\n\nBad:\n- Even though the model can provide some uncertainty estimates, it cannot estimate the uncertainty in the model parameters. The authors do not comment on that.\n- One of the main motivations of an EBM prior is that a correlated prior/posterior should do better than (conditionally-) independent ones as in a GAN or a VAE. While \"obvious\" on the surface, we have to remember that an independent prior + a single MLP layer (not to mention more) can lead to a highly-correlated prior, cf. normalizing flows. Therefore, I would say that his motivation is invalid---despite the fact that the authors instantiate GAN and VAE variants of their model and show they underperform the EBM. \n- I am not convinced that Eq. 4 is fully correct. I tried deriving it, but I only managed to get the expectation of the gradient with respect to the model distribution of (latent, saliency), but training is done with ground-truth saliency and posterior-sampled latents. Could you please walk me through the derivation?\n- In L198 you say that your method is more memory-efficient than e.g. VAE because there is no encoder. However, for SGLD, you need to evaluate the prior and the decoder multiple times. Do you backprop through SGLD? If yes, then you need to store activations of every prior/decoder evaluation, which renders your method *very* memory-inefficient. Care to elaborate?\n\nClarity: \n- This submission would benefit from some more writing work. It is quite unclear in places, see comments below. \n- There is a sufficient level of detail to reimplement the model, provided that the reader can understand the paper.\n- There are a lot of typos, e.g. L44 none->non, L75 modal->modality, L226 & L278 analysis -> analyse.\n- \"SOD\" is first used in L65 but not explained; I needed to infer that it means \"salient object detection\". Similar \"saliency prediction problem\" appears in L103 without definition. The paper does not state until very far into the text that it tackles supervised learning.\n- The related works section is almost unreadable. It lists tens of different works without giving any differences between them, nor specifying how they relate to the current work.\n- Since prior/posterior samples are defined with + and - signs, it would be nice to define the related hyperparams in Alg 1 using + and - as well (instead of 0 and 1).\n- L192 \"and other newly added layers randomly initialized\" <- I've no idea what this refers to.\n- It is very unclear that Tab. 3 lists deterministic baselines; perhaps add that in the caption? and also mention in Tab. 4 caption that a deterministic variant of \"ours\" is available in Tab. 3. The paragraph describing the deterministic backbone study (L227-238) is very difficult to read and the fact that the model is now deterministic is highly obscured.\n- Unclear what \"scales\" in Fig. 4 means.\n- Unclear what 0/1 as supervision in L251 means.\n- Fig. 5 is difficult to read; please provide an interpretation.\n- Please add a ylabel to Fig. 6.\n\n\nSignificance: \n- While it is nice to see an application of energy-based generative modelling to supervised learning, I am not sure how significant this is. I am not an expert in this field, and I have no idea where SOD is used and what are the benefits as compared to e.g. object detection or instance/panoptic segmentation.\n- Given that there is a number of SOD baselines in the paper, I can be convinced that SOD is an important research area. Since the presented results are quite a bit better than the baselines, and the method is simple, it is likely that it will be built upon. \n\nI would be happy to increase my score, potentially quite significantly, if the authors address my concerns above and clearly delineate themselves from prior art.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequate discussion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a novel transformer-based latent variable model for salient object detection. Reviewers were generally positive on the approach and results, and the work was deemed to be technically sound. The reviews and authors had a productive discussion which we hope will lead to some clarifications that will enhance the paper. One reviewer provided a score below the threshold for acceptance, but this reviewer appears to have had some confusion about the nature of the method and did not reengage with authors when they sought to clarify; as such, this review was down-weighted in the overall evaluation. All in all, this is an interesting method producing state of the art results, with solid theoretical grounding.
Decision: Accept (Poster)