Learning Generative Vision Transformer with Energy-Based Latent Space for Saliency Prediction
Year: 2021

Abstract:
Vision transformer networks have shown superiority in many computer vision tasks. In this paper, we take a step further by proposing a novel generative vision transformer with latent variables following an informative energy-based prior for salient object detection. Both the vision transformer network and the energy-based prior model are jointly trained via Markov chain Monte Carlo-based maximum likelihood estimation, in which the sampling from the intractable posterior and prior distributions of the latent variables are performed by Langevin dynamics. Further, with the generative vision transformer, we can easily obtain a pixel-wise uncertainty map from an image, which indicates the model confidence in predicting saliency from the image. Different from the existing generative models which define the prior distribution of the latent variables as a simple isotropic Gaussian distribution, our model uses an energy-based informative prior which can be more expressive to capture the latent space of the data. We apply the proposed framework to both RGB and RGB-D salient object detection tasks. Extensive experimental results show that our framework can achieve not only accurate saliency predictions but also meaningful uncertainty maps that are consistent with the human perception.

heading: 1 Introduction
text: In the field of computer vision, salient object detection [64, 65, 16, 17, 5, 89] (SOD) or visual saliency prediction, which aims at highlighting objects more attentive than the surrounding areas in images, has achieved significant performance improvement with the deep convolutional neural network revolution. Given a set of training images along with their saliency annotations, the conventional SOD models seek to learn a deterministic one-to-one mapping from image domain to saliency domain.
Two main issues exist in the above conventional deep saliency prediction framework: (i) the convolution operation based on sliding window makes the deep saliency prediction model less effective in modeling the global contrast of the image, which is essential for salient object detection [7]; (ii) the one-to-one deterministic mapping mechanism makes the current framework not only impossible to represent the pixel-wise uncertainty in predicting salient objects, but also hard to handle incomplete data in a weakly supervised scenario [89]. Besides, given an image, the saliency output of a human is subjective, therefore, a stochastic generative model is more natural than a deterministic model for representing saliency prediction. Although [85] introduces a conditional variational autoencoder (CVAE) [56] for RGB-D salient object detection, the potential posterior collapse problem [23] makes the stochastic predictions less effective in generating meaningful uncertainty estimation.
Motivated by the above two issues, we propose a novel framework, the generative vision transformer, for salient object detection, where a vision transformer structure [40] is used as a backbone and latent variables are introduced in designing our generative framework. On the one hand, transformers [60] have proven to be very effective in long-range dependency modeling, and are capable of modeling
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
various scopes of object context information with the multi-head self-attention module. With such an architecture, we can achieve global context modeling for effective salient object detection. On the other hand, the latent variables account for randomness and uncertainty in modeling the mapping from image domain to saliency domain, and also enable the model to produce stochastic saliency predictions for uncertainty estimation. Therefore, the proposed model is a latent variable transformer.
Nowadays, there are two types of generative models that have been widely used, namely the variational autoencoder (VAE) [31] and the generative adversarial network (GAN) [20], which correspond to two different generative learning strategies to train latent variable models. To train a top-down latent variable generator, the VAE introduces an extra encoder to approximate the intractable posterior distribution of the latent variables, and trains the generator via a perturbation of maximum likelihood; while the GAN introduces a discriminator to distinguish between generated samples and real data, and trains the generator to fool the discriminator. [22, 69] present the third learning strategy, namely alternating back-propagation (ABP), to train the generator with latent variables being directly sampled from the true posterior distribution by using a gradient-based Markov chain Monte Carlo (MCMC) [38], e.g., Langevin dynamics [43, 66, 12]. All the three generative models define the prior distribution of the latent variables as a simple non-informative isotropic Gaussian distribution, which is less expressive in capturing meaningful latent representation of the data.
In this paper, we investigate generative modeling and learning of the vision transformer. We construct a generative model for salient object detection in the form of a top-down conditional latent variable model. Specifically, we propose a generative vision transformer by adding latent variables into the traditional deterministic vision transformer, and assume the latent variables follow an informative trainable energy-based prior distribution [47, 48]. Following [72], we parameterize the energy function of the energy-based model (EBM) by a deep net. Instead of using variational learning or adversarial learning, we jointly train the parameters of the EBM prior and the transformer network by maximum likelihood estimation (MLE). The MLE algorithm relies on MCMC sampling to evaluate the intractable prior and posterior distributions of the latent variables.
Experimental results on RGB and RGB-D salient object detections [64, 68, 85, 16] show that the generative framework equipped with the EBM prior and the transformer-based non-linear mapping is powerful in representing the conditional distribution of object saliency given an image, leading to more reasonable uncertainty estimation as shown in Figure 1, where stochastic saliency prediction is provided by a learned model and the visualization of the pixel-wise uncertainty is presented.
We summarize our main contributions and novelties as follows: (i) we propose a novel top-down generative vision transformer network with an energy-based prior distribution defined on latent space for salient object detection; (ii) we jointly train the vision transformer network and the energy-based prior model by an MCMC-based maximum likelihood estimation, without relying on any extra assisting network for adversarial learning or variational learning; (iii) we achieve new benchmark results for both RGB and RGB-D salient object detections, and obtain meaningful uncertainty maps that are highly consistent with human perception for saliency prediction.

heading: 2 Related Work
text: Salient object detection: The main goal of the existing deep fully-supervised RGB image-based salient object detection models [67, 41, 68, 64, 61, 87, 65] is to achieve structure preserving saliency prediction, by either sophisticated feature aggregation [67, 61], auxiliary edge detection [68, 52, 65], or resorting to structure-aware loss functions [41, 64]. With extra depth information, RGB-D salient object detection models [53, 85, 5, 27, 93, 90, 16, 28, 51, 92, 59, 86] mainly focus on effective multi-modal modeling. Our paper solves the same problems, i.e., RGB and RGB-D salient object detection, by developing a new generative transformer-based framework.
Vision transformers: The breakthroughs of the Transformer networks [60] in natural language processing (NLP) domain have sparked the interest of the computer vision community in developing vision transformers for different computer vision tasks, such as image classification [10, 40], object detection [4, 63, 6, 40], image segmentation [96, 54, 63, 40], object tracking [80, 81], pose estimation [42, 58], etc. Among them, DPT [54] adopts a U-shape structure and uses ViT [10] as an encoder to perform semantic segmentation and monocular depth estimation. Swin [40] presents a hierarchical transformer with a shifted windowing scheme to achieve an efficient transformer network with high resolution images as input. Different from the above vision transformers that mainly focus on discriminative modeling and learning, our paper emphasizes generative modeling and learning of the vision transformer by involving latent variables and MCMC inference.
Dense prediction with generative models: VAEs have been successfully applied to image segmentation [3, 32]. For saliency prediction, [34] adopts a VAE to model the image background, and separates salient objects from the background through reconstruction residuals. [85, 84] design CVAEs to model the subjective nature of saliency. GAN-based methods can be divided into two categories, namely fully-supervised and semi-supervised settings. The former [21, 33] uses the discriminator to distinguish model predictions from ground truths, while the latter [57, 26] uses the GAN to explore the contribution of unlabeled data. [88] uses a cooperative learning framework [71, 74] for generative saliency prediction. [84] trains a single top-down generator in the ABP framework for RGB-D saliency prediction. Our model generalizes [84] by replacing the simple Gaussian prior by a learnable EBM prior and adopting a vision transformer-based generator for salient object prediction.
Energy-based models: Recent works have shown strong performance of data space EBMs [72, 44] in modeling high-dimensional complex dependencies, such as images [97, 95, 18, 11, 19], videos [78, 79], 3D shapes [75, 76], and point clouds [73], and also demonstrated the effectiveness of latent space EBMs [47] in improving the model expressivity for text [48], image [47], and trajectory [49] generation. Our paper also learns a latent space EBM as the prior model but builds the EBM on top of a vision transformer generator for image-conditioned saliency map prediction.

heading: 3 Generative Vision Transformer with Energy-Based Latent Space

heading: 3.1 Model
text: We formulate the supervised saliency prediction problem as a conditional generative learning problem. Let I ∈ Rh×w×3 be an observed RGB image, s ∈ Rh×w×1 be the saliency map, and z ∈ R1×1×d be the d-dimensional vector of latent variables, where h× w d. Consider the following generative model to predict a saliency map s from an image I,
s = Tθ(I, z) + , z ∼ pα(z), ∼ N (0, σ2 I), (1) where Tθ is the non-linear mapping process from [z, I] to s with parameters θ, pα(z) is the prior distribution with parameters α, and ∼ N (0, σ2 I) is the observation residual of saliency with σ being given. Due to the stochasticity of the latent variables z, given an image I, its saliency map is also stochastic. Such a probabilistic model is in accord with the uncertainty of the image saliency.
Following [47], the prior pα(z) is not assumed to be a simple isotropic Gaussian distribution as GAN [20], VAE [31, 56] or ABP [22]. Specifically, it is in the form of the energy-based correction or exponential tilting [72] of an isotropic Gaussian reference distribution p0(z) = N (0, σ2zI), i.e.,
pα(z) = 1
Z(α) exp [−Uα(z)] p0(z) ∝ exp
[ −Uα(z)− 1
2σ2z ||z||2
] , (2)
where Eα(z) = Uα(z) + 12σ2z ||z|| 2 is the energy function that maps the latent variables z to a scalar, and Uα(z) is parameterized by a multi-layer perceptron (MLP) with trainable parameters α. The standard deviation σz is a hyperparameter. Z(α) = ∫ exp[−Uα(z)]p0(z)dz is the intractable normalizing constant that resolves the requirement for a probability distribution to have a total probability equal to one. pα(z) is an informative prior distribution in our model and its parameters α need to be estimated along with the non-linear mapping function Tθ from the training data.
The mapping function Tθ is parameterized by a vision transformer [40] with self-attention mechanism, which encodes the input image I and then decodes it along with the vector of latent variables z to the saliency map s, thus, pθ(s|I, z) = N (Tθ(I, z), σ2 I). The resulting generative model is a conditional directed graphical model that combines the EBM prior [47] and the vision transformer [40].

heading: 3.2 Learning
text: The generative transformer with an energy-based prior, which is presented in Eq. (1), can be trained via maximum likelihood estimation. For notation simplicity, let β = (θ, α). For the training examples {(Ii, si), i = 1, ..., n}, the observed-data log-likelihood function is defined as
L(β) = n∑ i=1 log pβ(si|Ii) = n∑ i=1 log [∫ pβ(si, zi|Ii)dzi ] = n∑ i=1 log [∫ pα(zi)pθ(si|Ii, zi)dzi ] .
Maximizing L(β) is equivalent to minimizing the Kullback-Leibler (KL) divergence between the model pβ(s|I) and the data distribution pdata(s|I). The gradient of L(β) can be computed based on ∇β log pβ(s|I) = Epβ(z|s,I) [∇β log pβ(s, z|I)] = Epβ(z|s,I)[∇β(log pα(z) + log pθ(s|I, z))], (3) where the posterior distribution pβ(z|s, I) = pβ(s, z|I)/pβ(s|I) = pα(z)pθ(s|I, z)/pβ(s|I). The learning gradient in Eq. (3) can be decomposed into two parts, i.e., the gradient for the energybased model α
Epβ(z|s,I)[∇α log pα(z)] = Epα(z)[∇αUα(z)]− Epβ(z|s,I)[∇αUα(z)], (4) and the gradient for the transformer θ
Epβ(z|s,I)[∇θ log pθ(s|I, z)] = Epβ(z|s,I) [ 1
σ2 (s− Tθ(I, z))∇θTθ(I, z)
] . (5)
∇αUα(z) in Eq. (4) and ∇θTθ(I, z) in Eq. (5) can be efficiently computed via back-propagation. Both Eq. (4) and Eq. (5) include intractable expectation terms Ep(·), which can be approximated by MCMC samples. To be specific, we can use a gradient-based MCMC, e.g., Langevin dynamics, which is initialized with a Gaussian noise distribution p0, to draw samples from the energy-based prior model pα(z) ∝ exp [−Eα(z)] by iterating
zt+1 = zt − δ∇zEα(zt) + √ 2δet, z0 ∼ p0(z), et ∼ N (0, I), (6)
and draw samples from the posterior distribution pβ(z|s, I) by iterating zt+1 = zt−δ [ ∇zEα(zt)− 1
σ2 (s− Tθ(I, zt))∇zTθ(I, zt)
] + √ 2δet, z0 ∼ p0(z), et ∼ N (0, I).
(7) δ is the Langevin step size and can be specified independently in Eq. (6) and Eq. (7). We use {z+i } and {z−i } to denote, respectively, the samples from the posterior distribution pβ(z|s, I) and the prior distribution pα(z). The gradients of α and θ can be computed with {(Ii, si)}, {z+i } and {z − i } by
∇α = 1 n n∑ i=1 [∇αUα(z−i )]− 1 n n∑ i=1 [ ∇αUα(z+i ) ] , (8)
∇θ = 1 n n∑ i=1 [ 1 σ2 (si − Tθ(Ii, z+i ))∇θTθ(Ii, z + i ) ] , (9)
We can update the parameters with ∇α and ∇θ via the Adam optimizer [30]. We present the full learning and sampling algorithm of our model in Algorithm 1.
Algorithm 1 Maximum likelihood learning algorithm for generative vision transformer with energybased latent space for saliency prediction Input: (1) Training images {Ii}ni with associated saliency maps {si}ni ; (2) Number of learning iterationsM ; (3) Numbers of Langevin steps for prior and posterior {K−,K+}; (4) Langevin step sizes for prior and posterior {δ−, δ+}; (5) Learning rates for energy-based prior and transformer {ξα, ξθ}; (6) batch size n′. Output: Parameters θ for the transformer and α for the energy-based prior model 1: Initialize θ and α 2: for m← 1 to M do 3: Sample observed image-saliency pairs {(Ii, si)}n ′ i
4: For each (Ii, si), sample the prior z−i ∼ pαm(z) using K − Langevin steps in Eq. (6) with a step size
δ−. 5: For each (Ii, si), sample the posterior z+i ∼ pβm(z|si, Ii) using K
+ Langevin steps in Eq. (7) with a step size δ+. 6: Update energy-based prior by Adam with the gradient∇α computed in Eq. (8) and a learning rate ξα. 7: Update transformer by Adam with the gradient∇θ computed in Eq. (9) and a learning rate ξθ . 8: end for

heading: 3.3 Network
text: Generative vision transformer: We design the generative vision transformer using the Swin transformer [40] backbone as shown in Figure 2, which takes a three-channel image I and the latent variables z as input and outputs a one-channel saliency map Tθ(I, z). Two main modules are included in our generative vision transformer, including a “Transformer Encoder” module and a “Feature Aggregation” module. The former takes I as input and produces a set of feature maps {fl}5l=1 of channel sizes 128, 256, 512, 1024 and 1024, respectively, while the latter takes {fl}5l=1 and the vector of latent variables z as input to generate the saliency prediction s.
Specifically, we first feed each fl to a 3 × 3 convolutional layer to reduce the channel dimension to 32, and obtain a new set of feature maps {f ′l}5l=1 after channel reduction. Then, we replicate the vector z spatially and perform a channel-wise concatenation with f ′5, followed by a 3× 3 convolutional layer that seeks to produce a feature map
F5 with same number of channels as that of f ′5. Finally, we sequentially concatenate feature maps from high level to low level via feature aggregation, i.e., from l = 4 to 1, we compute Fl = Conv3×3(M(Concat(f ′l , Fl+1, ..., F5))), where Conv3×3(·) is a 3× 3 convolutional layer that reduces the channel dimension to 32, M(·) is the channel attention module [91], and Concat(·) is the channel-wise concatenation operation. Note that, we upsample the higher level feature map to the same spatial size as that of the lower level feature map before the concatenation operation. We feed F1 to a 3× 3 convolutional layer to obtain the one-channel saliency map Tθ(I, z). Energy-based prior model: We design an energy-based model for the latent variables z by parameterizing the function Uα(z) via a multilayer perceptron (MLP), which uses three fully connected layers to map the latent variables z to a scalar. The sizes of the feature maps of different layers of the MLP are Ce, Ce and 1, respectively. We will simply use Ce to represent the size of the EBM prior and set Ce = 60 in our experiment. GELU [25] activation is used after each layer except the last one.

heading: 3.4 Analysis
text: Convergence: Theoretically, when the Adam optimizer of β = (θ, α) in the learning algorithm converges to a local minimum, it solves the following estimating equations
∇α = 0 ⇒ 1 n n∑ i=1 Epβ(zi|si,Ii)[∇αUα(zi)]− Epα(z)[∇αUα(z)] = 0, (10)
∇θ = 0 ⇒ 1 n n∑ i=1 Epβ(zi|si,Ii) [ 1 σ2 (si − Tθ(Ii, zi))∇θTθ(Ii, zi) ] = 0, (11)
which are the maximum likelihood estimating equations. However, in practise, the Langevin dynamics in Eq. (8) and Eq. (9) might not converge to the target distributions due to the use of a small number of Langevin steps (i.e., short-run MCMC), the estimating equations in Eq. (10) and Eq. (11) will correspond to a perturbation of the MLE estimating equation according to [44, 45, 47]. The learning algorithm can be justified as a Robbins-Monro [55] algorithm, whose convergence is theoretically sound. Our model can also be trained with an extra encoder as an amortized inference network [31] for pβ(z|s, I) and an extra generator as an amortized sampling network [70, 71, 77] for pα(z) . In this work, we prefer to keep our training algorithm simple in order to avoid extra efforts for the design of the auxiliary network architectures. We will study the joint training strategy in our future work.
Accuracy: Compared with the GAN-based generative framework, our model is a likelihood-based generative framework that will not suffer from mode collapse [2]. In comparison with VAE-based generative framework, whose training is also based on likelihood, our MCMC-based maximum likelihood learning algorithm will not encounter the posterior collapse issue that is caused by amortized inference. On the other hand, the variational inference typically relies on an extra inference network for efficient inference of the latent variables given an image and saliency pair, however, the approximate inference might not be able to take the full place of the posterior distribution in practise. To be specific, we use qφ(z|s, I) to denote the tractable approximate inference network with parameters φ. The variational
inference seeks to optimize minβ minφ KL(pdata(s|I)qφ(z|s, I)||pβ(z, s|I)), which can be further decomposed into minβ minφ KL(pdata(s|I)||pβ(s|I)) + KL(qφ(z|s, I)||pβ(z|s, I)). That is, the variational inference maximizes the conditional data likelihood plus a KL-divergence between the approximate inference network and the posterior distribution. Only when the KL(qφ(z|s, I)||pβ(z|s, I))→ 0, the variational inference will lead to the MLE solution, which is exactly the objective of our model. However, there might exist a gap between them in practise due to the improper design of the inference network. Our learning algorithm is much simpler and more accurate than amortized inference.
Computational and memory costs analysis: From the learning perspective, due to the iterative Langevin sampling for the posterior and prior distributions of latent variables, our model is more time-consuming for training than generative models with amortized inference, such as VAE, which is roughly 2.4 times faster than ours on RGB image-based salient object detection. However, for VAE, the inference model is parameterized by another set of parameters, which need to be updated by back-propagation. In our model, the Langevin dynamics is not treated as a model because once the top-down generator is updated in each iteration, the posterior distribution can be derived from the generator. With the posterior distribution, the Langevin sampling is just an optimization-like process to find fair samples in the latent space defined by the posterior distribution. Without relying on an extra inference network, our framework is efficient in memory and friendly for network design.

heading: 4 Experimental Results

heading: 4.1 Setup
text: Datasets: For RGB SOD, we train models on DUTS training set [62], and test them on five benchmark datasets, including DUTS testing set, ECSSD [82], DUT [83], HKU-IS [35] and PASCAL-S [36]. For RGB-D SOD, we follow the conventional training setting, in which the training set is a combination of 1,485 images from NJU2K dataset [29] and 700 images from NLPR dataset [50]. We test the trained models on NJU2K testing set, NLPR testing set, DES [8], SSB [46] and SIP [15] testing set.
Evaluation Metrics: We adopt four evaluation metrics to measure the performance, including Mean Absolute ErrorM, Mean F-measure (Fβ), Mean E-measure (Eξ) [14] and S-measure (Sα) [13]. Implementation Details: Our generative vision transformer is built upon the Swin transformer [40] and uses it as the backbone of the encoder in our framework. The Swin backbone can be initialized with the parameters pretrained on the ImageNet-1K [9] dataset for image classification, and the other parameters of the newly added components, including the decoder part and the MLP of the energy based prior model, will be randomly initialized from the Gaussian distribution N (0, 0.01). Empirically we set the number of dimensions d of the latent variables z as d = 32. We set σ = 1 in Eq. (1) and σz = 1 in Eq. (2). We resize all the images and the saliency maps to the resolution of 384× 384 pixels to fit the Swin transformer. The maximum epoch is 50. The initial learning rates
are 2.5× 10−5. The whole training takes 9 hours with a batch size n′ = 10 on one NVIDIA GTX 2080Ti GPU for each model. During testing, our model can process 15 images per second.

heading: 4.2 Performance Comparison
text: We compare our framework with the state-of-the-art RGB SOD models and RGB-D SOD models, and show a comparison of performance in Table 1 and Table 2 respectively, where VST [39] is the only transformer-based saliency detection model. We observe consistently better performance of the proposed frameworks. Especially for PASCAL-S dataset [36], which contains more than 40% examples with large-sized salient objects, the significant performance gap between our method and the existing solutions demonstrates the effectiveness of our generative framework for saliency prediction. We further show a qualitative comparison between our RGB-D saliency prediction model and the BBSNet [16] in Figure 3. As we can see, our transformer-based framework, with an effective global context modeling, is superior to its competitor in detecting various sizes of salient objects.

heading: 4.3 Backbone Analysis
text: To test the performance of the transformer structure proposed in Section 3.3, we compare it with a conventional convolutional backbone for salient object detection. We create the baseline by replacing the Swin encoder in our transformer structure by the ResNet50 [24] encoder and keeping the decoder part (i.e., the “Feature Aggregation” module shown in Figure 2) unchanged for a fair comparison.
We first test them without involving latent variables, which means that we need to train them in a discriminative manner. We use “Ours-Swin-D” to denote the model using the Swin encoder and “OursRes50-D” to denote the one using the ResNet50 encoder. Model performance in the task of RGB-D saliency detection are shown in Table 3. We observe that “Ours-Swin-D” outperforms “Ours-Res50D”, which clearly indicates the effectiveness of transformer backbone for salient object detection.
We further study the influence of different encoder backbones in the context of the proposed generative saliency prediction framework with an EBM prior. Table 4 and Table 5, respectively, depict the performance comparisons of our frameworks using different backbones in the tasks of RGB and RGBD salient object detections. We compare the ResNet50 encoder backbone and the Swin transformer encoder backbone. We also modify the DPT [54] backbone such that it can adapt to the tasks of RGB and RGB-D salient object detections with our framework. The DPT built on the ViT transformer [10] is originally designed for semantic segmentation and depth estimation. The comparison results verify the effectiveness of the vision transformer used in our generative framework. In particular, our current solution with the Swin transformer encoder backbone can achieve the best performance for saliency prediction. Further, the performance gap between “Ours-Swin-D” in Table 3 and “Ours-Swin” in Table 5 indicates the effectiveness of the generative learning with an expressive latent space.

heading: 4.4 Generative Learning Analysis
text: We compare our framework with other alternative generative solutions in Table 6. The results are reported in the task of RGB-D saliency detection. For fair comparison, we implement an ABP-based model [22], a GAN-based model [20], and a VAE-based model [31, 56] using the same transformerbased generator as ours. The latent variables in these models are still assumed to follow the isotropic Gaussian distribution, as in their original algorithms. To be specific, for the ABP-based model, we use MCMC-based inference while training, and sample the latent variables directly from the Gaussian distribution during testing. For the GAN-based alternative, we design a fully convolutional discriminator [26] that consists of five 3 × 3 convolutional layers with a stride of 2 in each layer. The discriminator takes the concatenation of an image and a saliency map as input, and is trained to distinguish between the predicted saliency map and the ground truth given an image. The numbers of the output channels of the discriminator are 64, 64, 64, 64 and 1. For the VAE-based alternative, we introduce an extra encoder as an approximate Gaussian inference model via the reparameterization trick. The encoder consists of four 4 × 4 convolutional layers with a stride of 2 in each layer and maps the concatenation of an image and a saliency map to feature maps of channel sizes 64, 64, 64 and 64 sequentially. After that, two fully connected layers are adopted to produce the mean and the standard deviation of the inference model. For all the three alternative generative models, we set the numbers of the dimension of the latent space the same as that in our model, which is d = 32.
As shown in Table 6, compared with the deterministic baseline “Ours-Swin-D” (in Table 3), the three generative frameworks in achieve better or comparable performance. Especially in the DES dataset [8], they achieve large performance improvements. Our model outperforms all these alternative generative solutions, showing the effectiveness of the informative EBM prior distribution used in our model.

heading: 4.5 Hyperparameter Analysis
text: The main hyperparameters in our framework include the number of Langevin steps K, the Langevin step size δ, the number of dimensions of the latent space d, and the size of EBM prior Ce. We have two sets of hyperparameters {δ−,K−} and {δ+,K+} of the Langevin dynamics for sampling from the prior distribution and the posterior distribution, respectively. As to the Langevin step size, we find stable model performance with δ− ∈ [0.2, 0.6] and δ+ ∈ [0.05, 0.3], and we set δ− = 0.4 and δ+ = 0.1 in our paper. For the number of Langevin steps, we empirically set K− = K+ = 5 to achieve a trade-off between the training efficiency and the model performance, as more Langevin steps will lead to longer training time but more convergent inference results. Additionally, we investigate the influence of the number of latent dimensions by varying d = {8, 16, 32, 64}, and observe comparable performance among different choices of d. We set d = 32 in our paper. We also investigate the influence of the EBM size by varying Ce = {20, 60, 100}, and show the results in Table 7, in which we find Ce = 60 can provide optimal saliency prediction performance.

heading: 4.6 Explainability Analysis
text: As a generative model, our framework is capable of obtaining a meaningful uncertainty map that summarizes the stochastic behavior of the model in performing saliency prediction. The uncertainty in our paper refers to the stochastic property of generating the saliency prediction from an input image, which is captured by the probabilistic model pβ(s|I). In general, a generative saliency prediction method can provide not only accurate predictions but also reasonable uncertainty maps that represent the “subjective nature” of the human visual saliency. In this section, we propose to use the uncertainty map to help qualitatively evaluate different generative saliency prediction frameworks. Specifically, we propose to compute the uncertainty map as the variance of multiple saliency predictions produced from the learned probabilistic model. In the experiment, for each input image, we first output ten saliency maps by using the Langevin sampling from the learned conditional distribution, and then compute the variance map (uncertainty) based on the generated saliency predictions.
To quantify the complexity of a color image, we calculate a complexity score by the following way: (i) we first over-segment the image with the SLIC [1] to obtain 200 superpixels, (ii) then we compute the similarity of each superpixel with the others using handcrafted features from [99], which gives us a 200-dimensional contrast vector, representing an overall contrast of the image, (iii) we define the complexity of the image as the mean entropy of the contrast vector. The higher the score, the more complicated the image. In general, a reasonable generative saliency prediction model should have more confident predictions on images with simpler backgrounds (i.e., lower complexity scores), and less confident predictions on images with complicated backgrounds (i.e., higher complexity scores).
In Figure 4, we compare the uncertainty maps of different generative models that we have presented in Section 4.4. For each row of Figure 4, we display an input image, which is tagged with an image complexity score, and the associated ground truth saliency map, followed by the predicted saliency maps and the uncertainty maps obtained by the ABP-based model, the GAN-based model, the VAE-based model, and our model, respectively. As shown in Figure 4, the complexity score of each image is reasonable in the sense that it is consistent with the human perception. We observe that the uncertainty maps obtained by the other generative models fail to reflect the difficulties of the images for saliency prediction. For example, the image shown in the first row of Figure 4 has a structured foreground (i.e., a temple) and a textured background (i.e., a forest), which leads to a large ambiguity of the boundary between the foreground and the background. The uncertainty map obtained by our model indicates a big variance around the boundary region, which demonstrates the explainability and the reasonability of our model. Note that only generative saliency prediction frameworks can provide such an explainability analysis based on the uncertainty maps. This might motivate us to develop generative frameworks for explainable saliency prediction in the future.

heading: 5 Conclusion and Discussion
text: In this paper, we study the generative modeling and learning of vision transformer in the context of RGB and RGB-D salient object detections. We start from defining a conditional probability distribution of saliency map given an input image by a top-down latent variable generative framework, in which the non-linear mapping from image domain to saliency domain is parameterized by a proposed vision transformer network and the prior distribution of the low-dimensional latent space is represented by a trainable energy-based model. Instead of using amortized inference and sampling strategies, we learn the model by the MCMC-based maximum likelihood, where the Langevin sampling is used to evaluate the intractable posterior and prior distributions of the latent variables for calculating the learning gradients of the model parameters. With the informative energy-based prior and the expressive top-down vision transformer network, our model can achieve both accurate predictions and meaningful uncertainty maps that are consistent with the human perception.
From the machine learning perspective, our model is a likelihood-based top-down deep conditional generative model, which is neither CGAN-based nor CVAE-based frameworks, and it is trained without relying on any assisting network. The learning algorithm derived from the proposed model is based on MCMC inference for the posterior and MCMC sampling for the prior, which makes our framework more natural, principled, and statistically rigorous than others. The MCMC-based inference is immediately available in the sense that there is nothing to worry about the non-trivial design and training of a separate inference model as in VAEs. Such a framework is not only useful for saliency prediction (Though this is what we target in this paper) but also applicable to a vast of conditional learning scenarios, such as semantic segmentation, image-to-image translation, etc. Thus, the proposed generative model and the learning algorithm are generic and universal.
From the computer vision perspective, our model with a special network design to handle saliency prediction is a new member of the family of saliency prediction methods. In comparison with the traditional discriminative saliency prediction methods, our generative method is natural and reasonable because it models the saliency prediction as a conditional probability distribution. Moreover, it demonstrates impressive performance over all RGB and RGB-D SOD benchmarks. As we know, the computer vision community has started to develop vision transformer networks for various computer vision tasks, such as image classification, segmentation, detection, generation, etc. Our paper is the first one to present a generative vision transformer for both RGB and RGB-D saliency predictions. Thus, the proposed framework is significantly important for the computer vision community.

Review summaries:
The author construct a generative model for salient object detection in the form of top-down conditional latent variable model. A generative vision transformer is used by adding latent variables into the traditional deterministic transformer. Experimental results on RGB and RGB-D image salient object detection show that the generative framework equipped with the EBM prior and the transformer-based non-linear mapping is powerful in representing the conditional distribution of object saliency given an image.

This paper introduces a novel* supervised generative model for detecting salient image regions, where saliency is defined as containing object. This is a latent-variable model with an energy-based prior and a transformer decoder that maps the latent variable to a saliency map. The prior and posterior inference is done via SGLD. The model is well-evaluated on several datasets, with several metrics, and compared across a range of state-of-the-art baselines. It not only outperforms every single baseline (setting new SOTA), but also has the added benefit of providing high-quality uncertainty estimates. Detailed ablation studies show the relative importance of various model components.  \* I think it is novel, but difficult to say as I am not expert in the field and there is very information in the paper about prior art.  # Update I'm happy with the author response and am increasing my score to 7. I still have some doubts about the motivation of this work, but the theory is sound and the the results are very good.

This paper proposes a method based on generative vision transformer with latent variables  following an informative energy-based prior for salient object detection. Both the vision transformer network and the energy based prior model are jointly trained  via Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation.  Extensive experiments reveal both high performance model and meaningful confidence map scores, as well as estimation of uncertainty maps.

The submission proposes a method for saliency detection based on a generative adversary learning framework, which uses a transformer architecture to replace the backbone of the framework and is jointly trained with a Markov chain Monte Carlo-based maximum likelihood estimation to improve the performance. The performance of the proposed method is evaluated on both RGBD- and RGB-based saliency prediction datasets. The paper is generally written well and easy to follow.