Variational Structured Attention Networks for Dense Pixel-Wise Prediction
Authors: Guanglei Yang
Paolo Rota
Xavier Alameda-Pineda
Dan Xu
Mingli Ding
Elisa Ricci
Venue:  (2021)
Keywords: attention network
pixel-wise prediction

Abstract: State-of-the-art performances in dense pixel-wise prediction tasks are obtained with specifically designed convolutional networks. These models often benefit from attention mechanisms that allow better learning of deep representations. Recent works showed the importance of estimating both spatial- and channel-wise attention tensors. In this paper, we propose a unified approach to jointly estimate spatial attention maps and channel attention vectors so as to structure the resulting attention tensor. Moreover, we integrate the estimation of the attention within a probabilistic framework, leading to VarIational STructured Attention networks(VISTA). We implement the inference rules within the neural network, thus allowing for joint learning of the probabilistic and the CNN front-end parameters. Importantly, as demonstrated by our extensive empirical evaluation on six large-scale datasets VISTA outperforms the state-of-the-art in multiple continuous and discrete pixel-level prediction tasks, thus confirming the benefit of structuring the attention tensor and of inferring it within a probabilistic formulation.

Under review as a conference paper at ICLR 2021
VARIATIONAL STRUCTURED ATTENTION NETWORKS FOR DENSE PIXEL-WISE PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-arr performances in dense pixel-wise prediction tasks are obtained with specifically designed convolutional networks. These models often benefit from attention mechanisms that allow better learning of deep representations. Re- cent works showed the importance of estimating both spatial- and channel-wise attention tensors. In this paper propose unified approach to jointly estimate spatial attention maps and channel attention vectors so as to structure the result- ing attention tensor. Moreover, integrate the estimation of the attention within a probabilistic framework, leading to Varlational STructured Attention networks (VISTA-Net). We implement the inference rules within the neural network, thus allowing for joint learning of the probabilistic and the CNN front-end parameters. Importantly, as demonstrated by our extensive empirical evaluation on six large- scale datasets, VISTA-Net outperforms the state-of-the of-the--rt-tr in multiple continuous and discrete pixel-level prediction tasks, thus confirming the benefit of structuring the attention tensor and of inferring it within a probabilistic formulation.
INTRODUCTION
Over the past decade, convolutional neural networks (CNNs) have become the privileged method- ology to address computer vision tasks requiring dense pixel-wise prediction, such as semantic seg- mentation (Chen et al. 2016bb; Fu et al., 2019), monocular depth prediction (Liu et al.
2015 Roy &
Todorovic 2016), contour detection (Xu et al.) (2) and normal surface computation (Eigen al.]
2014]. Recent studies provided clear evidence that attention mechanisms (Mnih et al.)
2014) within
deep networks are undoubtedly crucial factor improving the performance (Chen al. 2016b
Xu et al.] 2017a Fu et al., 2019 Zhan et al., 2018). In particular, previous works demonstrated that deeply learned attentions acting as soft weights to interact with different deep features at each chan-
nel (Zhong et al. 2020; Zhang al. 2018, Song al. 2020) and at each pixel location (Li et al.) 2020a] Johnston & Carneiro 2020 Tay et al. 2019) permits to improve the pixel-wise prediction accuracy (see l a and Fig b).. Recently, Fu al. (2019) proposed the Dual Attention Network (DANet), embedding in a fully convolutional network (FCN) two complementary attention modules, specifically conceived to model separately the semantic dependencies associated the spatial and to the channel dimensions [Fi
Concurrently, other approaches have considered the use of structured attention models integrated within a graph network framework (Zhang et al. 2020 Chen et al. [2019] Xu et al. 2017a), showing the empirical advantage of adopting graphical model effectively capture the structured infor- mation present in the hidden layers of the neural network and thus enabling the learning of better deep feature representations. Notably, Xu al.] (2017a) first introduced attention- gated conditional random fields (AG-CRFs), a convolutional neural network implementing probabilistic graphical model that considers attention variables as gates (Minka & Winn 2009) order to learn improved deep features and effectively fuse multi-scale information. However, their structured attention model is only learned at the spatial-wise level, while thannel-wise dependencies are not considered.
This paper advances the state of the art in dense pixel-wise prediction by proposing novel approach to learn more effective deep representations by integrating a structured attention model which jointly account for spatial- and channel-level dependencies using attention tensor (Fig d within a CRF framework. More precisely, inspired from Xu et al.] (201 7a) we model the attention as gates. Cru- cially, address the question on how enforce structure within these latent gates, in order to

---

Under review as a conference paper at ICLR 2021
Chennet
Chraclatc sdille toratrop
atfestice matris
Seuctord laine
Channel
Spuiil
Sirectred atrestion broaer
atterion
(c) Separate spatial and channel
(b) Spatial attention.
Proposed attention.
attention.
Different attention mechanisms. (a) and (b) correspond to channel-only and spatial-only attention, respectively. (c) corresponds to previous works (Fu et al. 2019) adding (#) channel and spatial attention tensor. (d) shows the attention mechanism of VISTA-Net: a channel-wise vector and spatial map are estimated then tensor-multipee mustip--ll (8) yielding a structured attention tensor. The attention tensor acts structured latent gate, producing probabilistically enhanced feature map.
Figure 1:
jointly model spatial- and channel-level dependencies while learning deep features. To do so, we hypothesize that the attention tensor is nothing but the sum of T rank-1 tensors, each of them being the tensor product of a spatial attention map and a channel attention vector. This attention tensor used as structured latent attention gate, enhancing the feature maps. We cast the inference problem into a maximum liiel-iiioeoood estimation formulation that is made computationally tractable thanks to a variational approximation. Furthermore, we implement the maximum likelihood update rules within neural network, so that they can be jointly learned with the preferred CNN front-end. We called our approach based on structured attention and variational inference Varlational STruc- tured Attention Networks VISTA-Net. We evaluate method multiple pixel-wise prediction problems, i.e. monocular depth estimation, semantic segmentation and surface normale prediction, considering six publicly available datasets, i.e. NYUD-VA V2 (Silberman et al. 2012), KITTI (Geiger (et al.] 2013), Pascal- Conttext (Mottaghi et al.] 2014), Pascal VOC2012 (Everingham et al. 2010), Cityscape (Cordts al., 2016) and ScanNet (Dai et al. 2017). Our results demonstrate that VISTA- Net is able to learn rich deep representations thanks to the proposed structured attention and our probabilistic formulation, outperforming state-of-the-art methods.
Related Work. Several works have considered integrating attention models within deep archi- tectures to improve performance in several tasks such as image categorization (Xiao et al. 2015).
speech recognition (Chorowski al. 2015) and machine translation (Vaswani al. 201 7: Kim et al. 2017 Luong et al. 2015). Focusing pixel-wise prediction, Chen et al. (201 first de- scribed an attention model to combine multi-scale features learned by FCN for semantic segmenta- tion. /Zhang al. (2018) designed EncNet, network equipped with channel attention mechanism to model global context. Zhao et al. (2018) proposed to account for pixel-wise dependencies in- troducing relative position information in spatial dimension within the convolutional layers. Huang ct al. (2019b) described CCNet, a deep architecture that embeds a criss-cross attention module with the idea of modeling contextual dependencies using pparsely-connecten graphs, such achieve higher computational efficiency. Fuu et al.] (2019) proposed to model semantic dependencies asso- ciated with spatial and channel dimensions by using two separate attention modules. Zhong et al. (2200) introduced a gquuezze-and-attetioo network (SANet) specialized to pixel-wise prediction that takes into account spatial and channel inter-dependencies in an efficient way.
Attention was first adopted within CRF framework by Xu et al. (2017a), which introduced gates control the message passing between latent variables and showed that this strategy is effective for contour detection. Our work significantly departs from these previous approaches, as we introduce a novel structured attention mechanism, jointly handling spatial- and channel-level dependencies within probabilistic framework. Notably, we also prove that our model can be successfully em- ployed in case of several challenging dense pixel-level prediction tasks. Our work is also closely related to previous studies on dual graph convolutional network ŞZhang et al.) 2019c) and dynamic graph message passing networks (Zhang et al. 2020). which have been successfully used for pixel- level prediction tasks. However, while they also resort message passing for learning refined deep feature representations, they lack probabilistic formulation. Finally, previous studies (Xu et al.) 2017c Arnab al. 2016, Chen et al. 2019) described CRF-based models for pixel-wise estima-
2

---

Under review as a conference paper at ICLR 2021
tion, e.g. to learn and optimally fuse deep representations multiple scales. However, they did not
employ structured attention gates.
VARIATIONAL STRUCTURED ATTENTION NETWORKS: VISTA-NET
As previously discussed, we aim to enhance the learned representation by structuring the attention within a probabilistic formulation. One the one side, inducing structure in the attention mechanisms has been proven to be successful (Fu et al. 2019 Zhong et al. 2020). On the other side, probabilistic formulations combined with deep architectures are interesting for pixel-level prediction tasks (Xu et al. 201 Up to our knowledge, we are the first to bring together recent advances in pixel-wise prediction by formulating a novel structured attention mechanism within probabilistic CRF-like inference framework. Inspired by Fu et al. (2019), where two spatial- and channel-wist full- rank tensors are computed, we opt to infer different spatial and channel attention variables. Very differently from Fu al./ (2019), we propose to structure generic attention tensor of dimension W x x C' (widht, height, channels), as the sum T rank- I tensors:
m 2 v
m' E RIIWXH vt RCXIX
meaning that m' can be understood as an image of W x H pixels and v as a vector of dimension C, and @ denotes the tensor product, in the case above leading to a 3-way tensor of dimensions W x H. X C. Each of the tensor products within the sum yields a tensor of rank-1, 1, consequently limiting the rank of a to be at maximum T. Equation (T) is the algebraic expression of the proposed structured attention mechanism, and is the methodological foundation of VISTA-Net.
Moreover, we inspire from the CRF formulation with gating variables proposed (Xu al. 2017a), and derive a new energy function and variational approximation to enable efficient learning and inference procedures. Additionally, this formulation allows us to consider the CRF kernels as latent variables and infer them from the data, together with the structured attention variables m' and v*. We believe learning the kernels is important because allows the CRF to weight the information flow depending on the content of the rather than keeping the same weights for all images.
We assume generic CNN front-end providing a set of S multi-scale feature maps = (f.)s= To ease notation, we assume that each feature map has P pixels and C channels, but in practice these dimensions depend on the scale s. For each scale, we also consider the set of hidden variables z, ) ===-- These hidden variables correspond to refined convolutional
corresponding to fg, and Z { futures that incorporate information and attention from other feature maps, so as to better represent the key information for the pixel-level task at hand. Intuitively, the structured attention tensor should help refining the hidden variables to allow better performance at various pixel-level prediction tasks.
As in (Xu et al. 2017a), for every pair of emitting e and receiving scales, consider dedicated attention tensor ag Very importantly, in our case this attention tensor is structured following ( S.S,T
and have set of hidden spatial attention maps M {mm, Ie..r, and hidden channel
attention vectors V = = (Ve.rJe,,;=1 More precisely, m E {0, 1)" and ve,- E 10 are a binary spatial map and a stochastic channel-wise vector, hence De vie = 1. In this way, reduce ambiguity and ease the learning. This also means that the model is conceived to pay attention to only T channels of the feature map. While this could seem limiting at first glance we remark that: (i) the model learns which are the optimal T channels among the possible C that have to be used to refine the hidden variables and (ii) the posterior distribution of m' boils down to a convex combination of all channels, it will appear clear when discussing the inference procedure.
S.S.T
2.1 ENERGY FUNCTION AND VARIATIONAL APPROXIMATION
Our model consists on three different latent variables: the hidden features Z, and the hidden attention maps M and vectors V. In addition, also consider inferring the CRF kernels, denoted by K from
3

---

Under review as a conference paper at ICLR 2021
the data. More precisely, the energy function associated to the proposed models writes:
= ΣΣ
(2( $1.'0)
P.C
+Σ oor P.c.ple'
Σ
Aapohesspp spd seple' m " (22
soid prre + @n(sF " kealee )
(2)
where 6s, 0k and e are potentials to be defined and ke.ple' denotes the kernel value weighting the p.p.cc
information flow from the (pl c')-th value of the feature map of scale to the (p, c)-th value of the feature map of scale r. Since the exact posterior distribution is not computationally tractable, we opt to approximate it with the following family of separable distributions:
pZZ M, V, F, )) = (Z. V, K) = (z (M) ( (K).
(3)
In that case, the optimal solution for each of the factors of the distribution take the expectation w.r.t. to all the others, for instance:
q= (Z) exp (- (-E (MTya (IIZ.M.VK....)).
(4)
It can be shown that the optimal variational factors write:
(=(=f() x exp
(= P.C (5%0)) +
ΣΣ
GP meir Heit
By (1b(2p 22 sid A)
estt
8
Pie
Im (m)))) x exp mm
Σ
Eyg (vb2p
P.e, zid kepplee 13).
pie!
(5)
go(u exp w Σ
Egg (1/220 sale negle))) 11):
may
Io (kkkgple & exp (F ppld' kagld
Σ
m L. Se (1 thd keplee)))
where ma = E, [m denotes the the posterior mean, and analogously for dhe. This result also implies that thanks to the variational approximation 3 the posterior distributions factorise pr.p.cm
in each of the variables above, e.g. ge (z( 11 S.P.C (2( The relation between the various
hidden variables as for their inference is shown Figure 2 (left). addition, we also show the information flow between the hidden variables using arrows. Finally, on Figure Figure[(right (right) we show the relation between the channel-wise and spatial attention variables and how the final structured attention tensor is computed.
2.2 INFERENCE WITH VISTA-NET
In order to construct an operative model we need to define the potentials Φ2, dh and th. In our case, the unary potentials correspond to:
bp.c
(xpie SP.0)
(zf
f2.0??
spld kewidd (f feed pr.ppc
(kep'e' - fp pold (6)
where bg.5 > is a weighting factor. vb is bilinear in the hidden feature maps:
eP-epeple' plee T
Using the over bar notation also for the hidden features and kernels, e.g. z8.c Ty, (2ge), and by combining the kernel definitions (6) and ( with the expression of the variational factors 5 we
obtain the following update rules for the latent variables.
Z-step. It can be seen that the posterior distribution on ys is Gaussian with mean:
gPCC
ΣΣ
t. ma G..
Σ
sple"
(8)
PP.C
4

---

Under review as a conference paper at ICLR 2021
Observed features
f
f'
(C.H.1]
aar
V m.
1
Convolutional kernels
m
k
M
ver
m
FC,
[C.M0]]
(C..J))
Hidden features
Structured attention tensor
Structured attention tensor
Figure 2: (left) Schematic representation of the various hidden variables in VISTA-Net. For each pair of emitting e and receiving scales, their respective convolutional features are shown in blue, their hidden variables green, the associated learned kemnel k in yellow, and the channel-wise and spatial attention vector and matrix and m in red. Arros of the corresponding color denote the flow of information when updating the variable. (right) The computational relationships between the channel-we -wise and spatial attention variables is shown, well as the operations required to compute the final structured attention tensor a.
This corresponds the update rule obtained in (Xu et al. 201 with two remarkable differences. First, the posterior of the attention gate corresponds to the posterior of the structured tensor of rank T. Second, the impact of the neighboring features is weighted by the expected kernel value keppl' r,p,e
M-step. The variational approximation leads to a Bernoulli distribution for qm (mf,")) which boils down to the following a posterior mean value using the sigmoid function a:
m
Σ The
zp. Teople spie" 2s
Pyppe 28'
(9)
pice
V-step. It can be shown that the approximated posterior distribution is categorical, and that the expected value of each dimension of ve,, can be computed using the softmax operator:
(o(e)== softmax
Σ
Lp mir
Σ
EpeTTeppe splce)
28
pipii )
C
(10)
ple'
K-step. Finally, we need to derive the update rules for K. By further deriving the corresponding variational posterior distribution, can be shown that the a posterior distribution for the kernels is Gaussian distribution with the following mean:
pep r.p.c
f.. fgle ppld
+ Σ
(11)
This solution is very straightforward, but since the kernels estimated independently for each pair of receiving (r.p.e) c)- emitting (e.p,c') pixels, it has two major drawbacks. First, the kernel values estimated without any spatial context. Second, given the large amount of kernel values, one must find a very efficient way to compute them. We propose to kill two birds with one stone by learning the kernels from the features using convolutional layers. By design, they take spatial context into account, and many popular libraries have efficient implementations of the convolution operation. The estimated kernel corresponding to the input channel of scale kadd' is computed convolutional operation. The input of the convolution is a concatenation of the tensor f, +
z, E m 2 v and the image z resized to the spatial size f,.
Joint learning. We implement the inference procedure described before within the neural network, on the top of the CNN front-end. Indeed, implementing all inference operations using available deep learning operators has prominent advantages. First, the CNN front-end at the same time, within the same formalism and for the same aim. Second, this allows direct parallelisation of method, speeding up training and inference.
perform the inference and learning
5

---

Under review as a conference paper at ICLR 2021
The precise implementation goes as follows. Regarding Zr, we first apply message passing from the e-th scale to the s,-th scale is performed with Ze-r + kt @ Ze. where @ denotes the convolutional operation and kद denotes the corresponding learned convolution kemell. We then apply element- wise product with the corresponding structured attention tensor Σ m @ V...- Finally we compute the element-wise sum with other emiting scales and the feature maps f,, see (8) Regarding me.r; we first compute the element-wise product between 2, and Ze-er. The sum over channels weighted by Ve,r is computed previous to applying pixel-wise sigmoid, see (9) Regarding Ve.r we operate in a very similar fashion, but weighting each pixel with m and then summing every channel independently, before applying softmax, see (10). Regarding kp as discussed before, it is computed via a convolutional operation on the concatenations of f... ge... and the image x resized to the spatial size of f.. In terms of initialisation, we draw a random guess for M and V, and set Z to F. This allows us update the kernels, then the other variables.
Once the hidden variables are updated, we use them to address several different pixel-wise prediction tasks involving continuous and discrete variables, including monocular depth estimation, surface normal estimation and semantic segmentation. Following previous works, the network optimization losses for these three tasks are standard 1.2 loss (Xu et al. 2017c), a consine similarity loss (Eigen et al., 2014) and a cross-entropy loss (Chen et al. 2016a), respectively. The CNN front-end and VISTA-Net, are jointly trained end-to-end.
3
EXPERIMENTAL EVALUATION
3.1 DATASETS AND EXPERIMENTAL PROTOCOL
Tasks and Datasets. We demonstrate the effectiveness of VISTA-Net on two tasks; monocular depth estimation on the NYU-v2 (Silberman et al., a 2012) and the KITTI (Geiger et al., 2013) datasets and semantic segmentation on the Pascal-Context (Mottaghi et al. 2014), the Pascal VOC2012 (Ev- cringham et al. 2010) and the Cityscape (Cordts et al.| 2016). We also conducted experiments on the surface normal estimation task on ScanNet (Dai et al. 2017) but due to lack of space the associated results are reported in the Appendix.
For NYU-v2 and KITTI follow the experimental settings proposed by Eigen et al. (Eigen al, 2014). For NYU-v2 we use 120K BeB-teeth pairs with a resolution of 480 x 640 pixels, acquired with a Microsoft Kinect device from 464 indoor scenes, using 249 scenes for training and 215 scenes (654 images) for test. For KITTI we specifically use 22,600 frames from 32 scenes for training and 697 frames from the rest 29 scenes for test.
For Pascal-Context we follow the works (Chen et al. 2016a Zhang et al. 2018 and we consider the most frequent 59 classes. The remaining classes are masked during training and test. Pascal VOC2012 contains 20 classes divided in 10582 training. 1449 validation and 1456 test images. Our method is trained using the protocol described in (Zhong et al. 2020 Long et al. 2015). For Cityscape dataset, only the 5,000 finely annotated images are used in our experiments, split into 2,975/500/1,52 images for training, validation, and test.
Evaluation Metrics. To evaluate the performance on monocular depth estimation, we consider several metrics as in (Eigen & Fergus 2015), including mean relative error (rel), root mean squared error (rms), mean log10 error (log10), and accuracy with threshold (t E (1.25, 1. 1.25 1.25*) As for semantic segmentation, we consider two metrics following (Zhou et al., 2017 Zhang et al.) 2018). i.e. pixel accuracy (pixAcc) and mean intersection over union (mloU), averaged over classes. Implementation Details. VISTA-Net is implemented in Pytorch. The experiments are conducted on four Nvidia Quadro RTX 6000 GPUs, each with 24 GB memory. The ResNet-101 architec- ture pretrained on ImageNet (Deng et al. 2009) is considered in all the experiments for initializing the backbone network of VISTA-Net except in for the experiments on the Cityscape dataset where we choose HRNet 448 (whose complexity is comparable to Gilated-RReNett-ll parison with previous works. Our model can be used for effective deep feature learning in both single-scale and multi-scale contexts. To boost the performance, following previous works (Xie & Tu 2015 Xu et al. 2017a), we also consider features output by different convolutional blocks of CNN backbone (e.g. resăc, ref4f, ref5d of ResNet-50).. For the semantic segmentation task, use a learning rate of 0.001 on Pascal-context and Pascal-VOS 2012 and 0.01 on cityscapes with a momentum of 0.9 and weight decay of 0.0001 using polynomial learning rate scheduler as pre-
for fair com-

---

Under review as a conference paper at ICLR 2021
Table 1: Depth Estimation: KITTI dataset. Only monocular estimation methods are reported.
Error (lower is better)
Accuracy (higher is better)
Method
abs-rel
sq Sq-rel
rms
log-rms
rms
1.25
6<11.222
6x11.25
CC (Ranjan al.
2019
0.140
1.070
5.326
0.217
0.826
0.941
0.975
Bian at mi. 2019
0.137
1.089
5.439
0.217
0.830
0.942
0.975
N" heng cl al.
2020a
0.124
0,826
4.981
0.200
018866
0.955
0.982
MS-CKF Xu al al.
201 7C
0.125
0.899
4.685
0.816
0.951
0.983
AG-CRF Xt 2017B
0.126
0.901
4.689
0.157
0.813
0,950
0.982
Monodepti2 (Godlard el
2019
0.115
0,903
4.863
0.193
0.877
0.959
0.981
pRGBD Tiwari u. 2020
0.113
0.793
4.655
0.188
0,874
0.96
0.983
SGDDpthh
2020
0.107
0.768
4.468
0.180
0.891
0.963
0.982
Johnston & Carneiro 2421
442207
0.106
0,861
4.699
0.185
0,889
0.962
0.982
Shu al. 2020
0.104
0,729
4.481
0.179
0.893
0.965
0.984
DOKN
2018
0.072
0.307
2.727
0.120
0.932
0,984
0.994
Yin et al. 2019
0.072
3.258
0,117
0.938
0,990
0.998
PackNet-Si (Guizilini et al. 2020
0.071
0,359
3.153
0.109
0,944
0.990
0.997
| al. 2019)
0.061
0.261
2.834
0.099
0.954
0.992
0.998
VISTA-Net (ours)
0.063
0.255
2.776
0.099
0.954
0.993
0.998
Erur (lower betier)
Avcurac Origher betler)
Method
rel.
log10
mms
6-c1
6<c1.25
6-:1.25
PAD-Net Xu
2018
0.214
0.091
0.792
0.6:3
0,9012
0.977
0.152
0.064
0.611
2020
0.128
0.171
0.493
0.991
Laina el d 1010
0.127
0.055
0.573
0.811
0,953
0.988
01
7017c
0.121
0.052
0.586
0,811
0.954
0.987
Lor Kin
20240
0.119
0.5
0.87
0.974
0.993
af
70
0.112
0.051
0.526
0.989
DORN
2018
0.115
0.051
0.509
0.828
0.992
Xia ct
2020
0.116
0.512
0.861
0.969
0.991
Tinerap al
20119
0.108
0.648
0.416
0.875
0.976
0.994
Lo0 2r
2019
0.113
0.349
0.407
0.871
0.977
0.998
VISTA-Net
0.111
0,048
0.393
0.996
Original DORN (Fu 2018 VISTA-Net
Figure 3: Depth estimation: quantitative (left) and qualitative (right) comparison on NYU-v2.
viously done in ZZhng et al. 2018 Chen et al. 2016a). For the monocular depth estimation task, the learning rate is set to 10 with weight decay of 0.01. The Adam optimizer is used in all our
periments with a batch size of 8 for monocular depth estimation and 16 for semantic segmentation.
The total training epochs are set to 50 for depth prediction experiments, to 150 for Pascal-context and Pascal VOC 2012 2 datasets and to 500 for the Cityscapes dataset.
3.2 EXPERIMENTAL RESULTS AND ANALYSIS
Monocular Depth Estimation. Comparative results on KITTI dataset are shown in Table [ We propose comparison with state of the art models such as (Eigen et al. 2014; Ranjan et al. 2019
Bian et al. 2019 Godard et al. 2019 Fu et al. 2018 Yin et al. 2019 Lee et al.) 2019; Guizilini et al. 2020). In addition we demonstrate the effectiveness of our VISTA-NG Net comparing with MS- CRF (Xu et al. 2017c), a previous approach which exploit a probabilistic framework for multi- scale feature learning but does not consider an attention mechanisms. Our approach is superior, thus demonstrating the effectiveness of the proposed attention model. We also compare with AG- CRF (Xu et al.) (017a) adapting their model to the monocular depth estimation problem. Also in this case VISTA-Net outperforms the competitor confirming the importance of having joint structured spatial- and channel-wise attention model. Note that AG-CRF (Xu et al., (007a) and VISTA-Net are compared using the same backbone. In order to demonstrate the competitiveness of our approach in an indoor scenario we also report the results on NYUD-V? dataset in Fig. 3 Similarly to the experiments on KITTI, VISTA-Net outperforms both state of the art approaches and previous methods based on attention gates and CRFs (Xu et al. a).
Semantic Segmentation. We first compare VISTA-Net with the most recent methods on the Pascal- Context dataset, including (Zhang et al. 2018 Fu et al. 2019 Zhu et al. 2019: Ding et al., 2019 Zhang et al. 2019b Wang al. 2020 He et al. 2019). As for the depth estimation task, also in this case we evaluate the performance of AG-CRF Xu et al. 70). adapting the original code to the semantic segmentation task. VesTA-Net, as shown in Table 2 is 0.6 points better according to the mloU metric than the best available method, i.e. AG-CRF. Importantly, VISTA-Net outperforms
EncNet (Zhang al. 2018), which uses only channel-wis attention, as well DANet (Fu al.)
7

---

Under review as a conference paper at ICLR 2021
Table Semantic Segmentation on PASCAL-Context. D-ResNet-101 denotes Dilated ResNet-101.
Method
Backbone
pixAcet
mloU%
CFM (VGG+MCG) Dui et al.
2015b
VGG-16
34.4
DeepLab-v2 (Chen
2016a
VGG-16
37.6
FCN-8s (Long of 2015
VGG-16
50.7
37.8
BoxSup
20154
VGG-16
40.5
ConvPP xx X16 of 2016
VGG-16
41.0
PixelNet Bansa
VGG-16
51.5
41.4
HRNetV2 (Wang et
2026
54.0
EncNet [hang et al.
2018
D. D-ResNet-101
79.23
51.7
DANet (Fu
2019
D-ResNct-101
52.6
ANN (Zhu et a. 2019
D-ResNet-101
52.8
SpyGit al
ResNet-101
SANet (Zhong al.
2020
ResNet 101
80.6
53.0
SVCNer (Ding et el.
2119
ResNet-101
53.2
CFNet Zhane cl
20196
ResNet-101
54.0
APCNeT [e al 2019
D-ResNet-101
54.7
AG CRF eu 20174
D ResNet- 101
80.8
54.8
VISTA Net (ours)
D D-ResNet-111 101
81.1
55.4
results on both Cityscapes validation and testing (trained on the standard training set) (left) and on PASCAL VOC 2012 validation set (right). D-ResNet-101 means
Table 3: Semantic Segmentation:
(*) indicates COCO pretrained weights.
Method
Backbone
Test
mloU
Method
Backbone
mloUs
Dynamic et 20206
Layer33-PSP
Val
SpyGR TTrepu 202u0
ResNet-101
Val
80.5
Docplab
Chen et
2017
D-RosNat-101
75.7
CCNet Hoang 2019b
ResNet-101
Val
81.3
Panoptic- DeSe 20r Uele
2020b
D-RosNet-101
Val
Dynamic
2020b
Layer33
CDGCNet (Hu al 2040
DeResNet-101
Val
81.9
Res2Net GGo
2019
Res2Nes-101
VISTA-Net
HRNe1V2 W48
DANc
Pu al.
2019
ResNe-101
80.4
PSANet (Zhao al.
2018
D-RosNet-101
Test
Auto Deoplab Liu et
2019
ResNee- 101
82.0
PAN (Li brai 2013
D-ResNet-100 101
Test
78.6
AAF
D-RosNot-101
Test
79.1
EncNet Zhang al.
2018
D- D-ResNet-101
85.9
HRNer Wang or al.
2020
HRNetV2-W48
Test
80.4
Zhong
Dynamic
Layer33.PSP
Test
80.7
ViSTA-Net
Test
81.4
VISTA-Nel
D-RosNet-101
2019) and SANet[Zhong et al. 2020), which considers inter-dependencies both at spatial and chan- level in their attention model. In rrihtt are shown results on PASCAL VOC2012. Again, our method outperforms EncNet (Zhang et al. 2018), SANet (Zhong et al., (2200) and DANet (Fu cct al. 2019). In particular, VISTA-Net is 3.7 points better according to the mloU metric than the best available method, i.e. SANet. Finally, Table. lefft) reports the results Cityscape. As in the pre- vious two datasets VISTA-Net outperforms the competitors (by nearly one point mloU). Additional results reported in the Appendix.
Ablation Study. We also performed an ablation study on the Pascal-context dataset to further demonstrate each proposed component's impact. (le-)) shows that the performance of VISTA- Net degrades not only when the model does not employ the structured attention mechanism, but also when only channel-wise or spatial-wise attention is used. Moreover, can also see the advantage of using the proposed probabilistic formulation for joint modeling both spatial- and channel- atten- tion principled manner. Interestingly, the performance achieved in each of the variants (spatial, channel. no probabilistic formulation) is similar. This leads us to believe that the proposed method's competitive advantage is in combining structured attention with probabilistic formulation. No- tably, the feature refinement through message passing seems to be the most crucial contribution to improve the performance. For the sake of completeness, we also report the results of DANet, and of AG-CRF (which corresponds to the Multiple-scalc/Spaae the performance of VISTA-Net for different values of the tensor rank T. is important to notice that the framework reaches better performance when T is higher. Fig. Iclearly illustrate the percep- tual improvement in segmentation masks obtained with higher values the attention tensor rank. Additional examples are provided in the supplementary material.
setting in). Finally, in Fig 5 we show
CONCLUSIONS
In this paper we proposed a novel approach to improve the learning of deep features representations for dense pixel-wise prediction tasks. Our approach seamlessly integrates novel structured atten-
8

---

Under review as a conference paper at ICLR 2021
(a) Original Image
(b) Gr
(c) VeSTA (wio AIL)
(
VIStA-Net(rani-l))
VISTA-Net (rank-9)
Figure 4: Semantic segmentation maps obtained with VISTA-Net on the Pascal-Context dataset.
Scales
Structured Atiention
Probabilistic
mloU
PixAce
DANet (Fu et nl. 2019
Segarate Attention
No
52.6
50
No
16L-
Single
scale
Channel
Sputial
Yes
Yes
53.1
53.0
79.7
79.8
51.2
Low-rank
Low-rank tensor
Yes
53.9
80.3
No structure
Yes
52.8
79.5
Multiple
Spatial
Yes
54.8
80.8
51
scale
Low-rank tensor
Channel
No
54.7
54.6
80.7
80.6
Low-rank
58.4
81.1
Number of Rank
Figure 5: Ablation study on the Pascal-context dataset: performance of VISTA-NG (left) for differ- attention mechanisms and scales, (right) for different values of the tensor rank T.
tion model within probabilistic framework. In particular, we proposed to structure the attention tensors the sum 7 rank-one tensors, each being the tensor-product of spatial attention map and channel attention vector. These two kinds of variables are jointly learned within the probabilistic formulation made tractable thanks to the variational approximation. The proposed structured atten- tion is rich enough to capture complex spatial- and channel-level inter-dependencies, while being efficient to compute. The overall optimisation of the probabilistic model and of the CNN front-end is performed jointly. Extensive experimental evaluations show that VISTA-Net outperforms state- oh-the-art methods on several datasets, thus confirming the importance of structuring the attention variables for dense pixel-level prediction tasks.
REFERENCES
Anurag Arnab, Sadeep Jayasumana, Shuai Zheng, and Philip HS Torr. Higher order conditional random fields in deep neural networks. In ECCV. Springer, 2016.
Aayush Bansal, Bryan Russell, and Abhinav Gupta. Marr revisited: 2d-3d alignment via surface normal prediction. In CVPR, pp. 5965-5974, 2016.
Aayush Bansal, Xinlei Chen, Bryan Russell, Abhinav Gupta, and Deva Ramanan. Pixelnet: Repre- sentation the pixels, by the pixels, and for the pixels. arXiv preprint arXiv:1702.06500 2017.
Jiawang Bian, Zhichao Li, Naiyan Wang. Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. In NewrIPS, 2019.
Liang- Chinh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. arXiv preprint 16Xiv:166..00 00915, 2016a.
Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan Yuille. Attention to scale: Scale- aware semantic image segmentation. In CVPR, 2016b.
Liang-Chie Chiih Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint MoXiv:/70.005500 2017.

---

Under review as a conference paper at ICLR 2021
Yunpeng Chen, Mareus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalan- tidis. Graph-based global reasoning networks. In CVPR, 2019.
Bin Cheng, Inderjot Singh Saggu, Raunak Shah, Gaurav Bansal, and Dinesh Bharadia. 83 net: Semantic-aware self-supervised depth estimation with monocular videos and synthetic data. ECCV, 2020a.
Bowen Cheng, Maxwell Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-deepla:: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In CVPR, pp. 12475-12485, 2020b.
Sungha Choi, Joanne T Kim, and Jaegul Choo. Cars can't unn't fly up in the sky: Improving urban-scene segmentation via height-driven attention networks. CVPR, pp. 9373-9383, 2020.
Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. Attention-based models for speech recognition. In NeurlPS, 2015.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pp. 5828- 5839, 2017.
Jifeng Dai, Kaiming He, and Jian Sun. Boxsup: Exploiting bounding boxes to supervise convolu- tional networks for semantic segmentation. In ICCV, 2015a.
Jifeng Dai, Kaiming He, and Jian Sun. Convolutional feature masking for joint object and stuff segmentation. In CVPR, 2015b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Henghui Ding, Xudong Jiang, Bing Shuai, Ai Qun Liu, and Gang Wang. Semantic correlation
promoted shape-variant context for segmentation. In CVPR, 2019.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. In ICCV,
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. In NeuriPS, 2014.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 88(2))303-338, 2010.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, 2018.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019.
Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Ya Yu Zhang, Ming-Hsuan Yang, and Philip HS Torr. Res2net: new multi-scale backbone architecture. TPAMI, 2019.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR. 2013.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth estimation. ICCV, 2019.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for seff-supervised monocular depth estimation. In CVPR, pp. 2485-2494, 2020.
10

---

Under review as a conference paper at ICLR 2021
Junjun He, Zhongying Deng, Lei Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. In CVPR, 2019.
Hanzhe Hu, Deyi Ji, Weihao Gan, Shuai Bai, Wei Wu, and Junjie Yan. Class-wise dynamic graph convolution for semantic segmentation. ECCV, 2020.
Jingwei Huang, Yichao Zhou, Thomas Funkhouser, and Leonidas Guibas. Framenet: Learning local canonical frames of 3d surfaces from a single rgh image. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8638-8647, 2019a.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cc- net: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 603-612, 2019b.
Adrian Johnston and Gustavo Carneiro. Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume. In CVPR, pp. 4756-4765, 2020.
Tsung- Meing-Weiube Ke, Jyh-Jing Hwang, Ziwei Liu, and Stella X Yu. Adaptive affinity fields for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 587- 602, 2018.
Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. Structured attention networks. In ICLR, 2017.
Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajezyk, and Tim Fingscheidt. Self-supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance. ECCV, 2020.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. arXiv preprint arXiv:1606.0033 2016.
tae-Han Lee and Chang-Su Kim. Multi-loss rebalancing algorithm for monocular depth estimation. ECCV, 2020.
Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint 190.v:1907.10326. 2019.
Hanchao I., Pengfei Xiong, Jie An, and Lingxue Wang. Pyramid attention network for semantic
segmentation. arXiv preprint arXiv:1805.10111 10180, 2018.
Jun Li, Reinhard Klein, and Angela Yao. A two-streamed network for estimating fine-scaled depth maps from single rgb images. In ICCV, 2017.
Xia Li, Yibo Yang, Qijie Zhao, Tiancheng Shen, Zhouchen Lin, and Hong Liu. Spatial pyramid based graph reasoning for semantic segmentation. CVPR, pp. 8950-8959, 8909, 2020a.
Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun. Learning dynamic routing for semantic segmentation. In CVPR, pp. 8553-8562, 2020b.
Wang Lijun, Zhang Jianming, Huchuan Lu Yifan, Wang, and Ruan Xiang. Cliffnet for monocular depth estimation with hierarchical embedding loss. ECCV, 2020.
Chenxi Liu, Liang- CChihh Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, pp. 82-92, 2019.
Fayao Liu, Chunhua Shen, and Guosheng Lin. Deep convolutional neural fields for depth estimation from single image. In CVPR, 2015.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
Minh-
Minh-Thang Thang Luong, Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. In EMNLP, 2015.

---

Under review as a conference paper at ICLR 2021
Tom Minka and John Winn. Gates. In NeurIPS, 2009.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In NeurlPS, 2014.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong- Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In CVPR, 2014.
Seong-Whan Whan Lee, Sanja Fidler,
Xiaojuan Qi, Renjie Liao, Zhengzhe Liu, Raquel Urtasun, and Jiaya Jia. Geonet: Geometric neural network for joint depth and surface normal estimation. In CVPR, pp. 283 223-291,12 2018.
Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera mo- tion, optical flow and motion segmentation. In CVPR, 2019.
Anirban Roy and Sinisa Todorovic. Monocular depth estimation using neural regression forest. In CVPR, 2016.
Chang Shu, Kun Yu, Zhixiang Duan, and Kuiyuan Yang. Feature-metric loss for self-supervised learning of depth and egomotion. ECCV, 2020.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. ECCV, 2012.
Xibin Song, Yuchao Dai, Dingfu Zhou, Liu Liu, Wei Li, Hongdong Li, and Ruigang Yang. Channel attention based iterative residual learning for depth map super-resolution. In CVPR, pp. 5631- 5640, 2020.
Chiat-Pin Tay, Sharmili Roy, and Kim-Hui Yap. Aanet: Attribute attention network for person re-identifications. In CVPR, pp. 7134 1134-7143,20 2019.
Lokender Tiwari, Pan Ji, Quoc-Huy Tran, Bingbing Zhuang, Saket Anand, and Manmohan Chan- draker. Pseudo rgb-d for self-improving monocular slam and depth prediction. ECCV, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng. Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, et al. Deep high-resolution representation learning for visual recognition. TPAMI, 2020.
Zhihao Xia, Patrick Sullivan, and Ayan Chakrabarti. Generating and exploiting probabilistic monoc- ular depth estimates. In CVPR, pp. 65-74, 2020.
Tianjun Xiao, Yichong Xu, Kuiyuan Yang, Jiaxing Zhang, Yuxin Peng, and Zheng Zhang. The application of two-level attention models in deep convolutional neural network for fine-grained image classification. CVPR, 2015.
Saining Xie and Zhuowen Tu. Holistically-neste edge detection. In ICCV, 2015.
Saining Xie, Xun Huang. and Zhuowen Tu. Top-down learning for structured labeling with convo- lutional pseudoprior. In ECCV. Springer, 2016.
Dan Xu, Wanli Ouyang, Xavier Alameda-Pineda Elisa Ricci, Xiaogang Wang, and Nicu Sebe. Learning deep structured multi-scale features using attention-gated crfs for contour prediction. In NeurIPS, 2017a.
Dan Xu, Wanli Ouyang, Elisa Ricci, Xiaogang Wang, and Nicu Sebe. Learning cross-modal deep representations for robust pedestrian detection. In CVPR, 2017b.
Dan Xu, Elisa Ricci, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Multi-scale continuous crfs sequential deep networks for monocular depth estimation. In CVPR, 2017c.
12

---

Under review as a conference paper at ICLR 2021
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction- and-distillation network for simultaneous depth estimation and scene parsing, In CVPR, 2018.
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. In ICCV, 2019.
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and lan Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, pp. 340-349, 2018.
Fan Zhang, Yanqin Chen, Zhihang Li, Zhibin Hong, Jingtuo Liu, Feifei Ma, Junyu Han, and Errui Ding. Acfnet: Attentional class feature network for semantic segmentation. In ICCV, pp. 6798- 6807, 2019a.
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In CVPR, 2018.
Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie. Co-occurrent features in semantic segmentation. In CVPR, 2019b.
Li Zhang, Xiangtai Li, Anurag Amab, Kuiyuan Yang, Yunhai Tong, and Philip HS Torr. Dual graph convolutional network for semantic segmentation. arXiv preprint 1rXiv:19090000 2019c.
Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR, pp. 3726-3735, 2020.
Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon- Young Lee, Hailin Jin, and Thomas Funkhouser. Physically-based rendering for indoor scene understanding using convolutional neu- ral networks. In CVPR, pp. 5287-5295, 2017.
Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy. Dahua Lin, and Jiaya Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, 2018.
Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei- Shi Zheng, Jonathan Li, and Alexander Wong. Squeeze-and-attentioo networks for semantic segmentation. In CVPR, pp. 13065-13074, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In ICCV, 2019.
A APPENDIX: ADDITIONAL EXPERIMENTAL RESULTS
In this section we report additional quantitative and qualitative results.
Cityscapes. Table reports the results of VISTA-Net trained on the training+validation set and tested on the test set of Cityscapes dataset. We focus on comparing our method with recent attention
networks, including (Zhang al. 2019a Fu et al. 2019, Zhang al. 2020 Huang al. 2019b]
Choi et al,, 2020). This table confirms that our structured attention module is better than the other attention methods and sets the new state-of-hhe-ae on this dataset.
Surface normal Estimation. We conduct experiments on the ScanNet dataset. ScanNet is large scale RGB-D dataset for 3D scene understanding, we follow the protocol proposed in (Dai et al.) 2017) with split of 189.916/20,942 images for training and test respectively. The surface normal prediction performance is evaluated using five metrics. We compute the per-pixel angle distance between prediction and ground-truth, mean and median for valid pixels with given ground-truth normal. In addition to mean and median, we also compute the fraction of pixels with angle difference w.r.t. ground-trute toound-truth less than [11.25 22.5* 300 used in (Eigen et al.] 2014).. We compare
VISTA-Net with other state-of oftte-of-the-rr RGB-based methods including (Bansal et al.] 2016 Zhang
al. 2017 Qi al. 2018. Huang al. 2019a). Quantitative results shown in Table. Our
13

---

Under review as a conference paper at ICLR 2021
Table 4: Semantic segmentation results on Cityscapes test set (learned on the train+val set, multi- scale and flipping). D-ResNet- 101 is short for
Method
Backbone
mloU
iloU cla
IoU cut.
iloU cat.
DeepLab
Chen al.
1 2016a.
D-ResNet-101
70.4
42.6
67.7 58.8
86.4 90.8
PADNet
eu al. 2018
D-ResNet-101
80.3
78.5
Dynamic
La
20100
Layer33-PSP
80.7
SVCNet
Ding et ml 2015
ResNet-101
81.0
ANN
Zhu
D-ResNet-
81.3
CCNet
Hulig
al.
2019b
D-ResNet-100 101
81.4
DANet
Fü el al. 2019
D-ResNet-100 101
81.5
IXIMN
Zhang al. 2020
D-ResNet-
SpyGR
Li el al. 20203
ResNet-101
81.6
HRNet
Wang al. 2020
HRNaVV2 W48
81.6
61.8
92.1
82.2
ACFNe
Ziang et ai. 2019a
ResNet-101
81.8
NG
Zhang et il. 2019c
ResNet-101
82.0
Nel
Chh et al. 2020]
ResNext- 101
82.1
VISTA
HRNetV2- W48
82.2
62.7
91.9
method outperforms the FrameNet by more than 3 % in 11.25 and leaves behind by significant margin the other methods. In addition we show some qualitative examples in Fig. 6 These results clearly indicate that predictions extremely accurate also in the case of objects (i.e. waste bin, water closet, chair supports, etc.).
Depth estimation: qualitative results. In Fig. is shown qualitative comparison of our method with DORN (Fu et al. 2018). Results indicate that Distee-Negeessrttes better depth maps, in par- the sides.
ticular one can apprectate the opening of the sky and the smoothness of the prediction Fig shows a similar comparison done on NYU-D dataset. The same accuracy in the prediction is visible also in this case, objects
distinguishable w.r.t. DORN (e.g. the bathtub in row 2 and the desks in row 5). Finally, we also provided the computed attention maps some sample images in KITTI dataset Fig. [13] As expected, the final structured attention tensors manage capture important information among different depths. For example, in the fourth row, structured attention focus on farthest frost, middle jungle, and close road.
Semantic segmentation: qualitative results. In Fig. we propose a few qualitative results on Pascal-context dataset. In the figure is shown the importance of the attention model and the result obtained increasing the iterations. In the odd rows are shown the misclassified pixels (in black). The image shows clearly how the proposed iterative approach based on message passing is beneficial for the final prediction. Additional qualitative results Cityscapes dataset are shown in Fig. I0 Also in this case, the segmentation maps produced by VISTA-Netare more precise w.r.t those of the competitors (Fu et al.] 2019 Wang et al. 2020), in particular one notice the superior accuracy in the right hand side of the images where VISTA-Netshows better visual predictions.
Semantic segmentation: computational cost. In Table. we show the results of our experiments in order to analyze the computational cost of our method. In particular, we perform an analysis on the Pascal-context dataset and varying T. In the table FPS means Frames Per Second. As expected, when the rank increases we also observe an increase in term of parameters and reduction in term of speed.
Additional qualitative maps. Fig. 11 shows different visualizations regarding the learned struc- tured attention on an image from the Pascal-Context dataset. The first row shows the original image, together with four slices (channels) of the overall structured attention tensor as defined in ( The second row shows the T spatial maps of the structured tensor m While the latter seem to be spread all along the dog's 's body with different shapes, we observe that by optimally combining the m using the v²'s, different slices of the final structured attention tensor are able to on different important parts of the dog: the head, the body, the tail and the paws, thus allowing to take much more accurate pixel-level predictions for segmentation.
Meanwhile, Fig. 12 depicts segmentation maps obtained the Pascal-Context dataset using differ- ent versions of our method. In particular, we visualize (c) VISTA-Net w/o Attention, (d) VISTA-Net w/o Spatial Attention, (e) VISTA-Net w/o Channel Attention and (f) VISTA-Net (full model). From left to right, the results become more similar to the ground truth indicating the clear advantage of our proposed attention model.
14

---

Under review as a conference paper at ICLR 2021
Table Quantitative evaluation on surface normal prediction.
Error metric median
Accuracy metric
Methods
11.25
22.5
30
Skip Net
Bansal Net/aansal al.
2016
26.2
20.6
28.8
54.3
67.0
Zhanget ot. [Zhang et 2017
23.3
16.0
40.4
63.1
71.9
GeoNet 2t
19.8
11.3
49.7
70.4
77.7
FrameNer Huang a1 VISTA-Nei
2019a
15.3
8.1
60.6
78.6
84.7
15.1
7.5
63.8
80.0
85.2
Table 6: Pascal-context dataset: computational cost analysis for different values of T.
rank
IoU
pixace
parameters
FPS
0
54.2
80.4
45.80M
1.106
54.6
80.7
49.85M
1.075
3
54.8
80.8
52.68M
1.011
5
54.5
80.4
54.89M
1.068
7
55.3
81.1
56.79M
0.957
9
55.4
81.1
58.85M
0.868
Network detail. The overview of VISTA-Net is depicted in Fig. (a). From left right features are extracted from the backhone after layers 2 to and then are fed to the structured attention gate module (a detailed view is shown in Fig. [4 (b)) producing the attention maps. For each emitting and receiving scale (map), one structured attention gate is computed and exploited to gate the information from the emitting feature map the receiving feature map. inner view of the attention gate and its connections are shown in Fig. [4 (b). The blue cross & denotes the convolution operation while green cross refers to the conditional kernel prediction. The symbols @ and ( denote element-wise multiplication and addition, respectively while @ .. refer to the sigmoid function and feature concatenation. Finally the color code for arrows green for M-step, yellow for V-step and red for K-step. The algorithm is also described in Algorithm [
Algorithm 1: Our structured attention algorithm (VISTA-Net)
Input
Je - emitting feature, size is [B, C, H, W]
fr -rreeeiiing feature, size is [B. C, W]
Output:
- updated receiving feature, size [B, C, H]
feoncat + concat( f
L t- Conoodlfaonce
LISe-SS + Conv22d(Ja)
TI - Conv2d((r)
hee + unfold(fe)
her unfold(fr)
7 + aLL fc Lsc-ssr Rse + Lor- hrr)
for t t 1 to T do
A + randn (B, C,
Act + softmar Art))
10
Asp (Ach A)
11
App t sigmoid(Agg (Aop)
12
Act + Σ. E) (App A)
13
Act softmar (Act)
end for
16 frr f (L. hae) (Aor Asp A) + fr
return J.
15

---

Under review as a conference paper at ICLR 2021
(a) Image
(b) GT
(c) Prediction
(d) Image
(c) GT
(t) Prediction
Figure 6: ScanNet dataset. Qualitative examples on the surface normal prediction task.
(a) Image
(b) GT
(c) DORN
(d) VISTA-Net (ours)
Figure 7: KITTI dataset. Qualitative examples on the monocular depth prediction task. We per- formed bilinear interpolation on the sparse ground-truth truth depth maps for better visualization.
16

---

Under review as a conference paper at ICLR 2021
(a) Image
(b) GT
(c) DORN
(d) VISTA-Net (ours)
Figure 8: NYU-V2 dataset. Qualitative examples on the monocular depth prediction task. We
performed bilinear interpolation on the sparse ground-truth depth maps for better visualization.
Image
B/W Image
GT
wio attention
Rank-1
Rank-9
Imaage
B/W Image
Rank--9
Figure 9: Pascal-context dataset. Comparison of different variations of VIeTA-Net.
17

---

Under review as a conference paper at ICLR 2021
(a) Image
(b)
(c) DANet
(d) HRNet
(e) VISTA
Figure 10: Qualitative semantic segmentation results on Cityscapes dataset.
Figure 11: First row: original image and four channels (slices) of the structured attention tensor a, see Second row, the T attention maps m 1....,5. 5.
(a) Image
(b) GT
(c) w/o Att
(d) w/ Att.p
Atf
full (rank=9)
Figure 12: Pascal-context dataset. Net.
Segmentation maps using different attention models in VISTA-
Figure 13: Qualitative structured attention examples of monocular depth prediction on the KITTI raw dataset. First column is original image and next four columns is structured attention.
18

---

Under review as a conference paper at ICLR 2021
Input
(Layert)
Layer]
Layer2
Layer3
SAC
Task head
Oulput
Feuts rosk!]]
SAG
SAG
Layor4
SAG
Faselo)
I
h
Structure Attention Module
Acti
SM
f,
Aex
Ase
(b)
Figure 14: The overview of VISTA-Net (a). SAG is short for structure attention gate module. (b) is shown detailed view of the attention-gate model
19


Reviews:
[{"review_id": "k3R53120e8L", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel structure attention mechanism with a probabilistic CRF-like inference framework for the pixel-wise prediction, which not only models the spatial-wise dependencies but also considers the channel-level dependencies using an attention tensor. This attention mechanism is obviously different from the existing attention mechanism. Additionally, many quantitative and qualitative experimental results on three pixel-wise prediction tasks have demonstrated the superiority of the proposed method. The proposed attention mechanism is useful and flexible, which could be integrated into any pixel-wise prediction frameworks in theory. However, I think there exist some disadvantages:\n\n1. There are some unclear statements in the paper. For instance, the authors say “the attention tensor is nothing but the sum of T rank-1 tensors”, but how are these rank-1 tensors generated by the spatial attention map and the channel-wise attention vector? Is each rank-1 tensor associated with the special regions of different objects? If the authors visualize the attention map, it would be better to understand. Moreover, the overall architecture of the proposed method is not clear. The authors do not specify the details of the multi-scale features, including the dimensions of them, where they come from, etc. The dimensions of the most important variables are also not specified, which makes the paper is hard to follow.\n\n2. In the ablation study, when the model is ‘no structure’ on a single scale, does it denote the model does not integrate any attention mechanism? If so, when only considering the spatial or channel attention, why does it outperform DANet that includes dual attentions? If not, I think the comparison is not fair. \n\n3. Although the performance of the proposed attention mechanism outperforms most existing methods, the authors do not report the runtimes and parameters. Moreover, whether does it significantly burden the complexity of the model with the increase of T?\n\n4. There are some grammar mistakes and typos. For example, “Since the exact a posteriori distribution is not computationally tractable, ...”, “Again, our method not outperforms …”\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_3gW7h79Mg3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview\n\nThis paper carefully designs a structured attention network incorporating with a variational solution, to benefit the dense pixel-wise prediction via inferring the latent attention gate between spatial and channel-level features. Based on the suppose of T rank-1 attention tensors, the proposed structured attention network performs a CRF formulation with latent gating variables. \n\nStrengths\n\n1.\tExperiments are conducted on various dataset KITTI and NYU-v2 for depth estimation, PASCAL and Cityscapes for semantic segmentation, ScanNet for surface normal prediction. All the results demonstrate the outperformance.\n\n2.\tThe motivation is clear. Jointly estimating spatial attention maps and channel attention vectors with a probabilistic framework.\n\nWeakness\n\n1.\tThe contribution is limited. Just a combination of two existing works, e.g., CRF-based models for multi-scale attention estimation and DANet for spatial and channel-wise attention. Why the T rank attention tensors are advanced than full-rank ones? How to guarantee the T channels optimal than the possible C?\n\n2.\tThe generation of structured latent attention gate is confused. Compared with the work (Xu et al., 2017a), which illustrates the attention-gated CRFs with a clear graph, Figure 1 confuses me. Where is the gate? How to produce a probabilistically enhanced feature map? \n\n3.\tThe model is complex with four sets of latent variables (Z, M, V, and K) to be inferred. It is hard to be reproduced.\n\nQuestions:\n\n1.\tHow to choose the parameter T? Ablation study might be added to demonstrate the difference with different T.\n\n2.\tIn the inference stage, there are Z-step, M-step, V-step, and K-step. How to deal with so many variables to obtain the optimal performance? Is there any dependence on data size?\n\n3.\tFor experiments of depth estimation, why the proposed method obtains optimal metrics but rel?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S01pt94K-nR", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes the VarIational STructured Attention networks (VISTA-Net), which improves pervious SOTA models for dense pixel-wise prediction tasks. The proposed VISTA-Net is featured by two aspects: 1) A new structured attention is proposed, which is able to jointly model spatial-level and channel-level dependencies; 2) It incorporates the proposed structured attention with a CRF-like inference framework, which allows the probabilistic inference. Experimental studies are conducted on monocular depth estimation and semantic image segmentation, showing improved performances of VISTA-Net consistently.\n\n##########################################################################\n\nReasons for score:\n\nOverall, I vote for rejection. My major concerns lie in three aspects, as detailed in Cons below: 1) This work is highly similar to Xu et al. (2017a) in terms of both methods and presentations. The difference is not significant; 2) While the presentation mainly follows Xu et al. (2017a), it needs some improvement; 3) The experimental studies lack more detailed analysis on the proposed method.\n\n##########################################################################\n\nPros: \n\n1. The work is well-motivated. The aim of the proposed method sounds natural to me.\n\n2. I like the ablation studies. But they could be performed on at least one more dataset.\n\n##########################################################################\n\nCons: \n\n1. This work is highly similar to Xu et al. (2017a) in terms of both methods and presentations. The difference is not significant.\nMethod-wise, as discussed in Related Work, the difference only lies in that VISTA-Net takes channel-level dependencies into consideration. First, this means that \"Moreover, we integrate the estimation of the attention within a probabilistic framework\" (quoted from abstract) is not a novel contribution. Second, considering channel-level dependencies in attention has limited novelty. As discussed in Related Work, multiple studies have explored several ways. In addition, a key step in the proposed method is Equation (1), where the tensor multiplication operator is not explained. In my understanding, it should be the outer product, or more generally, Kronecker product. Missing the clear definition of this operator hinders the clarity in describing the proposed method.\nPresentation-wise, the similarity is even higher. The entire section 2 follows the exact organization of section 2 in Xu et al. (2017a). By comparing the equations and presentations, it's more convincing that the novelty of this work is quite limited.\n\n2. While the presentation mainly follows Xu et al. (2017a), it needs some improvement. First, the same notations as in Xu et al. (2017a) are used. However, some key things are not well explained. For example, the set of hidden variables $z_s$ corresponding to $f_s$ comes out from nowhere. I have to resort to Xu et al. (2017a) to know why we need $z_s$. Second, as mentioned above, the key steps like Equation (1) lack clear explanations.\n\n3. The experimental studies lack more detailed analysis on the proposed method. It would be more meaningful to visualize those attention maps/gates instead of dense prediction results.\n\n##########################################################################\n\nQuestions during rebuttal period: \n\nPlease address and clarify the cons above\n\n##########################################################################\n\nComments after the rebuttal period:\n\nPros:\n\nFirst, it is totally acceptable to follow the notations and organization of Xu et al. (2017a), as long as the statements are clear and self-contained. The original submission failed on providing key details. The authors have made revisions to address this concern. Thanks!\n\nSecond, I appreciate the extra experimental results and visualizations.\n\nCons:\n\nHowever, the authors' responses do not fully address my concerns about the novelty, especially method-wise. I will raise my score to 5, but still recommend rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GN46_RX6O81", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a unified method to combine spatial and channel attention in a probabilistic framework, so that spatial and channel attention weights and probabilistic variables can be jointly optimized. The proposed method is incoporated in the proposed VISTA-Net to achieve state of the art performance in two dense pixel-wise prediction tasks: monocular depth prediction, semantic segmentation.\n\nOverall this work is well motivated and organized in a good shape, so that I am on the positive side. I have some concerns as listed below.\n\n1. My main concern is that this paper combines ideas of (Fu et al., 2019) and (Xu etal., 2017a), i.e., combining spatial and channel attention from (Fu et al., 2019) and Attention-Gated CRFs from (Xu etal., 2017a). \n\n2. In Eq.1, the attention tensor is limited to T rank. However, it is not clear to me how this can faciliate the inference in Eq.9.\n\n3. In Eq.2, this paper proposes to additionally model CRF kernels. However, it is not explained why this is necessary. This is important as it is listed as a difference to existing method (Xu etal., 2017a). \n\n4. The approximation used in Eq.3 needs more details or reference to support.\n\n\nMinor issues.\n1. There are a lot of symbols in section 2. It would be better to draw a figure to include these symbols, while illustrating how the CRF formulation is incorporated in the attention mechanism.\n2. The joint learning in section 2.2 can be formed in a algorithm procedure, so that others could see it more clearly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers were concerned with the novelty, although appreciated sota results in extensive experiments.
Decision: Reject