Probabilistic Attention for Interactive Segmentation
Authors: Prasad Gabbur
Manjot Bilkhu
Javier Movellan
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Attention
Transformers
Probabilistic model
Gaussian mixture model
Interactive segmentation
Semantic segmentation

Abstract: We provide a probabilistic interpretation of attention and show that the standard dot-product attention in transformers is a special case of Maximum A Posteriori (MAP) inference. The proposed approach suggests the use of Expectation Maximization algorithms for on-line adaptation of key and value model parameters. This approach is  useful for cases in which external agents, e.g., annotators, provide inference-time information about the correct values of some tokens, e.g., the semantic category of some pixels,  and we need for this new information to propagate to other tokens in a principled manner. We illustrate the approach on an interactive semantic segmentation task in which annotators and models collaborate online to improve annotation efficiency. Using standard benchmarks, we observe that key adaptation boosts model performance ($\sim10\%$ mIoU) in the low feedback regime and value propagation improves model responsiveness in the high feedback regime. A PyTorch layer implementation of our probabilistic attention model is available here: https://github.com/apple/ml-probabilistic-attention.

Probabilistic Attention for Interactive Segmentation
Prasad Gabbur
Manjot Bilkhu Apple mbi mbilkhu@apple.com
Javier Movellan Apple
Apple pgabbur@apple.com
movellan@apple..
Abstract
We provide probabilistic interpretation of attention and show that the standard dot- product attention in transformers is a special case of Maximum A Posteriori (MAP) inference. The proposed approach suggests the use of Expectation Maximization algorithms for online adaptation key and value model parameters. This approach is useful for cases in which external agents, e.g., annotators, provide inference-ting information about the correct values of some tokens, c.g., the semantic category of some pixels, and need for this new information propagate to other tokens in principled manner. We illustrate the approach on an interactive semantic segmentation task in which annotators and models collaborate online to improve annotation efficiency. Using standard benchmarks, we observe that key adaptation boosts model performance (~ 10% mloU) in the low feedback regime and value propagation improves model responsiveness in the high feedback regime, A Py- Torch layer implementation of probabilistic attention model available here: https: http//gtttuuu/ com/ ccwp:pppppeeeal-oooooobaaaliiic.
Introduction
Attention was first introduced computational primitive for natural language processing and has since been widely adopted [17
13 14] as replacement for recurrent primitives such as LSTMs (26). More recently it has been making inroads into computer vision 4
63
69
19 49 as a replacement for the long accepted convolution as the main computational primitive. Self-attention based architectures have demonstrated ofate-of-the-art results in fundamental vision problems including image classification 5
43 69 19 49), object detection [8] [7 image and
video semantic segmentation [55 29 57
and tracking [66] to state a few.
There are a few different perspectives on the reasons for success of self-attention in computer vision and its superiority over convolution. This includes view that the self-attention mechanism allows modeling spatially varying dynamic convolution filters [2) and at the same time enabling parameter independent scaling of receptive fields [52). Another includes their ability to capture global context through long range interactions especially when full attention is feasible [49 at reduced spatial
resolution maps using an approximation of full attention with axial (55) or criss-cross attention
29 A recent work [45] introduces modern Hopfield networks with continuous states where the update mechanism shown to be equivalent to the update mechanism
standard dot-product attention [53]. They show that such network has the capacity to store exponentially many patterns and retrieve them with high fidelity. In this work, we provide a novel interpretation of attention as probabilistic generative model for queries and values. Specifically a bank of probabilistic memory units, each of which maintains joint probability distribution over queries and values parameterized through keys. A query/value pair is generated by first sampling unit (from a prior over units) followed by sampling the pair from the unit specific joint distribution. This is equivalent generating the queries and values through probabilistic mixture model over the units. particular form for unit joint likelihoods expressed as Gaussians for both the query and value marginals, assuming their independence conditioned on unit, turns out to be equivalent traditional
hypothesize the existence of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

dot product attention under few constraints. As shown in Section 3.3 inference for the corresponding value given a query is equivalent to standard dot-product attention.
maximum likelihood (ML)
Our probabilistic interpretation provides a systematic framework for online update of mixture model parameters based set of observed queries. It also allows propagation of correct values provided by an external agent for some of the units to all other units, Using Bayesian inference in the constrained case, we derive update rules for online unsupervised adaptation (Section 3.5) of query/key likelihood parameters based on a set of observed queries. We also derive update equations for online value propagation (Section 3.6 across units based on fixed externally specified values for subset of units. The latter is specifically useful for interactive segmentation where a correction provided by an annotator has to be propagated globally to make the process more efficient. We use probabilistic attention in place of standard attention in deep architectures for interactive segmentation both within the backbone and at the network head as a classifier. Specifically we use probabilistic attention updates in the BoTNNee500 [49] architecture and show that adapting keys to incoming queries leads to better model performance in the low annotator feedback regime. Using value propagation within probabilistic attention layer at the head of the segmentation network leads to more responsive model through effective feedback propagation in the high feedback regime. We also use both key adaptation and value propagation together and demonstrate the complementary effects of the two in both the low and high annotator feedback regimes.
2 Related Work
2.1 Attention
Natural language processing has seen the rise 53] and widespread adoption of attention recent years [7 63 13 14 59 One of the first works on visual attention was on learning to attend to image
regions for caption generation (60. Since then there has been a steady progress on using attention
primitives within vision models for recognition and classification [27 57 5
54
43
69
19 52, 49),
detection 88] 71), segmentation (55 29], tracking [66] and video analysis 442 41). There have been
numerous works interpreting the attention mechanism as form of computing non-local means 7 57],
approximating dynamic convolution filters [32 52], and capturing global context through long range also formulated as emerging from the update rule of modern Hopfield networks [45). Our work introduces attention mechanism from novel perspective as that of inferring from a probabilistic memory bank. To our knowledge, the only work that is closest to our approach is [8]. which also proposes a similar interpretation but only for the queries in order study the explaining away effect of attention update. Other notable works modeling attention in a probabilistic framework have done so using neural processes [25] variational approximations for modeling alignments latent variables [16] or to overcome the bypassing phenomenon of deterministic attention [4] in sequence tasks. Being different from these approaches, our formulation encapsulates queries and values in single generative model and provides an interpretation of standard dot-product attention as constrained Bayesian inference. Doubly normalized attention scheme [8] also emerges as a special case of key adaptation in our framework.
interactions [27 43 55]. The standard dot dot-product attention [53] update
2.2 Interactive Segmentation
Deep neural networks have set state-of-the art in semantic segmentation through the use of fully convolutional architectures 33 70
12 50 56 67 and more recently using hybrid convolution and self-attention (64) or stand-alone self-attention architectures (55)) The input domain of interactive segmentation includes user input in the form of clicks or scribbles in addition to the visual signal (images or videos). The earliest works in interactive segmentation were based on algorithmic approaches for incorporating human inputs into region [46] ST 65 or boundary [40 20 processing pipelines. [44] provides comprehensive survey of interactive segmentation approaches. More recently deep networks have been used to incorporate user feedback to guide their output predictions at the pixel level. Following a similar taxonomy, these can be roughly categorized into region based
62 61 37
33
2
28
68
34
31 or boundary based approaches g 3535 58) Deep Extreme
Cut (DEXTR) 38 demonstrated that user guidance in the form of extreme points could be used
in addition to the input channels to accurately localize the object of interest. More recently [68] argued that three points are sufficient as input guidance to localize the object but additional corrective
2

---

clicks could be used to further refine the prediction. Other works have used the corrective clicks adapt the network imputs [30], embeddings 48 or parameters [3] online. Different from these previous approaches, we use corrective clicks as providing fixed values for subset of units in the proposed probabilistic attention framework. These values are propagated globally through the attention mechanism to directly and more effectively influence the outputs towards user intended values.
Method
We provide a probabilistic interpretation of attention generative model for queries and values through a set of memory units. Using this formulation, traditional attention in transformers [53] reduces to the special case of maximum a posteriori (MAP) inference of values given queries, assuming Gaussians for the likelihoods. Using Bayesian inference, we provide a systematic approach adapt keys online as locally ML update of the corresponding model parameters. Our formulation also allows to fix the values of certain units and propagate their influence to other units online by conditioning on the fixed unit values. The following sections provide more details on the probabilistic model.
3.1 Probabilistic attention
We assume that there vector E Rdd to yield an output value vector e; E R!" The queries and the corresponding values may depend The joint distribution of queries and values conditioned on the input æ is assumed to factorize over memory units
memory units, indexed by i, each of which be queried through
input æ. For example, each memory unit may represent pixel æ; in image
p(q [2) = 1P: Pi Us Ix),
where the conditioning input, fı= = fq..... ]. Pim = {vi. )n and qi Rd, o R are the query, value vectors for unit i respectively. The per-unit joint likelihood p;(4s, 0 x)) is probabilistic mixture model given by
= E = Epla,,u, = Σ (i)(#) pe(a. xy,
where we have dropped the subscript from q and for simplicity. the above, ú, indexes unit (T. is the probability of activating unit when unit is queried, pi(g. v uy. is the likelihood of observing the pair (g. v). given the pair is generated through unit j in the mixture, conditioned on the input x.
3.2 Value inference
Using the above model, it is possible to find the most likely value è given query q unit
è argmax (v)q..x).
(3)
We use Expectation Maximization (EM) the most probable value and iterating over the standard EM auxiliary function Q.. Given the latest known estimate v', the M step produces a new estimate w1+1 that increases Q; by maximizing w.r.t. w+1 This guarantees local maximization of ps (s(vgg,
achieve this: starting with an initial estimate n of
Qelv',utt *+1 (x) = Σ wi, log pi(u,, 1+1 12),
(4)
where
Ti. (x) pelg, v² (u,,)))
w pi(uu 19.0 x0))
(5)
E, Tey (x) pa(g, |uy,x)
3

---

The n x n matrix formed by the entries wi, corresponds to the attention matrix in standard trans- formers. The optimal value é is obtained by taking the gradient with respect to g and setting it to zero
ut+ Qi(0', 2 x)) Σ Wi, Veets log logpi(q u Ug |x),
(6)
where V,44a log Ps v4+1; Uy x)) is the Fisher Score for unit with respect of+1
3.3
Relationship to standard attention
We show that standard attention in transformers [53 solves Eq. (6) under the special case of constrained Gaussian mixture model (GMM). Assuming isotropic Gaussians with conditionally independent queries and values given input and mixture component
pi v x)) pi(g
uj,
p;(t) (j,
pi(4 ui
(z(2z)
d/2
-
(
2T
(g(2)
m/2
3360) (1-1,(0011
pi(v] uy,
2T
where (zx are precision parameters, {z(z) Rd 11, R are the key and expected value parameters for unit j given the input æ. The dependency Pr(g, Uy. x) on x is through the function of x. For simplicity, we treat to be fixed and leave the dependency on x implicit in our notation. In order to obtain the standard attention update equation, constrain the precision parameters be the same across units: an
fact that the parameters ozj [3 Ti.j. E M4
On a. 31 3n = 3, and link the priors of each unit to the lengths of the corresponding key and expected value vectors
Tijj - 1l e msll?
1
(10)
e |eyll 1l
(11)
Assuming - 0 and solving for optimal wi in Eq. 6 (see Appendix(B.I
obtain the standard attention update
r+ Σ WigHt
(12)
caE,
w1, E, C
(13)
where each μy is the value associated with unit and u is the output at unit or token i after the attention update. In this case, wf,, is longer function of and thus only one EM iteration is needed.
3.4 Offline supervised learning
As is commonly done in standard transformers, the relationship between the input ir and the mixture model parameters: x <<x),{(x) μ(x) can be modeled using deep network, whose parameters can be trained off-line with task specific supervision.
3.5 Online unsupervised mixture model adaptation
Our framework provides a way to adapt the mixture model parameters based on all the observed input queries prior to doing value inference. This process can be seen as an inference-time adaptation of the model using the additional information contained in the queries. We propose an unsupervised Bayesian approach to do this adaptation for the per-unit key vectors εi. -...-.., . and

---

precision parameters 01: optimal value inference given by
fa1...
an) given queries 9i. = {91, , -4. For each unit i, the
0 argmax ps q1:.).
(14)
Assuming prior for the key vectors given the observed queries p(Eim Iqım), the likelihood (: gmm)) can be written as
Ps(v (1:)) = p(E1.0 (1m));(( Os: Samn)dn ~ ps( qa fim),
(15)
where the expectation over the posterior is approximated by its maximum posteriori (MAP) value
Si. = argmax p(1::n (mma)
(16)
mm
0. argmax ( q:: (im).
(17)
In order solve (16). we use an iterative EM approach. The initial key parameters Si. are provided by the pre-trained model. To avoid overfitting to the current query vectors, we use a Gaussian prior centered on the key parameters provided by the pre-trained network, i.e., Si.m. with finite precision 0 > 0. The EM update for the key parameters at any iteration i is given by (see Appendix B.2)
0gfh + ork Ei=1 W 0g 0 R W
ct+1
(18)
Analogous to the keys, can also adapt the aj precision parameters (see Appendix B.3
3.6 Online value propagation
The proposed model allows for fixing the outputs of selected subset of units predefined values and letting them propagate to other units in a principled way. This aspect of our model is of particular interest to interactive semantic segmentation, where human annotator provides corrections the output of a semantic segmentation model. In this case, assuming an attention layer at the output of deep model, the memory units correspond to pixels and the output values correspond to the semantic label for that pixel, e.g. foreground or background. Based on the network's 's prediction, an annotator provides corrections for subset of the pixels, which are the ground truth for those pixels. These correspond to the fixed predefined values for those units, whose effect is to be propagated to semantically similar pixels globally across the image make the process more efficient. More formally, suppose the annotator has provided the correct values for the first s < n units. We want for this information improve the inference about the value for all the other units s. Within our framework, this inference is given by
0 argmax Pb(U) 9t. 91:ns U1:s), for ci<=n. n.
(19)
In order do this inference, we adopt Bayesian approach similar model adaptation of Section 3.5 Let À represent the set of network parameters, e.g., T, T,E, a, B. Writing the inference as an expectation over the model posterior p(> 9i:n; V1:s)
(1(0 G1:ns u1:.) = p(A Iimn U1:s Ph:u) gt. A)dA zs p.(v 94 2),
(20)
where we approximate the expectation with its MAP estimate as before
X argmax p(A qrens V1:s)
(21)
6 argmax (M qi 2).
(22)
Eq. [1 is solved using EM. Specifically, value propagation across units is achieved by updating the
Hg for each unit & starting with the initial value u provided by the pre-trained model (Section 3.4).
Following a similar approach as in Section 3.5 the EM update for pk
t+1 at iteration is given by
0 0 1 + 3r Dim1 wh
x+1
(23)
9, BR Σ -== w(t
Dim
Te.k p(gs uk. Ek) p(e: uk. up.
whe (14 qi, Vt, m(.))
(24)
Σラ Ti p(g: ugj {x) p(u: Ujj k
where 0, is the precision for the Gaussian prior over each Mkk See also Appendix B.4
5

---

3.7 Combining offline learning and online adaptation
The inference time adaptation of parameters differentiable. So it can be included as part of the traditional supervised optimization egg. via stochastic gradient descent and used to learn the parameters of the prior distributions over /4 at U,, n.
3.8 Position embeddings
Positional embeddings 533 47 are useful in attention models to encode the relative or absolute positions of tokens. In computer vision applications, relative position embeddings have been found to be critical to capture the interactions between features based on their pairwise positional relations [43 49 55]. We propose to encode relative position embeddings by introducing extra parameters in the per-unit likelihoods of the mixture components and their priors. Let r and r denote the relative position embeddings for query and key interacting at units and respectively. The query/key marginal with the position embeddings is given by (see Appendix B.5
1 Ia) x N(g
pi(a Esrrj-s uz) x N( Ej. aj
La)a(a re-i 0j
Ia).
(25)
2
20j
where N(a b, c) is the Gaussian likelihood function over with mean b and covariance matrix la d d identity matrix. The mixture component priors with position embeddings take the form
Taj (211e)1² 111--111 + 1r3-.113) exp
Bg |,s)11]
Ty x N(S 1- Ta)exp
(26)
2
4 Experiments
In this section, we report the results of using probabilistic attention at various stages of deep interactive semantic segmentation network. Specifically, we use it within the BoTNet50 backbone 49 in place of standard attention and also part of self-attention based classification head at the network output. We quantify model performance (mean IOU relative to ground truth) as a function of the number of clicks [6] on two widely used public benchmarks for this task: GrabCut (46) and Berkeley (39). Appendix proovides more details on the interactive segmentation model architectures, training and evaluation protocols.
4.1 Probabilistic attention within backbone
We adopt the recent work on BoTNet 44) by replacing the convolutional layers with attention layers in the last bottleneck block (c5) of the ResNet50 (24 architecture. Specifically,
probabilistic attention layers in place of standard attention using either full or axial (55) attention. We experiment with either factored 49 or full relative positional encoding. Factored encoding uses (H + W)d parameters for an image size H, W) factoring them along the height and width dimensions, whereas full encoding uses 2(H W)d parameters, per relative offset. Our models are trained on the LVIS (21 dataset at a resolution of 256 pixels. The results are shown in Fig. [ The results suggest that using probabilistic attention in the Boo backbone leads to better performance especially for smaller number of clicks. This is true with both full and axial attention BoTNe using probabilistic attention. Using full relative position encoding helps more than using factored encoding perhaps due to the larger number of parameters.
4.2 Key adaptation
We experiment with unsupervised model adaptation described in Section 3.5 by adapting the keys online (Eq. (18) based on the observed queries. The degree of adaptation is controlled by the prior precision parameter 9g with lower values leading higher degree of adaptation due to the lower weight on the prior keys. Using the probabilistic attention BoTNet50 backbones of the previous section, we experiment with and without key adaptation. With key adaptation, we use two different values of the precision prior, 0.001 and O, with the latter corresponding to maximum likelihood update of the keys given observed queries. The results in Fig. showww the mean IoU as function of number of clicks using the ProbBoTTNe550-Facttrrr
model. We observe that key adaptation leads

---

0.
0.7
0.7
ResNet50
ResNet50
0.6
Bo TNNet50-FactoreePP
BoTNee50-FactoredPE
Bo Fulle
BoTNet5O-FullPE
ProbBoTNetSO- FactoreedPE
ProBBoTNet50-Factoreee
ProbBcTNetSO-FullPE
ProbBoTNet50-FuIIE
0.4
ProbAkia
ProbkklaBBoTNe550FFuIPP
0.1
26
20
1s 18
Number of Clicks
Number Clicks
(a) GrabCut
(b) Berkeley
Figure 1: Probabilistic attention layers in BoTNet architecture. Mean IoU function of #clicks using different attention layers, position embeddings and full or axial attention the BoTNet architecture. These compared against their fully convolutional conterpart ResNet50. The left and the right plots correspond to the GrabCut and Berkeley datasets respectively.
0.9
0.5
0.7
ProbBoTNet5 O- FactoreedPE
0.6
ProbBeTNet50 FactoredPE
ProbooTNet5 DaaatoorePP BecetoredPE-KAA-PPee KKA--Pre00.00
Prob0ciNe:50 FactoredPE: Peebbee/NetOOFacttrreee KAL Preco.oo
ProbBeTNet50 Faattordee BiiendPPE---APPPP Precc.0
0.4
0.5
20
12 Number of Clicks
Number of Clicks
(a) GrabCut
(b) Berkeley
Figure 2: Unsupervised key adaptation. Mean IoU vs #clicks with and without key adaptation (KA) the GrabCut and Berkeley datasets. Probabilistic BoTNets with factored position encodings are evaluated without using KA or using iteration of KA with two different prior precision (Prec.) values of 0.001 or 0.
higher IoUs without any corrective clicks or using only a few corrective clicks. Specifically there is an absolute improvement of about 10% in mean loU using key adaptation and without using any corrective clicks. Additional results using ProbBoTNet5O-Fulll
are shown in the the Appendix D Using lower value of prior precision seems beneficial and the extreme case of maximum likelihood adaptation leads to the best performance. Note that this effect has been observed in previous work [8], where it is perceived as a doubly normalized attention scheme (DNAS). This can be attributed to the unsupervised model adaptation accounting for the small domain shift introduced by models GrabCut and Berkeley datasets. Without using key adaptation
trained LVIS and evaluated additional user input in the form of corrective clicks is required to account for this shift as can be seen by the asymptotically similar behavior with increasing number of clicks.
4.3 Probabilistic attention as classifier
10 for semantic segmentation have a fully connected (Ix1 conv) layer as their classification head. We replace this layer with corrective self-attention [53 module
Deep architectures 36
23
7

---

that takes corrective click locations as additional inputs to more effectively propagate corrections as follows.
Corrective self attention. We append the corrective channels to the features of the penultimate decoder layer and feed them as inputs the value embedding layer (see Fig. B) The query and the key embeddings do not use the corrective channels as their inputs, which allows attention maps to be computed based only on the semantics captured by the features. However, the weights of the value embedding layer can be trained to output the desired labels at the locations of the corrective clicks. Fig. 3 shows block diagram of our corrective self-attention (CSA) layer. We choose to use probabilistic self- self-attention for computing the output values.
HcWa26,
H&Wal
HeWwo
HeWNCp
(Height)
Axial Attention (Width)
R
JPM 2
Updets
Artention Upcele
HoWe?
Figure 3: Corrective self attention layer. We propose a self-attention based classification head at the output of the decoder to more effectively propagate corrective clicks. Cin. channels from the penultimate decoder layer are reduced to CB by bottleneck layer with weights WB. These are input to pair of densely connected Axial Attention [55] modules along height and width dimensions to produce the output logit at full image resolution (H W). The corrective channels (P and N) are fed only to the value embedding functions (W and WW) ) of the attention modules to propagate the corrections more effectively.
The CSA layer is used at the network output by up-sampling the final feature map of the decoder to the input image resolution and appending positive and negative corrective channels containing only the click locations. The local context size for Axial attention modules is chosen to be 64 pixels. The output of the axial attention block is passed through a sigmoid estimate the pixelwise probabilities of the object mask.
4.4 Value propagation
We demonstrate the effect of propagating annotator feedback across pixels using online value propagation as described in Section We the CSA layer described above in place of the 1x1 conv classifier head of the HRNetV2+OCR [67 architecture pre-trained on Imagenet classification and fine-tuned for interactive segmentation on SBD [22 resolution of 256 pixels. Note that value propagation requires learning one additional parameter per output class (2 in our case) to estimate the fixed logit that corresponds the annotator feedhack for that class the corrective locations. These are learnt as part of network training using gradient descent. Following standard protocols, test on the GrabCut (46) and Berkeley (39 datasets. Fig. 4 shows the effect of using different number of value propagation iterations (1 and 5) within the probabilistic attention layer. Clearly, value propagation leads more effective propagation of labeler feedback relative to not using it. For this experiment, we do value propagation only in the output width block of the CSA layer (Fig. (3) as we found that doing in both the height and width layers did not work so well in our experiments. We hypothesize that this is probably due to the difficulty in learning the high dimensional fixed parameters corresponding to annotator feedback the height block of the axial attention layer.
4.5 Combining key adaptation and value propagation
In this section, we experiment with combining key adaptation and value propagation in single model. For this experiment we use BoTNe architecture with a corrective self-attention classification head at a resolution of 256 pixels. We use axial attention in both the backbone and the classification head with probabilistic self attention updates. The model trained on LVVIS and evaluated on GrabCut and Berkeley datasets. Fig. shows the effect of using either key adaptation or value propagation
8

---

0.94
0.90
0.925
0.500
0.88
0.875
0.86
0.550
ProbAxialAtE-noBP ProbAxialAtt-BP1
ProbAxialAtt-noBP
0.525
ProbAxialAtt-BPI
0.800
ProbAxialAt--PP
0.2
ProbAxialAt-8P5
26
20
10
28
Number of Clicks
Number of Clicks
(a) GrabCut
(b) Berkeley
Figure 4: Effect of value propagation using Axial attention. Mcan IoU #clicks with and without value propagation using an axial attention based probabilistic CSA layer at the output. We use (BP1) and 5 (BP5) iterations of value propagation at the CSA layer and test the GrabCut (left) and Berkeley (right) datasets.
0.55
0.80
0.55
1991
0.80
0.75
0.70
TrobAxiaBoTTNe:55 ProbAaiiiAAtCCSsssssooooo -oo.alAAAtttt--ooooooooo0
-+-
ProbAxisiBo Tatt:iieTTtttt-ooAArat Prw.AAoololoASSSSS.o.... KA1
0.65
ProbAxislBoTNett ProbAstllAtCis
PeaasualBeMeettt-OPoaoalllllld KA1-8P5
ProbAxia BoTTle:5 ProbAkislAUCIS KAL BP5
040
12
34
16
18 20
13
12
14
18
20
Number of Clicks
Number of Clicks
(a) GrabCut
(b) Berkeley
Figure 5: Effect of combining key adaptation and value propagation. Mean IoU vs. #clicks using one or both key adaptation and value propagation single model. Key adaptation run for iteration (KA1) and value propagation for 5 iterations (BP5) when either or both are used.
both relative to not using them. As observed separately in the previous plots, key adaptation helps in the small #clicks regime whereas value propagation shows greater benefits with increasing #clicks. Using the two jointly allows the model to respond quickly to annotator feedback in both the regimes.
Conclusion
We provide probabilistic interpretation of the attention mechanism in transformers as generative mixture model over queries and values parameterized through keys. Using our framework, the attention update is maximum posteriori inference over values given queries. Specifically, the standard dot-prooduct attention is a special case assuming Gaussians for the mixture likelihoods and few other constraints. Using Bayesian inference, our interpretation allows for online update of the mixture model parameters as well as the propagation external agent. Although we demonstrate the utility of these aspects on the problem of interactive segmentation, the proposed model is generic and can be extended other domains with suitable distributional forms for the mixture likelihoods.
set of fixed values specified by an

---

Acknowledgments and Disclosure of Funding
We would like to thank Russ Webb, Joshua Susskind, Parisa Assar, and numerous other colleagues for their helpful discussions, draft and code reviews.
References
[1] D. Acuna, H. Ling, A. Kar, and S. Fidler. Efficient interactive annotation of segmentation datasets with polygon-mnn+. In CVPR, 2018.
[2] E. Agustsson, J. R. Uijlings, and V. Ferrari. Interactive full image segmentation by considering
all regions jointly. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11614-11623, 2019.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ArXiv, 1409.0473, 09 2014.
[4] Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. Variational attention for sequence-to--equeeee sequence models. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1672-1682, August 2018.
[5] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V. Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[6] R. Benenson, S. Popov, and V. Ferrari. Large-scale interactive object segmentation with human annotators. In CVPR, 2019.
[7] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. A non-local algorithm for image denoising. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 60-65, 2005.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[9] L.. Castrejón, K. Kundu, R. Urtasun, and S. Fidler. Annotating object instances with a polygon- mn. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 4485-4493, 2017.
[10] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Murphy Kevin 0002, and L. Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence.
pages 834, 2018.
[11] Liang-Chieh Chen, G. Papandreou, Florian Schroff, and H. Adam. Rethinking atrous convolu- tion for semantic image segmentation. In Proceedings The IEEE Conference on Computer Vision and Patlern Recognition, 2017.
[12] Liang-Chiee Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[13) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-Xl:: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, July 2019.
[14] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. In International Conference on Learning Representations, 2019.
[15] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39:1-88 1977.
[16] Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M. Rush. Latent alignment
and variational attention. In NewrlPS, pages 9735-9747, 2018.
[17] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
[18) Nan Ding, Xinjie Fan, Zhenzhong Lan, Dale Schuurmans, and Radu Soricut. Attention that does not explain away, 2020.
10

---

[19]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[20] Alexandre X. Falcao, Jayaram K. Udupa, Supun Samarasekera, Shoba Sharma, Bruce Elliot Hirsch, and Roberto de A. Lotufo. User-steered image segmentation paradigms: Live wire and live lane. Graphical Models and Image Processing, 60(4):233 260, 1998.
[21]
Agrim Gupta, Piotr Dollár, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation, 2019.
[22] Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In International Conference on Computer Vision (ICCV), 2011.
[23] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[24] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. 6. [25] Jay Heo, Junhyeon Park, Hyewon Jeong, Kwang Joon Kim, Juho Lee, Eunho Yang, and Sung Ju Hwang. Cost-effective interactive attention learning with neural attention processes. In ICML, 2020.
[26] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation. 9(8):1735-1780 1997.
[27] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[28] Y. Hu, A. Soltoggio, R. Lock, and S. Carter. A fully convolutional two-stream fusion network for interactive image segmentation. Neural Networks, 109:31-42, 2019.
[29] Zilong Huang. Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[30] Won-Dong Jang and Chang-Su Kim. Interactive image segmentation via backpropagating refinement scheme. In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[31] Theodora Kontogianni, Michael Gygli, Jasper Uijlings, and Vittorio Ferrari. Continuous adaptation for interactive object segmentation by learning from corrections. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[32] Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Involution: Inverting the inherence of convolution for visual recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
[33] Z. Li. Q. Chen, and V. Koltun. Interactive image segmentation with latent diversity. In CVPR. 2018.
[34] Zheng Lin, Zhao Zhang, Lin-Zhuo Chen, Ming-Ming Cheng, and Shao-Ping Lu, Interactive image segmentation with first click attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[35] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object anno- tation with curve-gen. In 2018 IEFF Conference on Computer Vision and Pattern Recognition, 2019.
[36] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
[37] Sabarinath Mahadevan, Paul Voigtlaender, and Bastian Leibe. Iteratively trained interactive segmentation. In British Machine Vision Conference (BMVC), 2018.
[38] K.-K. Maninis, S. Caclles, J. Pont-Tuset, and L. Van Gool. Deep extreme cut: From extreme points to object segmentation. In Computer Vision and Pattern Recognition (CVPR), 2018. [39] D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2, pages 416-423, July 2001. [40] Eric N. Mortensen and William A. Barrett. Interactive segmentation with intelligent scissors. Graphical Models and Image Processing, 60(5):349 - 384, 1998.
11

---

[41] S. W. Oh, J. Y. Lee, N. Xu, and S. J. Kim. Space- time memory networks for video object seg- mentation with user guidance. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020.
[42] Rizard Renanda Adhi Pramono, Yie-Tarng Chen, and Wen-Hsien Fang. Hierarchical self- attention network for action localization in videos. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019.
[43] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self- eeff-attention in vision models, 2019.
[44] H. Ramadan, Chaymae Lachqar, and H. Tairi. A survey of recent interactive image segmentation methods. Computational Visual Media, pages 1 30, 2020.
[45] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp. Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. Hopfield networks is all you need. In International Conference on Learning Representations, 2021.
[46]
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "grabcut" interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics, pages 609-314,20 2004.
[47] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pages 464 468,,68 June 2018.
[48] K. Sofiuu,, I. Petrov, O. Barinova, and A. Konushin. F-brs: Rethinking backpropagating refinement for interactive segmentation. In CVPR, 2020.
[49] Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition, 2021.
[50] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[51] Meng Tang, Lena Gorelick, Olga Veksler, and Yuri Boykov. Grabcut in one cut. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2013.
[52] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones, 2021. [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in Neural Informa- tion Processing Systems 30, 2017.
[54] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[55] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and I Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In European Conference on Computer Vision (ECCV), 2020.
[56] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep high-resolution representation learning for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[57] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[58] Zian Wang, David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Object instance annotation with deep extreme level set evolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[59] Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less atten- tion with lightweight and dynamic convolutions. In International Conference on Learning Representations, 2019.
[60] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 2048-2057, 07-09 Jul 2015.
[61] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas Huang. Deep grabcut for object selection. In Proceedings of the British Machine Vision Conference (BMVC), 2017.
12

---

[62] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas S. Huang, Deep interactive object selection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[63]
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in Neural Information Processing Systems, volume 32, 2019.
[64] Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In ECCV, 2020.
[65] H. Yu, Y. Zhou, H. Qian, M. Xian, and S. Wang. Loosecut: Interactive image segmentation with loosely bounded boxes. 2017 IEEE International Conference on Image Processing (ICIP),
pages 3335-33 13399, 2017.
[66] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R. Scott. Deformable siamese attention networks for visual object tracking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[67] Y. Yuan, X. Chen, and J. Wang. Object-contextual representations for semantic segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2019.
[68] S. Zhang, H. Liew, Y. Wei, S. Wei, and Y. Zhao. Interactive object segmentation with inside-outside guidance. In CVPR, 2020.
[69] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[70] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[71] Xizhou Zhu. Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable (detr]: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.
13


Reviews:
[{"review_id": "fYbiCf8o0u", "reviewer": "Reviewer_iso7", "summary": "The paper presents a generative view of attention: N hidden memory units each of which generates observed a query and a value. The memory units densely connected with each other. The paper shows that, given queries, the value of a memory unit, in a certain condition, is computed in a similar way as in attention, using EM. \n\nApplying this new attention, called probabilistic attention, to interactive image segmentation tasks (where users provide some ground-truth pixels' values on the fly), the paper shows better results than using traditional attention on GrabCut and Berkeley datasets. \n\n**Contributions** \n1. novel explanation of attention, leading to new attention mechanism relying on probabilistic ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality** \n\nExplaining attention is not new but using probabilistic generative view in this paper is novel and very interesting. It will open a door to a new class of attention layers that have wider probabilistic properties. \n\nI particularly like the trick of letting beta -> 0 in section 3.3. This is counterintuitive to me as beta -> 0, eq 9 implies that v is more and more independent of all u_j. Turns out, in the end, we have traditional attention. Also, line 142, it is interesting that we just need to use one EM update, exactly like the attention. \n\n\n**Quality** \n\nThe paper technically is sound. I found the maths correct (though some steps are missing, such as from B3 to B4 where beta1=beta2=...=beta is crucial). \n\nI would like to see more in the paper: \n1. What is the effects and use of EM? It's unclear to me whether EM is used together with training the networks for priors. If it is, how? \n2. Interactive image segmentation looks like a good tasks for attention. The authors claim this probabilistic attention is more generic applicable. Is there another successful story? Is it simple just to pickup a task, a model which uses attention, and replace that attention by the probabilistic attention? Is tuning precision prior important?\n\n**Clarity** \n\nThe submission is well written. \n\n**Significance**\n\nThe results shown in the paper is important as now we can have more freedom to extend attention, especially in probabilistic ways. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative societal impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7h"}, "extra_text": {}}, {"review_id": "OkWztLNdmyk", "reviewer": "Reviewer_k1NM", "summary": "Summary and Contributions\n\nThe authors study a probabilistic interpretation of attention mechanisms for interactive segmentation.  The goal is to show that self-attention is related to MAP and EM can be used to induce the posterior distribution over attention scores.  The main contributions are: probabilistic interpretation of attention as a generative model, interactive segmentation with the proposed attention model, and classification.  The study shows that the proposed model has competitive results compared to related work.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\nClear description of background knowledge, proposed model, and related work needed to understand the proposed model.  \nThe authors perform a comprehensive comparison across different tasks.\n\nWeaknesses\n\nIt is not clear the motivation to  define attention as a generative model. \nIt is not discussed the possible difference of the proposed models with variational attention models.\n\nQuestions to the Authors\nPlease address the following questions during the rebuttal:\n\nWhich is the relation of the proposed model and variational attention models used in NLP applications? e.g. Variational Attention for Sequence-to-Sequence Models\nA possible extra contribution could be the discussion of the proposed model for text classification tasks. e.g. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors addressed  limitations.\n", "review_text": "", "extra_scores": {"ethical_concerns": "I do not have concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "G_BnryUP19y", "reviewer": "Reviewer_oG8X", "summary": "This paper offers a new interpretation of standard dot-product\nattention as a mixture model. The model assumes memory units are\nlatent variables that dictate the probabilistic interaction between\nquery and value vectors. The significance of this probabilistic\ninterpretation is twofold:\n\n1. MAP inference under a carefully constrained Gaussian mixture model\n   case of the model yields exactly the update rule of standard\n   dot-product attention. Prior to this the most similar\n   interpretation of attention was through modern Hopfield networks\n   (Ramsauer et al., 2021).\n2. The model admits an EM algorithm which allows for either online or\n   offline learning in the precence of partial annotations. \n\nThe paper applies this probabilistic attention approach to interactive\nsemantic segmentation by incorporating it in various ways within\nBotNet. The probabilistic attention either replaces standard attention\nor is used within the self-attention at the last part of the\nnetwork. Under various settings (e.g. full or axial attention;\nfactored or full relative positional encoding), the incorporation of\nprobabilistic attention improves over the corresponding baseline. \n\nThe paper finally introduces a corrective self attention layer in\norder to more effectively pass along information from an annotators\ncorrections of the model output. Results show that such a use of these\ncorrections is more effective than not.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper presents a novel interpretation of standard dot-product attention as a probabilistic (mixture) model.\n- Bayesian inference in the model enables online updating of the\n  mixture parameters and corrections on a subset of values to be\n  propagated.\n- The results on interactive semantic segmentation suggest that the\n  approach is a drop in replacement for attention that can improve\n  over baselines without.\n\nWeaknesses:\n- From the initial description of the mixture model, it was a bit difficult to understand *why* it would make sense to have latent memory units at all. Only later upon thinking what these were being used for within the context of interactive semantic segmentation did this become clear.\n- The probabilistic attention mechanism is only applied to one task (interactive semantic segmentation). It would be interesting to see its effectiveness in other settings.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "-qYO1Ju30P", "reviewer": "Reviewer_nVNQ", "summary": "This paper describes an EM-based approach to propagating attention parameters, showing strong performance on an interactive segmentation task.", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a probabilistic attention model which contains standard transformer attention as a special case, along with an online value propagation method which allows humans to update attention weights in a principled manner. This is evaluated in the domain of interactive segmentation but could seemingly be applied to other domains such as interactive machine translation. This paper is not in my area, but to the best of my knowledge the approach for attention propagation is novel and potentially applicable to a wide number of human-in-the-loop tasks. Although the empirical gains are modest, both the theoretical and empirical sections are clearly written and motivated.\n\nThe empirical results show that:\n - Probabilistic attention outperforms standard attention in a low-click interactive segmentation domain and performs similarly to standard attention with a greater number of clicks\n - Unsupervised key adaptation (cf. Section 3.5) leads to significantly improved IoU in a non-interactive domain\n\nThe authors also released a Pytorch layer implementing their proposed approach. The paper is seemingly missing a comparison to other methods for interactive segmentation and does not justify its choice of the BoTNet50 architecture as a baseline model. Once again, though, the paper is out of my area and I am not familiar with the state-of-the-art in this domain. I will choose to defer to the other reviewers on my recommendation for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper builds and analyzes a probabilistic mixture model for which softmax attention mechanisms are a special case. Reviewers  find the construction enlightening, educational, and with high impact potential. They strongly recommend improving the presentation to make the "story" and motivation much clearer upfront: the reasons for the construction don't reveal themselves until the connections to attention. The constructions seem different, but uthors should discuss the relationship with other probabilstic attention work (e.g. variational attention by Deng et al, but also Heo et al, NeurIPS 19, Bahuleyan et al, Coling 19). The font sizes and line markers/colors in the plot should be changed to improve readability and skimmability of the paper.
Decision: Accept (Spotlight)