Probabilistic Attention for Interactive Segmentation
Year: 2021

Abstract:
We provide a probabilistic interpretation of attention and show that the standard dotproduct attention in transformers is a special case of Maximum A Posteriori (MAP) inference. The proposed approach suggests the use of Expectation Maximization algorithms for online adaptation of key and value model parameters. This approach is useful for cases in which external agents, e.g., annotators, provide inference-time information about the correct values of some tokens, e.g., the semantic category of some pixels, and we need for this new information to propagate to other tokens in a principled manner. We illustrate the approach on an interactive semantic segmentation task in which annotators and models collaborate online to improve annotation efficiency. Using standard benchmarks, we observe that key adaptation boosts model performance (∼ 10% mIoU) in the low feedback regime and value propagation improves model responsiveness in the high feedback regime. A PyTorch layer implementation of our probabilistic attention model is available here: https://github.com/apple/ml-probabilistic-attention.

heading: 1 Introduction
text: Attention was first introduced as a computational primitive for natural language processing [53] and has since been widely adopted [17, 63, 13, 14] as a replacement for recurrent primitives such as LSTMs [26]. More recently it has been making inroads into computer vision [43, 69, 57, 64, 5, 19, 49] as a replacement for the long accepted convolution as the main computational primitive. Self-attention based architectures have demonstrated state-of-the-art results in fundamental vision problems including image classification [5, 43, 69, 19, 49], object detection [8, 71, 57], image and video semantic segmentation [55, 29, 57, 41] and tracking [66] to state a few.
There are a few different perspectives on the reasons for success of self-attention in computer vision and its superiority over convolution. This includes a view that the self-attention mechanism allows modeling spatially varying dynamic convolution filters [32] and at the same time enabling parameter independent scaling of receptive fields [52]. Another includes their ability to capture global context through long range interactions especially when full attention is feasible [49] at reduced spatial resolution maps or using an approximation of full attention with axial [55] or criss-cross attention [29]. A recent work [45] introduces modern Hopfield networks with continuous states where the update mechanism is shown to be equivalent to the update mechanism of standard dot-product attention [53]. They show that such a network has the capacity to store exponentially many patterns and retrieve them with high fidelity. In this work, we provide a novel interpretation of attention as a probabilistic generative model for queries and values. Specifically we hypothesize the existence of a bank of probabilistic memory units, each of which maintains a joint probability distribution over queries and values parameterized through keys. A query/value pair is generated by first sampling a unit (from a prior over units) followed by sampling the pair from the unit specific joint distribution. This is equivalent to generating the queries and values through a probabilistic mixture model over the units. A particular form for unit joint likelihoods expressed as Gaussians for both the query and value marginals, assuming their independence conditioned on a unit, turns out to be equivalent to traditional
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
dot product attention under a few constraints. As shown in Section 3.3, maximum likelihood (ML) inference for the corresponding value given a query is equivalent to standard dot-product attention.
Our probabilistic interpretation provides a systematic framework for online update of mixture model parameters based on a set of observed queries. It also allows propagation of correct values provided by an external agent for some of the units to all other units. Using Bayesian inference in the constrained case, we derive update rules for online unsupervised adaptation (Section 3.5) of query/key likelihood parameters based on a set of observed queries. We also derive update equations for online value propagation (Section 3.6) across units based on fixed externally specified values for a subset of units. The latter is specifically useful for interactive segmentation where a correction provided by an annotator has to be propagated globally to make the process more efficient. We use probabilistic attention in place of standard attention in deep architectures for interactive segmentation both within the backbone and at the network head as a classifier. Specifically we use probabilistic attention updates in the BoTNet50 [49] architecture and show that adapting keys to incoming queries leads to better model performance in the low annotator feedback regime. Using value propagation within a probabilistic attention layer at the head of the segmentation network leads to a more responsive model through effective feedback propagation in the high feedback regime. We also use both key adaptation and value propagation together and demonstrate the complementary effects of the two in both the low and high annotator feedback regimes.

heading: 2 Related Work

heading: 2.1 Attention
text: Natural language processing has seen the rise [3, 53] and widespread adoption of attention in recent years [17, 63, 13, 14, 59]. One of the first works on visual attention was on learning to attend to image regions for caption generation [60]. Since then there has been a steady progress on using attention primitives within vision models for recognition and classification [27, 57, 5, 54, 43, 69, 19, 52, 49], detection [8, 71], segmentation [55, 29], tracking [66] and video analysis [42, 41]. There have been numerous works interpreting the attention mechanism as a form of computing non-local means [7, 57], approximating dynamic convolution filters [32, 52], and capturing global context through long range interactions [27, 43, 55]. The standard dot-product attention [53] update was also formulated as emerging from the update rule of modern Hopfield networks [45]. Our work introduces attention mechanism from a novel perspective as that of inferring from a probabilistic memory bank. To our knowledge, the only work that is closest to our approach is [18], which also proposes a similar interpretation but only for the queries in order to study the explaining away effect of attention update. Other notable works modeling attention in a probabilistic framework have done so using neural processes [25] or variational approximations for modeling alignments as latent variables [16] or to overcome the bypassing phenomenon of deterministic attention [4] in sequence tasks. Being different from these approaches, our formulation encapsulates queries and values in a single generative model and provides an interpretation of standard dot-product attention as constrained Bayesian inference. Doubly normalized attention scheme [18] also emerges as a special case of key adaptation in our framework.

heading: 2.2 Interactive Segmentation
text: Deep neural networks have set state-of-the art in semantic segmentation through the use of fully convolutional architectures [36, 70, 10, 11, 12, 50, 56, 67] and more recently using hybrid convolution and self-attention [64] or stand-alone self-attention architectures [55]. The input domain of interactive segmentation includes user input in the form of clicks or scribbles in addition to the visual signal (images or videos). The earliest works in interactive segmentation were based on algorithmic approaches for incorporating human inputs into region [46, 51, 65] or boundary [40, 20] processing pipelines. [44] provides a comprehensive survey of interactive segmentation approaches. More recently deep networks have been used to incorporate user feedback to guide their output predictions at the pixel level. Following a similar taxonomy, these can be roughly categorized into region based [62, 61, 37, 33, 6, 2, 28, 68, 34, 31] or boundary based approaches [9, 1, 35, 58]. Deep Extreme Cut (DEXTR) [38] demonstrated that user guidance in the form of extreme points could be used in addition to the input channels to accurately localize the object of interest. More recently [68] argued that three points are sufficient as input guidance to localize the object but additional corrective
clicks could be used to further refine the prediction. Other works have used the corrective clicks to adapt the network inputs [30], embeddings [48] or parameters [31] online. Different from these previous approaches, we use corrective clicks as providing fixed values for a subset of units in the proposed probabilistic attention framework. These values are propagated globally through the attention mechanism to directly and more effectively influence the outputs towards user intended values.

heading: 3 Method
text: We provide a probabilistic interpretation of attention as a generative model for queries and values through a set of memory units. Using this formulation, traditional attention in transformers [53] reduces to the special case of maximum a posteriori (MAP) inference of values given queries, assuming Gaussians for the likelihoods. Using Bayesian inference, we provide a systematic approach to adapt keys online as a locally ML update of the corresponding model parameters. Our formulation also allows to fix the values of certain units and propagate their influence to other units online by conditioning on the fixed unit values. The following sections provide more details on the probabilistic model.

heading: 3.1 Probabilistic attention
text: We assume that there are n memory units, indexed by i, each of which can be queried through a vector qi ∈ Rd to yield an output value vector vi ∈ Rm. The queries and the corresponding values may depend on an input x. For example, each memory unit may represent a pixel xi in an image x. The joint distribution of queries qi and values vi conditioned on the input x is assumed to factorize over memory units
p(q1:n, v1:n|x) = Πni=1pi(qi, vi | x), (1)
where x is the conditioning input, q1:n = {q1, · · · , qn}, v1:n = {v1, · · · , vn}, and qi ∈ Rd, vi ∈ Rm are the query, value vectors for unit i respectively. The per-unit joint likelihood pi(qi, vi | x) is a probabilistic mixture model given by
pi(q, v | x) = n∑ j=1 pi(q, v, uj | x) = n∑ j=1 πi,j(x) pi(q, v | uj , x), (2)
where we have dropped the subscript i from q and v for simplicity. In the above, uj indexes unit j, πi,j(x) is the probability of activating unit j when unit i is queried, pi(q, v | uj , x) is the likelihood of observing the pair (q, v), given the pair is generated through unit j in the mixture, conditioned on the input x.

heading: 3.2 Value inference
text: Using the above model, it is possible to find the most likely value v̂ given a query q to unit i
v̂ = argmax v
pi(v|q, x). (3)
We use Expectation Maximization (EM) [15] to achieve this: starting with an initial estimate v0 of the most probable value and iterating over the standard EM auxiliary function Qi. Given the latest known estimate vt, the M step produces a new estimate vt+1 that increases Qi by maximizing it w.r.t. vt+1. This guarantees local maximization of pi(v|q, x).
Qi(v t, vt+1 | x) = ∑ j wti,j log pi(uj , q, v t+1 | x), (4)
where
wti,j = pi(uj | q, vt, x) = πi,j(x) pi(q, v t | uj , x)∑ j πi,j(x) pi(q, v t | uj , x) . (5)
The n× n matrix formed by the entries wi,j corresponds to the attention matrix in standard transformers. The optimal value v̂ is obtained by taking the gradient with respect to vt+1 and setting it to zero
∇vt+1Qi(vt, vt+1 | x) = ∑ j wi,j∇vt+1 log pi(q, vt+1, uj | x), (6)
where ∇vt+1 log pi(q, vt+1, uj | x) is the Fisher Score for unit i with respect to vt+1.

heading: 3.3 Relationship to standard attention
text: We show that standard attention in transformers [53] solves Eq. (6) under the special case of a constrained Gaussian mixture model (GMM). Assuming isotropic Gaussians with conditionally independent queries and values given input and mixture component
pi(q, v | uj , x) = pi(q | uj , x) pi(v | uj , x) (7) pi(q | uj , x) = (αj(x)
2π
)d/2 e− αj(x) 2 ‖q−ξj(x)‖ 2
(8)
pi(v | uj , x) = (βj(x)
2π
)m/2 e− βj(x) 2 ‖v−µj(x)‖ 2 , (9)
where αj(x), βj(x) > 0 are precision parameters, ξj(x) ∈ Rd, µj(x) ∈ Rm are the key and expected value parameters for unit j given the input x. The dependency of pi(q, v | uj , x) on x is through the fact that the parameters αj , βj , πi,j , ξj , µj are a function of x. For simplicity, we treat x to be fixed and leave the dependency on x implicit in our notation. In order to obtain the standard attention update equation, we constrain the precision parameters to be the same across units: α1 = · · · = αn = α, β1 = · · · = βn = β, and link the priors of each unit to the lengths of the corresponding key and expected value vectors
πi,j = 1 z e α 2 ‖ξj‖ 2 e β 2 ‖µj‖ 2
(10) z = ∑ j e α 2 ‖ξj‖ 2 e β 2 ‖µj‖ 2 . (11)
Assuming β → 0 and solving for optimal vt+1 in Eq. (6), we obtain the standard attention update (see Appendix B.1)
vt+1 = ∑ j wi,jµj (12)
wti,j = eαξ T j q∑
j e αξTj q
, (13)
where each µj is the value associated with unit j and vt+1 is the output at unit or token i after the attention update. In this case, wti,j is no longer a function of t and thus only one EM iteration is needed.

heading: 3.4 Offline supervised learning
text: As is commonly done in standard transformers, the relationship between the input x and the mixture model parameters: π(x), ξ(x), µ(x) can be modeled using a deep network, whose parameters can be trained off-line with task specific supervision.

heading: 3.5 Online unsupervised mixture model adaptation
text: Our framework provides a way to adapt the mixture model parameters based on all the observed input queries prior to doing value inference. This process can be seen as an inference-time adaptation of the model using the additional information contained in the set of queries. We propose an unsupervised Bayesian approach to do this adaptation for the per-unit key vectors ξ1:n = {ξ1, · · · , ξn} and
precision parameters α1:n = {α1, · · · , αn} given queries q1:n = {q1, · · · , qn}. For each unit i, the optimal value inference is given by
v̂i = argmax v
pi(v | q1:n). (14)
Assuming a prior for the key vectors given the observed queries p(ξ1:n |q1:n), the likelihood pi(v |q1:n) can be written as pi(v | q1:n) = ∫ p(ξ1:n | q1:n)pi(v | qi, ξ1:n)dξ1:n ≈ pi(v | qi, ξ̂1:n), (15)
where the expectation over the posterior is approximated by its maximum a posteriori (MAP) value
ξ̂1:n = argmax ξ1:n
p(ξ1:n | q1:n) (16)
v̂i = argmax v
pi(v | qi, ξ̂1:n). (17)
In order to solve (16), we use an iterative EM approach. The initial key parameters ξ01:n are provided by the pre-trained model. To avoid overfitting to the current query vectors, we use a Gaussian prior centered on the key parameters provided by the pre-trained network, i.e., ξ01:n with a finite precision θξ > 0. The EM update for the key parameters at any iteration t is given by (see Appendix B.2)
ξt+1k = θξξ
t k + αk ∑n i=1 w t i,kqi
θξ + αk ∑n i=1 w t i,k . (18)
Analogous to the keys, we can also adapt the αj precision parameters (see Appendix B.3).

heading: 3.6 Online value propagation
text: The proposed model allows for fixing the outputs of a selected subset of units to predefined values and letting them propagate to other units in a principled way. This aspect of our model is of particular interest to interactive semantic segmentation, where a human annotator provides corrections to the output of a semantic segmentation model. In this case, assuming an attention layer at the output of a deep model, the memory units correspond to pixels and the output values correspond to the semantic label for that pixel, e.g. foreground or background. Based on the network’s prediction, an annotator provides corrections for a subset of the pixels, which are the ground truth for those pixels. These correspond to the fixed predefined values for those units, whose effect is to be propagated to semantically similar pixels globally across the image to make the process more efficient. More formally, suppose the annotator has provided the correct values for the first s < n units. We want for this information to improve the inference about the value for all the other units i > s. Within our framework, this inference is given by
v̂i = argmax v
pi(v | qi, q1:n, v1:s), for s < i <= n. (19)
In order to do this inference, we adopt a Bayesian approach similar to model adaptation of Section 3.5. Let λ represent the set of network parameters, e.g., π, ξ, µ, α, β. Writing the inference as an expectation over the model posterior p(λ | q1:n, v1:s)
pi(v | q1:n, v1:s) = ∫ p(λ | q1:n, v1:s)pi(v | qi, λ)dλ ≈ pi(v | qi, λ̂), (20)
where we approximate the expectation with its MAP estimate as before
λ̂ = argmax λ
p(λ | q1:n, v1:s) (21)
v̂i = argmax v
pi(v | qi, λ̂). (22)
Eq. (21) is solved using EM. Specifically, value propagation across units is achieved by updating the µk for each unit k starting with the initial value µ0k provided by the pre-trained model (Section 3.4). Following a similar approach as in Section 3.5, the EM update for µt+1k at iteration t is given by
µt+1k = θµµ
t k + βk ∑s i=1 w t i,kvi
θµ + βk ∑s i=1 w t i,k
(23)
wti,k = pi(uk | qi, vi, µt1:n) = πi,k p(qi | uk, ξk) p(vi | uk, µtk)∑n j=1 πi,k p(qi | uj , ξj) p(vi | uj , µtj) , (24)
where θµ is the precision for the Gaussian prior over each µk. See also Appendix B.4.

heading: 3.7 Combining offline learning and online adaptation
text: The inference time adaptation of parameters is differentiable. So it can be included as part of the traditional supervised optimization e.g. via stochastic gradient descent and used to learn the parameters of the prior distributions over ξ, µ, α, β, π.

heading: 3.8 Position embeddings
text: Positional embeddings [53, 47] are useful in attention models to encode the relative or absolute positions of tokens. In computer vision applications, relative position embeddings have been found to be critical to capture the interactions between features based on their pairwise positional relations [43, 49, 55]. We propose to encode relative position embeddings by introducing extra parameters in the per-unit likelihoods of the mixture components and their priors. Let rqj−i, and r k j−i denote the relative position embeddings for a query and key interacting at units i and j respectively. The query/key marginal with the position embeddings is given by (see Appendix B.5)
pi(q | ξj , rqj−i, uj) ∝ N (q | ξj , 1
αj Id)N (q | rqj−i,
1
αj Id) ∝ N (q |
ξj + r q j−i
2 ,
1
2αj Id), (25)
whereN (a | b, c) is the Gaussian likelihood function over a with mean b and covariance matrix c. Id is a d× d identity matrix. The mixture component priors with position embeddings take the form
πi,j ∝ N (ξj | rkj−i, 1
αj Id) exp [αj 2 ( 2‖ξj‖2 + ‖rqj−i‖ 2 + ‖rkj−i‖2 )] exp [ βj 2 ‖µj‖2 ] . (26)

heading: 4 Experiments
text: In this section, we report the results of using probabilistic attention at various stages of a deep interactive semantic segmentation network. Specifically, we use it within the BoTNet50 backbone [49] in place of standard attention and also as part of a self-attention based classification head at the network output. We quantify model performance (mean IOU relative to ground truth) as a function of the number of clicks [6] on two widely used public benchmarks for this task: GrabCut[46] and Berkeley [39]. Appendix C provides more details on the interactive segmentation model architectures, training and evaluation protocols.

heading: 4.1 Probabilistic attention within a backbone
text: We adopt the recent work on BoTNet [49] by replacing the convolutional layers with attention layers in the last bottleneck block (c5) of the ResNet50 [24] architecture. Specifically, we use probabilistic attention layers in place of standard attention using either full or axial [55] attention. We experiment with either factored [49] or full relative positional encoding. Factored encoding uses (H + W )d parameters for an image of size (H,W ) factoring them along the height and width dimensions, whereas full encoding uses 2(H ∗W )d− d parameters, d per relative offset. Our models are trained on the LVIS [21] dataset at a resolution of 256 pixels. The results are shown in Fig. 1. The results suggest that using probabilistic attention in the BoTNet50 backbone leads to better performance especially for smaller number of clicks. This is true with both full and axial attention BoTNets using probabilistic attention. Using full relative position encoding helps more than using factored encoding perhaps due to the larger number of parameters.

heading: 4.2 Key adaptation
text: We experiment with unsupervised model adaptation as described in Section 3.5 by adapting the keys online (Eq. (18)) based on the observed queries. The degree of adaptation is controlled by the prior precision parameter θξ with lower values leading to a higher degree of adaptation due to the lower weight on the prior keys. Using the probabilistic attention BoTNet50 backbones of the previous section, we experiment with and without key adaptation. With key adaptation, we use two different values of the precision prior, 0.001 and 0, with the latter corresponding to a maximum likelihood update of the keys given observed queries. The results in Fig. 2 show the mean IoU as a function of number of clicks using the ProbBoTNet50-FactoredPE model. We observe that key adaptation leads
to higher IoUs without any corrective clicks or using only a few corrective clicks. Specifically there is an absolute improvement of about 10% in mean IoU using key adaptation and without using any corrective clicks. Additional results using ProbBoTNet50-FullPE are shown in the the Appendix D. Using a lower value of prior precision seems beneficial and the extreme case of maximum likelihood adaptation leads to the best performance. Note that this effect has been observed in a previous work [18], where it is perceived as a doubly normalized attention scheme (DNAS). This can be attributed to the unsupervised model adaptation accounting for the small domain shift introduced by models trained on LVIS and evaluated on GrabCut and Berkeley datasets. Without using key adaptation additional user input in the form of corrective clicks is required to account for this shift as can be seen by the asymptotically similar behavior with increasing number of clicks.

heading: 4.3 Probabilistic attention as a classifier
text: Deep architectures [36, 23, 12, 10] for semantic segmentation have a fully connected (1x1 conv) layer as their classification head. We replace this layer with a corrective self-attention [53] module
that takes corrective click locations as additional inputs to more effectively propagate corrections as follows.
Corrective self attention. We append the corrective channels to the features of the penultimate decoder layer and feed them as inputs to the value embedding layer (see Fig. 3). The query and the key embeddings do not use the corrective channels as their inputs, which allows attention maps to be computed based only on the semantics captured by the features. However, the weights of the value embedding layer can be trained to output the desired labels at the locations of the corrective clicks. Fig. 3 shows a block diagram of our corrective self-attention (CSA) layer. We choose to use probabilistic self-attention for computing the output values.
The CSA layer is used at the network output by up-sampling the final feature map of the decoder to the input image resolution and appending positive and negative corrective channels containing only the click locations. The local context size for Axial attention modules is chosen to be 64 pixels. The output of the axial attention block is passed through a sigmoid to estimate the pixelwise probabilities of the object mask.

heading: 4.4 Value propagation
text: We demonstrate the effect of propagating annotator feedback across pixels using online value propagation as described in Section 3.6. We use the CSA layer described above in place of the 1x1 conv classifier head of the HRNetV2+OCR [67] architecture pre-trained on Imagenet classification and fine-tuned for interactive segmentation on SBD [22] at a resolution of 256 pixels. Note that value propagation requires learning one additional parameter per output class (2 in our case) to estimate the fixed logit that corresponds to the annotator feedback for that class at the corrective locations. These are learnt as part of network training using gradient descent. Following standard protocols, we test on the GrabCut [46] and Berkeley [39] datasets. Fig. 4 shows the effect of using different number of value propagation iterations (1 and 5) within the probabilistic attention layer. Clearly, value propagation leads to more effective propagation of labeler feedback relative to not using it. For this experiment, we do value propagation only in the output width block of the CSA layer (Fig. 3) as we found that doing so in both the height and width layers did not work so well in our experiments. We hypothesize that this is probably due to the difficulty in learning the high dimensional fixed parameters corresponding to annotator feedback in the height block of the axial attention layer.

heading: 4.5 Combining key adaptation and value propagation
text: In this section, we experiment with combining key adaptation and value propagation in a single model. For this experiment we use a BoTNet50 architecture with a corrective self-attention classification head at a resolution of 256 pixels. We use axial attention in both the backbone and the classification head with probabilistic self attention updates. The model is trained on LVIS and evaluated on GrabCut and Berkeley datasets. Fig. 5 shows the effect of using either key adaptation or value propagation or
both relative to not using them. As observed separately in the previous plots, key adaptation helps in the small #clicks regime whereas value propagation shows greater benefits with increasing #clicks. Using the two jointly allows the model to respond quickly to annotator feedback in both the regimes.

heading: 5 Conclusion
text: We provide a probabilistic interpretation of the attention mechanism in transformers as a generative mixture model over queries and values parameterized through keys. Using our framework, the attention update is maximum a posteriori inference over values given queries. Specifically, the standard dot-product attention is a special case assuming Gaussians for the mixture likelihoods and a few other constraints. Using Bayesian inference, our interpretation allows for online update of the mixture model parameters as well as the propagation of a set of fixed values specified by an external agent. Although we demonstrate the utility of these aspects on the problem of interactive segmentation, the proposed model is generic and can be extended to other domains with suitable distributional forms for the mixture likelihoods.
Acknowledgments and Disclosure of Funding
We would like to thank Russ Webb, Joshua Susskind, Parisa Assar, and numerous other colleagues for their helpful discussions, draft and code reviews.

Review summaries:
The paper presents a generative view of attention: N hidden memory units each of which generates observed a query and a value. The memory units densely connected with each other. The paper shows that, given queries, the value of a memory unit, in a certain condition, is computed in a similar way as in attention, using EM.   Applying this new attention, called probabilistic attention, to interactive image segmentation tasks (where users provide some ground-truth pixels' values on the fly), the paper shows better results than using traditional attention on GrabCut and Berkeley datasets.   **Contributions**  1. novel explanation of attention, leading to new attention mechanism relying on probabilistic 

This paper offers a new interpretation of standard dot-product attention as a mixture model. The model assumes memory units are latent variables that dictate the probabilistic interaction between query and value vectors. The significance of this probabilistic interpretation is twofold:  1. MAP inference under a carefully constrained Gaussian mixture model    case of the model yields exactly the update rule of standard    dot-product attention. Prior to this the most similar    interpretation of attention was through modern Hopfield networks    (Ramsauer et al., 2021). 2. The model admits an EM algorithm which allows for either online or    offline learning in the precence of partial annotations.   The paper applies this probabilistic attention approach to interactive semantic segmentation by incorporating it in various ways within BotNet. The probabilistic attention either replaces standard attention or is used within the self-attention at the last part of the network. Under various settings (e.g. full or axial attention; factored or full relative positional encoding), the incorporation of probabilistic attention improves over the corresponding baseline.   The paper finally introduces a corrective self attention layer in order to more effectively pass along information from an annotators corrections of the model output. Results show that such a use of these corrections is more effective than not.

Summary and Contributions  The authors study a probabilistic interpretation of attention mechanisms for interactive segmentation.  The goal is to show that self-attention is related to MAP and EM can be used to induce the posterior distribution over attention scores.  The main contributions are: probabilistic interpretation of attention as a generative model, interactive segmentation with the proposed attention model, and classification.  The study shows that the proposed model has competitive results compared to related work.