# Comprehensive Document Summarization with Refined Self-Matching Mechanism
## 3 Method
### Localness Modeling  
The self-matching mechanism can establish long-term dependencies for each sentence without concerning distance. Such an operation will disperse the attention distribution and result in overlooking neighboring signals. We argue that the self-matching mechanism may be further enhanced by local information modeling [28]. Conventional self-matching mechanisms focus on all sentences when collecting document information, but some secondary information may confuse the model and lead to suboptimal performance [29]. Moreover, the usage of weighted averages can inhibit the expression of relationships among neighboring words or sentences. In linguistic intuition, if a word \(x_{i}\) is aligned to \(x_{j}\) on the semantic matching, we also hope that words neighboring \(x_{j}\) can be perceived, which can capture phrases patterns or sentence segments that contain more explicit local context. Take the word-level local information as an example. In Figure 2, if \"children\" is aligned to \"make\", we hope to pay more attention to nearby \"a pray\". Eventually, \"children\" can aggregate the phrase information \"make a pray\" in the matching process. It is obvious that such neighboring information can contribute to expanding the relationship expression among words and the local context. Similar principles can be extended to sentences and related segments.  
A learnable Gaussian distribution is utilized to model the local context [30; 31] which contains valuable fine-grained information, whose scope is \(D\) as Figure 2. Similar to reference [30], the setting of scope size relies on the matching result itself. The Gaussian distribution will serve as the regularization term of attention and added to the original distribution.  
\[\alpha_{t}=softmax(q_{t}+G) \tag{14}\]  
The first term in the equation is the original dotted product distribution, and G is the local Gaussian bias term. The improved mechanism can model both the long term dependency and the localness, better simulating human coarse-to-fine understanding behavior. Since the prediction of each central position and window depend on their corresponding context representation, we apply a feed-forward network to transform \(q_{t}\) into the hidden state of center position and window. The motivation of this design is that the central position and window size interdependently locate local scope, hence condition on the similar hidden state. The center position scalar \(\mu_{t}\) and dynamic coverage scalar \(\sigma_{t}\) are predicted by the matching energy \(q_{t}\). They can be view as the center and the scope of the locality to be paid attention, intuitively they correspond to \"pray\" and Gaussian scope \"D\" in Figure 2. The center position scalar and the coverage scalar are calculated as follows:  
\[\mu_{t} =U_{p}^{T}tanh(W_{p}q_{t}) \tag{15}\] \[\sigma_{t} =U_{c}^{T}tanh(W_{g}q_{t}) \tag{16}\]  
where \(W_{p}\in\mathbb{R}^{d_{n}\times d_{n}}\) and \(W_{g}\in\mathbb{R}^{d_{n}\times d_{n}}\) are shared parameters, \(U_{c}\in\mathbb{R}^{d_{n}}\) and \(U_{p}\in\mathbb{R}^{d_{n}}\) are two different linear projection weighted vectors. \(\mu_{t}\) and \(\sigma_{t}\) are further normalized to the interval \([0,I]\), where \(I\) represents the number of input sentences.  
\[\begin{pmatrix}\hat{H_{t}}\\ \sigma_{t}\end{pmatrix}=I*sigmoid\begin{pmatrix}\mu_{t}\\ \sigma_{t}\end{pmatrix} \tag{17}\]  
Figure 2: Case for localness modeling.  
According to the definition of Gaussian distribution, local bias for the \(t\) encoding step is calculated with \(\tilde{\mu}_{t}\), \(\tilde{\sigma_{t}}\) :  
\[G_{t,i}=-\frac{(i-\tilde{\mu}_{t})}{{\tilde{\sigma_{t}}}^{2}/2} \tag{18}\]  
The local bias \(G\in(-\infty,0]\) is added to the original attention distribution is approximate to multiplying by the weight \((0,1]\) as the exponential operation.