# External Slot Relationship Memory for Multi-Domain Dialogue State Tracking
## 3 Model
### External Slot Relationship Memory Layer  
Despite the slot context attention layer extracting contextual information for each slot, the model still struggles to effectively capture the contextual information of slots that have co-reference or co-referential relationships due to the diverse expressions in natural language dialogues. Additionally, the slot context attention layer computes contextual relevance information for individual slots without considering the relationships between slots. Inspired by [21], in this work, an external slot relationship memory is constructed to capture the correlations between different slots. Algorithm 1 summarizes the pseudo-code of external slot relationship memory layer. Two external memory units are treated as a dictionary of relationships between slots. Given a feature map \(F_{t}^{l}=[C_{t}^{S_{1}},C_{t}^{S_{2}},\ldots,C_{t}^{S_{J}}]\in\mathbb{R}^{d \times l}\), \(ExternalLayer\) calculates the input features and the external memory unit \(M\in\mathbb{R}^{d\times J}\) using the following equation:  
\[\widetilde{F}_{t}^{l}=(\alpha)_{i,j}=Norm\left(F_{t}^{l}M_{k}^{T}\right) \tag{10}\]  
\[F_{out}=\widetilde{F}_{t}^{l}M_{v} \tag{11}\]  
where \((\alpha)_{i,j}\) represents the correlation weight between slot \(i\) and \(j\). \(M\) is a learnable parameter independent of the input, serving as a memory for capturing the relationships between multi-domain slot-related information. \(\widetilde{F}_{t}^{l}\) represents the inferred correlation weight graph from multi-domain data, and \(F_{out}\) denotes the most relevant feature information of the output slots. We utilize two separate memory processing units, \(M_{k}\) and \(M_{v}\), as keys and values. The computational complexity of the external attention is O(d)), where d and J are hyperparameters.  
```
Input: F, an array of shape [B, J, dim] (batch_size, number of slots, embed-ding_dimension)
2 Parameter: \(M_{k}\), a linear layer
3 Parameter: \(M_{v}\), a linear layer
4 Parameter: H, number of heads
5 Output: out, an array of shape [B, J, dim] F = query_linear(F) F = F:view(B, J, H, dim // H) F = F:Permute(0, 2, 1, 3) attn = \(M_{k}\)(F) attn = softmax(attn, dim=2) attn = norm(attn, dim=3) out = \(M_{v}\)(attn) out = out.permute(0, 2, 1, 3) out = out.view(B, J, dim) F = \(W_{o}\)(out)
```  
**Algorithm 1** A python-style pseudo-code for external slot relationship memory  
To filter out redundant information in slot relations, we adopt the double-normalization technique proposed in [31], which normalizes both the feature rows and feature columns. The single external layer computation process can be described as follows:  
\[\left(\vec{a}\right)_{i,j}=CM_{k}^{T} \tag{12}\]  
\[\hat{\alpha}_{i,j}=\exp\left(\hat{\alpha}_{i,j}\right)/\sum_{k}\exp(\hat{ \alpha}_{i,j}) \tag{13}\]  
\[\alpha_{i,j}=\hat{\alpha}_{i,j}/\sum_{k}\hat{\alpha}_{i,k} \tag{14}\]  
And we utilize the multi-head external layer, which is represented as follows:  
\[h_{i}=ExternalLayer(F_{t}^{l},M_{k},M_{v}) \tag{15}\]  
\[G_{t}^{l}=MultiHead(F_{t}^{l},M_{k},M_{v})=Concat(h_{1},\dots,h_{H})W_{o} \tag{16}\]  
\[F_{t}^{l+1}=FFN(G_{t}^{l})+G_{t}^{l} \tag{17}\]  
where \(h_{i}\) represents the index of the head, H represents the total number of heads, and \(W_{o}\) is a linear learnable parameter matrix. The method \(MultiHead()\) represents that we have adopted a multi head approach. Method \(FFN()\) indicates that we have adopted a linear fully connected layer. \(Concat()\) represents the splicing operation.  
The final slot vector representation is denoted as \(F_{t}^{L+1}=[f_{t}^{S_{1}},\dots,f_{t}^{S_{l}}]\), where \(f_{t}^{S_{j}}\) represents the representation of a specific slot information.