# Affection Enhanced Relational Graph Attention Network for Sarcasm Detection
## 3 Methodology
### Relational Graph Attention Network  
The graph attention network is a variant of graph neural networks, which leverages masked self-attention layers to encode graph structure. The vanilla GAT uses an adjacent matrix as structural information, thus omitting dependency label features. RGAT incorporates relational features into the attention calculation and aggregation process to obtain more informative representations. The details of the RGAT layer are illustrated in Figure 3.  
Input and Output: We denote the relation between word \(w_{i}\) and \(w_{j}\) as \(\mathcal{R}_{ij}\), we transform \(\mathcal{R}_{ij}\) into a vector \(\mathbf{r}_{ij}\in\mathbb{R}^{d_{r}}\), where \(d_{r}\) is the dimension of relation embeddings. The RGAT takes initial hidden representations \(\{h_{1}^{0},h_{2}^{0},\ldots,h_{n}^{0}\}\) learnt by BiLSTM or BERT, the adjacent matrix \(\mathbf{A}\) and the relation embeddings as input to produce a new set of word features \(\{h_{1}^{l},h_{2}^{l},\ldots,h_{n}^{l}\}\) as its output after \(l\) iteration.  
Relation-aware feature aggregation: An RGAT updates the hidden representation at \(l\)th layer by calculating a weighted sum of the neighbour states at the guidance of relation  
Figure 2: Architecture of the proposed ARGAT framework for sarcasm detection. It consists of a contextual encoder, RGAT layers, GCN layers and a classifier. The RGAT layers and GCN layers can be stacked to n layers.  
embeddings. Specifically, the aggregation process of a multi-head attention-based RGAT can be described as:  
\[\mathbf{h}_{i}^{l}=\big{|}_{z=1}^{Z}\sigma(\sum_{j\in N(i)}a_{ij}^{lz}(\mathbf{W} _{V}^{lz}\mathbf{h}_{j}^{l-1}+\mathbf{W}_{V_{V}^{lz}}^{lz}r_{ij})) \tag{2}\]  
where \(\mathbf{W}_{V}^{lz}\in\mathbb{R}^{\frac{d}{Z}\times d}\) and \(\mathbf{W}_{V_{V}}^{lz}\in\mathbb{R}^{\frac{d}{Z}\times d_{r}}\) are parameter matrices. \(a_{ij}^{lz}\) is the normalized attention scores calculated by the attention mechanism, which combines node-aware attention scores and relation-aware attention scores:  
\[a_{ij}^{lz}=\frac{exp(e_{ij}^{N}+e_{ij}^{R})}{\sum_{j^{\prime}\in\mathbf{N}(i)} exp(e_{ij^{\prime}}^{N}+e_{ij^{\prime}}^{R})} \tag{3}\]  
where \(e_{ij}^{N}\) and \(e_{ij}^{R}\) are node-aware attention scores and relation-aware scores, respectively. We calculate the two types of attention scores according to the equations denoted as:  
\[e_{ij}^{N}=\begin{cases}f(h_{i}^{l-1},h_{j}^{l-1}),&j\in\mathbf{N}(i)\\ -inf,&\text{otherwise}\end{cases} \tag{4}\]  
\[e_{ij}^{R}=\begin{cases}f(h_{i}^{l-1},r_{ij}),&j\in\mathbf{N}(i)\\ -inf,&\text{otherwise}\end{cases} \tag{5}\]  
For \(\mathbf{h}^{l}=\{\mathbf{h}_{0}^{l},\mathbf{h}_{1}^{l},\ldots,\mathbf{h}_{n}^ {l}\}\), we employ two normalization layers to get better representations:  
\[\mathbf{h}^{l}=Norm(\mathbf{h}^{l}+\mathbf{h}^{l-1}) \tag{6}\]  
\[\mathbf{h}^{l}=Norm(\mathbf{h}^{l}+FFN(\mathbf{h}^{l})) \tag{7}\]  
where \(FFN(\mathbf{x})=Relu(\mathbf{x}\mathbf{W}_{1}+\mathbf{b}_{1})\mathbf{W}_{2}+ \mathbf{b}_{2}\) is a two-layer multi-layer perceptron(MLP) with the activation function \(Relu\), \(Norm\) is a normalization layer, \(W_{1}\) and \(W_{2}\) are trainable parameters, and \(\mathbf{h}^{l}\) in Equation (9) is the final output the RGAT layer at iteration \(l\). In this way, we take both node features and relation features into consideration for better feature aggregation.  
Figure 3: Details of an RGAT layer and its corresponding relation-aware attention operation.