Question: Why do they adopt a single-gate mechanism?
Evidence: In order to select valuable and appropriate shared features for different tasks, we design a selected sharing layer following the shared layer. The selected sharing layer consists of two cells: gated sharing cell for filtering useless features and attention sharing cell for focusing on valuable shared features for specific tasks. The description of this layer is depicted in Figure FIGREF6 and Figure FIGREF15. In the following, we introduce two cells in details.
Gated Sharing Cell Inspired by forgotten gate mechanism of LSTM BIBREF32 and GRU BIBREF33, we design a single gated cell to filter useless shared features from shared layer. There are two reasons why we adopt single-gate mechanism. One is that transformer encoder in shared layer can efficiently capture the features of long-range dependencies. The features do not need to capture repeatedly by multiple complex gate mechanisms of LSTM and GRU. The other is that single-gate mechanism is more convenient for training BIBREF34. Formally, the gated sharing cell can be expressed as follows:
where ${\rm \textbf {H}}_{shared}\! \in \! \mathbb {R}^{1 \times l(d_p+d_w)}$ denotes the outputs of shared layer upstream, ${\rm \textbf {W}}_{fake} \in \mathbb {R}^{l(d_p+d_w) \times l(d_p+d_w)}$ and ${\rm \textbf {b}}_{fake} \in \mathbb {R}^{1 \times l(d_p+d_w)}$ are trainable parameters. $\sigma $ is a non-linear activation - sigmoid, which makes final choices for retaining and discarding features in shared layer.
Then the shared features after filtering via gated sharing cell ${\rm \textbf {g}}_{fake}$ for the task of fake news detection are represented as:
where $\odot $ denotes element-wise multiplication.
Similarly, for the auxiliary task - the task of stance detection, filtering process in the gated sharing cell is the same as the task of fake news detection, so we do not reiterate them here.
Attention Sharing Cell To focus on helpful shared features that are beneficial to specific tasks from upstream shared layer, we devise an attention sharing cell based on attention mechanism. Specifically, this cell utilizes input embeddings of the specific task to weight shared features for paying more attention to helpful features. The inputs of this cell include two matrixes: the input embeddings of the specific task and the shared features of both tasks. The basic attention architecture of this cell, the same as shared-private feature extractor, also adopts transformer encoder (the details in subsection SECREF8). However, in this architecture, query matrix and key matrix are not projections of the same matrix, i.e., query matrix ${\rm \textbf {E}}_{fake}$ is the input embeddings of fake news detection task, and key matrix ${\rm \textbf {K}}_{shared}$ and value matrix ${\rm \textbf {V}}_{shared}$ are the projections of shared features ${\rm \textbf {H}}_{shared}$. Formally, the attention sharing cell can be formalized as follows:
where the dimensions of ${\rm \textbf {E}}_{fake}$, ${\rm \textbf {K}}_{shared}$, and ${\rm \textbf {V}}_{shared}$ are all $\mathbb {R}^{l\times (d_p+d_w)}$. The dimensions of remaining parameters in Eqs.(DISPLAY_FORM16, DISPLAY_FORM17) are the same as in Eqs.(DISPLAY_FORM10, DISPLAY_FORM11). Moreover, in order to guarantee the diversity of focused shared features, the number of heads $h$ should not be set too large. Experiments show that our method performs the best performance when $h$ is equal to 2.
Integration of the Two Cells We first convert the output of the two cells to vectors ${\rm \textbf {G}}$ and ${\rm \textbf {A}}$, respectively, and then integrate the vectors in full by the absolute difference and element-wise product BIBREF35.
where $\odot $ denotes element-wise multiplication and $;$ denotes concatenation.
There are two reasons why we adopt single-gate mechanism. One is that transformer encoder in shared layer can efficiently capture the features of long-range dependencies. The features do not need to capture repeatedly by multiple complex gate mechanisms of LSTM and GRU. The other is that single-gate mechanism is more convenient for training (Srivastava et al., 2015).
Answer: One is that transformer encoder in shared layer can efficiently capture the features of long-range dependencies. The features do not need to capture repeatedly by multiple complex gate mechanisms of LSTM and GRU. The other is that single-gate mechanism is more convenient for training.