## 4 MultiDS Model
### Hierarchical Attention Generator via Multi-Level Distributional Signatures
#### 4.2.1 Domain-Agnostic Attention  
Inspired by the paradigm of applying pre-trained models to downstream tasks, we believe that distributional signatures computed from large amounts of data can also be  
Figure 2: (**a**) The workflow of MultiDS model. (**b**) Hierarchical attention generator. (**c**) Domain calibrator.  
transferred to specific domains as domain-independent features. Specifically, we use a document-level news dataset, 20 Newsgroups [23], to calculate domain-agnostic distributional signatures. As shown in Table 2, 20 Newsgroups contains news articles from 6 domains, roughly including computers, recreation, science, politics, religion, and for-sale. This multi-domain document-level dataset enables the model to obtain general distributional signatures without domain bias.  
Here, we leverage explicit distributional signatures, which mean word frequency, to infer word weights. The frequency of words appearing in documents is a kind of natural and annotation-free feature that effectively implies words importance in low-resource scenarios. For instance, frequently used words such as 'the', 'a', etc., are often considered less important, while those that rarely appear often contain more discriminative information.  
Firstly, given a word sequence \(w^{news}\) in 20 Newsgroups, our approach is to utilize an existing method [13] to make a rough estimate of word weights via word frequency in Equation (3),  
\[s^{a}(w^{news}_{i})=\frac{\eta}{p(w^{news}_{i})+\eta}, \tag{3}\]  
where \(p(w^{news}_{i})\) is the unigram likelihood of the \(i\)-th word over the dataset; \(\eta\) is a hyperparameter; \(s^{a}(w^{news}_{i})\) represents the noisy weight of the \(i\)-th word. \(s^{a}(\cdot)\) represents a mapping from words (within vocabulary) to their corresponding weights. According to the above hypothesis, the \(s^{a}(\cdot)\) computed from a large document-level dataset contains generalized semantic knowledge that can be applied to the processing of other texts.  
As the second step, we devise an attention translator that translates coarse word weights into fine-grained word-level attentions. Our attention translator consists of two components, a multi-layer perceptron (MLP) and a bidirectional LSTM (BiLSTM). Given an input sequence \(w^{a}\), the MLP projects coarse word weights into a higher-dimensional space in Equation (4). Weights with higher dimensions contain richer semantic information.  
\[d^{a}(w^{a})=W_{att}s^{a}(w^{a})+z_{att}, \tag{4}\]  
where \(W_{att}\), \(z_{att}\) are weight and bias of the MLP; \(d^{a}(w^{a})=\left\{d^{a}(w^{a}_{1}),\cdots,d^{a}(w^{a}_{i}),\cdots,d^{a}(w^{ a}_{l})\right\}\) are higher-dimensional word weights, and \(d^{a}(w^{a}_{i})\) means weight of the \(i\)-th word \(w^{a}_{i}\). Since each word weight is contextually affected, we employ a BiLSTM, which is adept at processing sequence information, to further encode these weights in Equations (5)-(7), so that each word weight implies the correlation between contextual information.  
\[g^{a}(\overset{\rightarrow}{w^{a}})=\underset{\text{LSTM}}{\overset{ \rightarrow}{\left(\left\{d^{a}(w^{a}_{1}),d^{a}(w^{a}_{2}),\cdots,d^{a}(w^{ a}_{l})\right\}\right\}} \tag{5}\]  
\[g^{a}(\overset{\leftarrow}{w^{a}})=\underset{\text{LSTM}}{\overset{ \leftarrow}{\left(\left\{d^{a}(w^{a}_{1}),d^{a}(w^{a}_{2}),\cdots,d^{a}(w^{ a}_{l})\right\}\right\}} \tag{6}\]  
\[att^{a}(w^{a})=g^{a}(w^{a})=[g^{a}(\overset{\rightarrow}{w^{a}});g^{a}( \overset{\leftarrow}{w^{a}})] \tag{7}\]  
The bidirectional hidden states \(g^{a}(\overset{\rightarrow}{w^{a}})\) and \(g^{a}(\overset{\leftarrow}{w^{a}})\) are calculated by Equations (5) and (6) using coarse word weights, respectively. Equation (7) concatenates two hidden states to get domain-agnostic attentions.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Example** & **Token/Example** & **Vocab** & **Domain** & **Example/Domain** & **Class** \\ \hline
18,828 & 340 & 32,137 & 6 & 3138 & 20 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of 20 Newsgroups.