Venue: NAACL 2019

# Star-Transformer

Qipeng Guo\({}^{\dagger}\) +
Footnote †: corresponding Author.

qpguo16@fudan.edu.cn

&Xipeng Qiu\({}^{\dagger}\) +

xpqiu@fudan.edu.cn

&Pengfei Liu\({}^{\dagger}\)

pfliu14@fudan.edu.cn

Yunfan Shao\({}^{\dagger}\) yfshao15@fudan.edu.cn

&Xiangyang Xue\({}^{\dagger}\) xyxue@fudan.edu.cn

&Zheng Zhang\({}^{\ddagger}\) zz@nyu.edu

Shanghai Key Laboratory of Intelligent Information Processing, Fudan University

\({}^{\dagger}\)School of Computer Science, Fudan University

\({}^{\ddagger}\)New York University

Corresponding Author.

###### Abstract

Although _Transformer_ has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present _Star-Transformer_, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving the capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

## 1 Introduction

Recently, the fully-connected attention-based models, like Transformer Vaswani et al. (2017), become popular in natural language processingNLP applications, notably machine translation Vaswani et al. (2017) and language modeling Radford et al. (2018). Some recent work also suggest that Transformer can be an alternative to recurrent neural networksRNNs and convolutional neural networks CNNs in many NLP tasks, such as GPT Radford et al. (2018), BERT Devlin et al. (2018), Transformer-XL Dai et al. (2019) and Universal Transformer Dehghani et al. (2018).

More specifically, there are two limitations of the Transformer. First, the computation and memory overhead of the Transformer are quadratic to the sequence length. This is especially problematic with long sentences. Transformer-XL Dai et al. (2019) provides a solution which achieves the acceleration and performance improvement, but it is specifically designed for the language modeling task. Second, studies indicate that Transformer would fail on many tasks if the training data is limited, unless it is pre-trained on a large corpus. Radford et al. (2018); Devlin et al. (2018).

A key observation is that Transformer does not exploit prior knowledge well. For example, the local compositionality is already a robust inductive bias for modeling the text sequence. However, the Transformer learns this bias from scratch, along with non-local compositionality, thereby increasing the learning cost. The key insight is then whether leveraging strong prior knowledge can help to "lighten up" the architecture.

To address the above limitation, we proposed a new lightweight model named "Star-Transformer". The core idea is to sparsify the

Figure 1: Left: Connections of one layer in Transformer, circle nodes indicate the hidden states of input tokens. Right: Connections of one layer in Star-Transformer, the square node is the virtual relay node. Red edges and blue edges are ring and radical connections, respectively.



architecture by moving the fully-connected topology into a star-shaped structure. Fig-1 gives an overview. Star-Transformer has two kinds of connections. The radical connections preserve the non-local communication and remove the redundancy in fully-connected network. The ring connections embody the local-compositionality prior, which has the same role as in CNNs/RNNs. The direct outcome of our design is the improvement of both efficiency and learning cost: the computation cost is reduced from quadratic to linear as a function of input sequence length. An inherent advantage is that the ring connections can effectively reduce the burden of the unbias learning of local and non-local compositionality and improve the generalization ability of the model. What remains to be tested is whether one shared relay node is capable of capturing the long-range dependencies.

We evaluate the Star-Transformer on three NLP tasks including Text Classification, Natural Language Inference, and Sequence Labelling. Experimental results show that Star-Transformer outperforms the standard Transformer consistently and has less computation complexity. An additional analysis on a simulation task indicates that Star-Transformer preserve the ability to handle with long-range dependencies which is a crucial feature of the standard Transformer.

In this paper, we claim three contributions as the following and our code is available on Github 1:

Footnote 1: [https://github.com/dmlc/dgl](https://github.com/dmlc/dgl) and [https://github.com/fastnlp/fastNLP](https://github.com/fastnlp/fastNLP)

* Compared to the standard Transformer, Star-Transformer has a lightweight structure but with an approximate ability to model the long-range dependencies. It reduces the number of connections from \(n^{2}\) to \(2n\), where \(n\) is the sequence length.
* The Star-Transformer divides the labor of semantic compositions between the radical and the ring connections. The radical connections focus on the non-local compositions and the ring connections focus on the local composition. Therefore, Star-Transformer works for modestly sized datasets and does not rely on heavy pre-training.
* We design a simulation task "Masked Summation" to probe the ability dealing with long-range dependencies. In this task, we verify that both Transformer and Star-Transformer are good at handling long-range dependencies compared to the LSTM and BiLSTM.

## 2 Related Work

Recently, neural networks have proved very successful in learning text representation and have achieved state-of-the-art results in many different tasks.

Modelling Local CompositionalityA popular approach is to represent each word as a low-dimensional vector and then learn the local semantic composition functions over the given sentence structures. For example, Kim (2014); Kalchbrenner et al. (2014) used CNNs to capture the semantic representation of sentences, whereas Cho et al. (2014) used RNNs.

These methods are biased for learning local compositional functions and are hard to capture the long-term dependencies in a text sequence. In order to augment the ability to model the non-local compositionality, a class of improved methods utilizes various self-attention mechanisms to aggregate the weighted information of each word, which can be used to get sentence-level representations for classification tasks Yang et al. (2016); Lin et al. (2017); Shen et al. (2018). Another class of improved methods augments neural networks with a re-reading ability or global state while processing each word Cheng et al. (2016); Zhang et al. (2018).

Modelling Non-Local CompositionalityThere are two kinds of methods to model the non-local semantic compositions in a text sequence directly.

One class of models incorporate syntactic tree into the network structure for learning sentence representations Tai et al. (2015); Zhu et al. (2015).

Another type of models learns the dependencies between words based entirely on self-attention without any recurrent or convolutional layers, such as Transformer Vaswani et al. (2017), which has achieved state-of-the-art results on a machine translation task. The success of Transformer has raised a large body of follow-up work. Therefore, some Transformer variations are also proposed, such as GPT Radford et al. (2018), BERT Devlin et al. (2018), Transformer-XL Dai et al. (2019) , Universal Transformer Dehghani et al. (2018) and CN\({}^{3}\)Liu et al. (2018) 

However, those Transformer-based methods usually require a large training corpus. When applying them on modestly sized datasets, we need the help of semi-supervised learning and unsupervised pretraining techniques (Radford et al., 2018).

Graph Neural NetworksStar-Transformer is also inspired by the recent graph networks (Gilmer et al., 2017; Kipf and Welling, 2016; Battaglia et al., 2018; Liu et al., 2018b), in which the information fusion progresses via message-passing across the whole graph.

The graph structure of the Star-Transformer is star-shaped by introducing a virtual relay node. The radical and ring connections give a better balance between the local and non-local compositionality. Compared to the previous augmented models (Yang et al., 2016; Lin et al., 2017; Shen et al., 2018a; Cheng et al., 2016; Zhang et al., 2018), the implementation of Star-Transform is purely based on the attention mechanism similar to the standard Transformer, which is simpler and well suited for parallel computation.

Due to its better parallel capacity and lower complexity, the Star-Transformer is faster than RNNs or Transformer, especially on modeling long sequences.

## 3 Model

### Architecture

The Star-Transformer consists of one relay node and \(n\) satellite nodes. The state of \(i\)-th satellite node represents the features of the \(i\)-th token in a text sequence. The relay node acts as a virtual hub to gather and scatter information from and to all the satellite nodes.

Star-Transformer has a star-shaped structure, with two kinds of connections in the: the radical connections and the ring connections.

Radical ConnectionsFor a network of \(n\) satellite nodes, there are \(n\) radical connections. Each connection links a satellite node to the shared relay node. With the radical connections, every two non-adjacent satellite nodes are two-hop neighbors and can receive non-local information with a two-step update.

Ring ConnectionsSince text input is a sequence, we bake such prior as an inductive bias. Therefore, we connect the adjacent satellite nodes to capture the relationship of local compositions. The first and last nodes are also connected. Thus, all these local connections constitute a ring-shaped structure. Note that the ring connections allow each satellite node to gather information from its neighbors and plays the same role to CNNs or bidirectional RNNs.

With the radical and ring connections, Star-Transformer can capture both the non-local and local compositions simultaneously. Different from the standard Transformer, we make a division of labor, where the radical connections capture nonlocal compositions, whereas the ring connections attend to local compositions.

### Implementation

The implementation of the Star-Transformer is very similar to the standard Transformer, in which the information exchange is based on the attention mechanism (Vaswani et al., 2017).

Multi-head AttentionJust as in the standard Transformer, we use the scaled dot-product attention (Vaswani et al., 2017). Given a sequence of vectors \({\bf H}\in\mathbb{R}^{n\times d}\), we can use a query vector \({\bf q}\in\mathbb{R}^{1\times d}\) to soft select the relevant information with attention.

\[{\rm Att}({\bf q},{\bf K},{\bf V})={\rm softmax}(\frac{{\bf q}{\bf K}^{T}}{ \sqrt{d}}){\bf V}, \tag{1}\]

where \({\bf K}={\bf HW}^{K},{\bf V}={\bf HW}^{V}\), and \({\bf W}^{K},{\bf W}^{V}\) are learnable parameters.

To gather more useful information from \({\bf H}\), similar to multi-channels in CNNs, we can use multi-head attention with \(k\) heads.

\[{\rm MultiAtt}({\bf q},{\bf H})=({\bf a}_{1}\oplus\cdots\oplus{ \bf a}_{k}){\bf W}^{O}, \tag{2}\] \[{\bf a}_{i}={\rm Att}({\bf q}{\bf W}^{Q}_{i},{\bf HW}^{K}_{i},{ \bf HW}^{V}_{i}),i\in[1,k] \tag{3}\]

where \(\oplus\) denotes the concatenation operation, and \({\bf W}^{Q}_{i},{\bf W}^{K}_{i},{\bf W}^{V}_{i},{\bf W}^{O}\) are learnable parameters.

UpdateLet \({\bf s}^{t}\in\mathbb{R}^{1\times d}\) and \({\bf H}^{t}\in\mathbb{R}^{n\times d}\) denote the states for the relay node and all the \(n\) satellite nodes at step \(t\). When using the Star-Transformer to encode a text sequence of length \(n\), we start from its embedding \({\bf E}=[{\bf e}_{1};\cdots;{\bf e}_{n}]\), where \({\bf e}_{i}\in\mathbb{R}^{1\times d}\) is the embedding of the i-th token.

We initialize the state with \({\bf H}^{0}={\bf E}\) and \({\bf s}^{0}=average({\bf E})\).

 

The update of the Star-Transformer at step \(t\) can be divided into two alternative phases: (1) the update of the satellite nodes and (2) the update of the relay node.

At the first phase, the state of each satellite node \({\bf h}_{i}\) are updated from its adjacent nodes, including the neighbor nodes \({\bf h}_{i-1}\), \({\bf h}_{i+1}\) in the sequence, the relay node \({\bf s}^{t}\), its previous state, and its corresponding token embedding.

\[{\bf C}_{i}^{t}=[{\bf h}_{i-1}^{t-1};{\bf h}_{i}^{t-1};{\bf h}_{i+1 }^{t-1};{\bf e}^{i};{\bf s}^{t-1}], \tag{4}\] \[{\bf h}_{i}^{t}={\rm MultiAtt}({\bf h}_{i}^{t-1},{\bf C}_{i}^{t}), \tag{5}\]

where \({\bf C}_{i}^{t}\) denotes the context information for the \(i\)-th satellite node. Thus, the update of each satellite node is similar to the recurrent network, except that the update fashion is based on attention mechanism. After the information exchange, a layer normalization operation Ba et al. (2016) is used.

\[{\bf h}_{i}^{t}={\rm LayerNorm}({\rm ReLU}({\bf h}_{i}^{t})), i\in[1,n]. \tag{6}\]

At the second phase, the relay node \({\bf s}^{t}\) summarizes the information of all the satellite nodes and its previous state.

\[{\bf s}^{t}={\rm MultiAtt}({\bf s}^{t-1},[{\bf s}^{t-1};{\bf H}^{ t}]), \tag{7}\] \[{\bf s}^{t}={\rm LayerNorm}({\rm ReLU}({\bf s}^{t})). \tag{8}\]

By alternatively updating update the satellite and relay nodes, the Star-Transformer finally captures all the local and non-local compositions for an input text sequence.

Position EmbeddingsTo incorporate the sequence information, we also add the learnable position embeddings, which are added with the token embeddings at the first layer.

The overall update algorithm of the Star-Transformer is shown in the Alg-1.

### Output

After \(T\) rounds of update, the final states of \({\bf H}^{T}\) and \({\bf s}^{T}\) can be used for various tasks such as sequence labeling and classification. For different tasks, we feed them to different task-specific modules. For classification, we generate the fix-length sentence-level vector representation by applying a max-pooling across the final layer and mixing it with \({\bf s}^{T}\), this vector is fed into a Multiple Layer Perceptron (MLP) classifier. For the sequence labeling task, the \({\bf H}^{T}\) provides features corresponding to all the input tokens.

```
1:Input: Number of layers \(T\), embedding of input tokens \(e_{1},\cdots,e_{n}\)
2:// Initialization
3:\({\bf h}_{1}^{0},\cdots,{\bf h}_{n}^{0}\leftarrow{\bf e}_{1},\cdots,{\bf e}_{n}\)
4:\({\bf s}^{0}\gets average({\bf e}_{1},\cdots,{\bf e}_{n})\)
5:for\(t\)from\(1\)to\(T\)do
6:// update the satellite nodes
7:for\(i\)from\(1\)to\(n\)do
8:\({\bf C}_{i}^{t}=[{\bf h}_{i-1}^{t-1};{\bf h}_{i}^{t-1};{\bf h}_{i+1}^{t-1};{ \bf e}^{i};{\bf s}^{t-1}]\)
9:\({\bf h}_{i}^{t}={\rm MultiAtt}({\bf h}_{i}^{t-1},{\bf C}_{i}^{t})\)
10:\({\bf h}_{i}^{t}={\rm LayerNorm}({\rm ReLU}({\bf h}_{i}^{t}))\)
11:// update the relay node
12:\({\bf s}^{t}={\rm MultiAtt}({\bf s}^{t-1},[{\bf s}^{t-1};{\bf H}^{t}])\)
13:\({\bf s}^{t}={\rm LayerNorm}({\rm ReLU}({\bf s}^{t}))\)
```

**Algorithm 1** The Update of Star-Transformer

## 4 Comparison to the standard Transformer

Since our goal is making the Transformer lightweight and easy to train with modestly sized dataset, we have removed many connections compared with the standard Transformer (see Fig-1). If the sequence length is \(n\) and the dimension of hidden states is \(d\), the computation complexity of one layer in the standard Transformer is \(O(n^{2}d)\). The Star-Transformer has two phases, the update of ring connections costs \(O(5nd)\) (the constant \(5\) comes from the size of context information \({\bf C}\)), and the update of radical connections costs \(O(nd)\), so the total cost of one layer in the Star-Transformer is \(O(6nd)\).

In theory, Star-Transformer can cover all the possible relationships in the standard Transformer. For example, any relationship \({\bf h}_{i}\rightarrow{\bf h}_{j}\) in the standard Transformer can be simulated by \({\bf h}_{i}\rightarrow{\bf s}\rightarrow{\bf h}_{j}\). The experiment on the simulation task in Sec-5.1 provides some evidence to show the virtual node \({\bf s}\) could handle long-range dependencies. Following this aspect, we can give a rough analysis of the path length of dependencies in these models. As discussed in the Transformer paper Vaswani et al. (2017), the maximum dependency path length of RNN and Transformer are \(O(n)\), \(O(1)\), respectively. Star-Transformer can pass the message from one node to another node via the relay node so that the maximum dependency path length is also \(O(1)\), with a constant two comparing to Transformer.

Compare with the standard Transformer, all positions are processed in parallel, pair-wise connec 

tions are replaced with a "gather and dispatch" mechanism. As a result, we accelerate the Transformer 10 times on the simulation task and 4.5 times on real tasks. The model also preserves the ability to handle long input sequences. Besides the acceleration, the Star-Transformer achieves significant improvement on some modestly sized datasets.

## 5 Experiments

We evaluate Star-Transformer on one simulation task to probe its behavior when challenged with long-range dependency problem, and three real tasks (Text Classification, Natural Language Inference, and Sequence Labelling). All experiments are ran on a NVIDIA Titan X card. Datasets used in this paper are listed in the Tab-1. We use the Adam Kingma and Ba (2014) as our optimizer. On the real task, we set the embedding size to 300 and initialized with GloVe Pennington et al. (2014). And the symbol "Ours + Char" means an additional character-level pre-trained embedding JMT Hashimoto et al. (2017) is used. Therefore, the total size of embedding should be 400 which as a result of the concatenation of GloVe and JMT. We also fix the embedding layer of the Star-Transformer in all experiments.

Since semi- or unsupervised model is also a feasible solution to improve the model in a parallel direction, such as the ELMo Peters et al. (2018) and BERT Devlin et al. (2018), we exclude these models in the comparison and focus on the relevant architectures.

### Masked Summation

In this section, we introduce a simulation task on the synthetic data to probe the efficiency and non-local/long-range dependencies of LSTM, Transformer, and the Star-Transformer. As mentioned in Vaswani et al. (2017), the maximum path length of long-range dependencies of LSTM and Transformer are \(O(n)\) and \(O(1)\), where \(n\) is the sequence length. The maximum dependency path length of Star-Transformer is \(O(1)\) with a constant two via the relay node. To validate the ability to deal with long-range dependencies, we design a simulation task named "Masked Summation". The input of this task is a matrix \({\bf X}\in\mathbb{R}^{n\times d}\), it has \(n\) columns and each column has \(d\) elements. The first dimension indicates the mask value \({\bf X}_{i0}\in\{0,1\}\), \(0\) means the column is ignored in summation. The rest \(d-1\) elements are real numbers in drawn uniformly from the range \([0,1)\). The target is a \(d-1\) dimensional vector which equals the summation of all the columns with the mask value \(1\). There is an implicit variable \(k\) to control the number of \(1\) in the input. Note that a simple baseline is always guessing the value \(k/2\).

The evaluation metric is the Mean Square Error (MSE), and the generated dataset has (10K/10k/10k) samples in (train/dev/test) sets. The Fig-2 show a case of the masked summation task.

The mask summation task asks the model to recognize the mask value and gather columns in different positions. When the sequence length \(n\) is significantly higher than the number of the columns \(k\), the model will face the long-range dependencies probo

\begin{table}
\begin{tabular}{c|l c l l l l l} \hline \hline Dataset & Train & Dev. & Test & \(|V|\) & H DIM & \#head & head DIM \\ \hline  & Masked Summation & 10k & 10k & 10k & - & 100 & 10 & 10 \\ \hline  & SST Socher et al. (2013) & 8k & 1k & 2k & 20k & 300 & 6 & 50 \\ \hline MTL–16 \({}^{\dagger}\) & \multicolumn{9}{l}{Apparel Baby Books Camera} \\ Liu et al. (Liu et al., 2017) & \multicolumn{9}{l}{DVD Electronics Health IMDB} \\
2017 & Kitchen Magazines MR Music Software Sports Toy Video & \multicolumn{9}{l}{1400} \\ \hline SNLI Bowman et al. (2015) & 550k & 10k & 10k & 34k & 600 & 6 & 100 \\ \hline PTB POS Marcus et al. (1993) & 38k & 5k & 5k & 44k & 300 & 6 & 50 \\ \hline CoNLL03 Sang and Meulder (2003) & 15k & 3k & 3k & 25k & 300 & 6 & 50 \\ \hline OntoNotes NER Pradhan et al. (2012) & 94k & 14k & 8k & 63k & 300 & 6 & 50 \\ \hline \hline \end{tabular}
\end{table}
Table 1: An overall of datasets and its hyper-parameters, “H DIM, #head, head DIM” indicates the dimension of hidden states, the number of heads in the Multi-head attention, the dimension of each head, respectively. MTL-16\({}^{\dagger}\) consists of 16 datasets, each of them has 1400/200/400 samples in train/dev/test.

 

formance curves of models on various lengths. Although the task is easy, the performance of LSTM and BiLSTM dropped quickly when the sequence length increased. However, both Transformer and Star-Transformer performed consistently on various lengths. The result indicates the Star-Transformer preserves the ability to deal with the non-local/long-range dependencies.

Besides the performance comparison, we also study the speed with this simulation task since we could ignore the affection of padding, masking, and data processing. We also report the inference time in the Fig-3b, which shows that Transformer is faster than LSTM and BiLSTM a lot, and Star-Transformer is faster than Transformer, especially on the long sequence.

### Text Classification

Text classification is a basic NLP task, and we select two datasets to observe the performance of our model in different conditions, Stanford Sentiment Treebank(SST) dataset [21] and MTL-16 [40] consists of 16 small datasets on various domains. We truncate the sequence which its length higher than 256 to ensure the standard Transformer can run on a single GPU card.

For classification tasks, we use the state of the relay node \({\bf s}^{T}\) plus the feature of max pooling on satellite nodes \(\max({\bf H}^{T})\) as the final representation and feed it into the softmax classifier. The description of hyper-parameters is listed in Tab-1 and Appendix.

Results on SST and MTL-16 datasets are listed in Tab-2,3, respectively. On the SST, the Star-Transformer achieves 2.5 points improvement against the standard Transformer and beat the most models.

Also, on the MTL-16, the Star-Transformer outperform the standard Transformer in all 16 datasets, the improvement of the average accuracy is 4.2. The Star-Transformer also gets better results compared with existing works. As we mentioned in the introduction, the standard Transformer requires large training set to reveal its power. Our experiments show the Star

\begin{table}
\begin{tabular}{l l} \hline \hline Model & Acc \\ \hline BiLSTM [40] & 49.8 \\ Tree-LSTM [40] & 51.0 \\ CNN-Tensor [40] & 51.2 \\ Emb + self-att [7] & 48.9 \\ BILSTM + self-att [21] & 50.4 \\ CNN + self-att [21] & 50.6 \\ Dynamic self-att [21] & 50.6 \\ DiSAN [21] & 51.7 \\ \hline Transformer & 50.4 \\ Ours & **52.9** \\ \hline \hline  & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Test Accuracy on SST dataset.

Figure 2: An example of the masked summation task, the input is a sequence of \(n\) vectors, each vector has \(d\) dimension, and there are total \(k\) vectors which have the mask value equals \(1\). The target is the summation of such masked vectors. In this figure, \(n=8,k=3,d=3\).

 

Transformer could work well on the small dataset which only has 1400 training samples. Results of the time-consuming show the Star-Transformer could be 4.5 times fast than the standard Transformer on average.

### Natural Language Inference

Natural Language Inference (NLI) asks the model to identify the semantic relationship between a premise sentence and a corresponding hypothesis sentence. In this paper, we use the Stanford Natural Language Inference (SNLI) Bowman et al. (2015) for evaluation. Since we want to study how the model encodes the sentence as a vector representation, we set Star-Transformer as a sentence vector-based model and compared it with sentence vector-based models.

In this experiment, we follow the previous work Bowman et al. (2016) to use concat\(({\bf r}_{1},{\bf r}_{2},\|{\bf r}_{1}-{\bf r}_{2}\|,{\bf r}_{1}-{\bf r}_{2})\) as the classification feature. The \({\bf r}_{1},{\bf r}_{2}\) are representations of premise and hypothesis sentence, it is calculated by \({\bf s}^{T}+\max({\bf H}^{T})\) which is same with the classification task. See Appendix for the detail of hyper-parameters.

As shown in Tab-4, the Star-Transformer outperforms most typical baselines DiSAN, SPINN) and achieves comparable results compared with the state-of-the-art model. Notably, our model beats standard Transformer by a large margin, which is easy to overfit although we have made a careful hyper-parameters' searching for Transformer.

The SNLI dataset is not a small dataset in NLP area, so improving the generalization ability of the Transformer is a significant topic.

The best result in Tab-4 Yoon et al. (2018) using a large network and fine-tuned hyper-parameters, they get the best result on SNLI but an undistinguished result on SST, see Tab-2.

### Sequence Labelling

To verify the ability of our model in sequence labeling, we choose two classic sequence labeling tasks: Part-of-Speech (POS) tagging and Named Entity Recognition (NER) task.

Three datasets are used as our benchmark: one POS tagging dataset from Penn Treebank (PTB) Marcus et al. (1993), and two NER datasets from CoNLL2003 Sang and Meulder (2003), CoNLL2012 Pradhan et al. (2012). We use the fi

\begin{table}
\begin{tabular}{l c} \hline \hline Model & Acc \\ \hline BiLSTM Liu et al. (2016) & 83.3 \\ BiLSTM + self-att Liu et al. (2016) & 84.2 \\
300D SPINN-PI Bowman et al. (2016) & 83.2 \\ Tree-based CNN Mou et al. (2016) & 82.1 \\
4096D BiLSTM-max Conneau et al. (2017) & 84.5 \\
300D DiSAN Shen et al. (2018a) & 85.6 \\ Residual encoders Nie and Bansal (2017) & 86.0 \\ Gumbel TreeLSTM Choi et al. (2018) & 86.0 \\ Reinforced self-att Shen et al. (2018b) & 86.3 \\
2400D Multiple DSA Yoon et al. (2018) & 87.4 \\ \hline Transformer & 82.2 \\ Star-Transformer & **86.0** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Test Accuracy on SNLI dataset.

\begin{table}
\begin{tabular}{l c c c c|c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{3}{c|}{Acc (\%)} & \multicolumn{3}{c}{Test Time (ms)} & \multirow{2}{*}{Len.} \\  & Star-Transformer & Transformer & BiLSTM & SLSTM & Star-Transformer & Transformer & BiLSTM \\ \hline Apparel & 88.25 & 82.25 & 86.05 & 85.75 & 11 & 34 & 114 & 65 \\ Baby & 87.75 & 84.50 & 84.51 & 86.25 & 11 & 50 & 141 & 109 \\ Books & 87.00 & 81.50 & 82.12 & 83.44 & 11 & 55 & 135 & 131 \\ Camera & 92.25 & 86.00 & 87.05 & 90.02 & 11 & 52 & 125 & 116 \\ DVD & 86.00 & 77.75 & 83.71 & 85.52 & 11 & 58 & 143 & 139 \\ Electronics & 82.75 & 81.50 & 82.51 & 83.25 & 10 & 49 & 136 & 105 \\ Health & 87.50 & 83.50 & 85.52 & 86.50 & 10 & 45 & 131 & 85 \\ IMDB & 88.00 & 82.50 & 86.02 & 87.15 & 12 & 77 & 174 & 201 \\ Kitchen & 85.50 & 83.00 & 82.22 & 84.54 & 11 & 47 & 130 & 92 \\ Magazines & 92.75 & 89.50 & 92.52 & 93.75 & 11 & 51 & 136 & 115 \\ MR & 79.00 & 77.25 & 75.73 & 76.20 & 12 & 14 & 26 & 22 \\ Music & 83.50 & 79.00 & 78.74 & 82.04 & 11 & 53 & 137 & 118 \\ Software & 90.50 & 85.25 & 86.73 & 87.75 & 11 & 53 & 140 & 117 \\ Sports & 86.25 & 84.75 & 84.04 & 85.75 & 10 & 49 & 134 & 103 \\ Toys & 88.00 & 82.00 & 85.72 & 85.25 & 11 & 46 & 137 & 96 \\ Video & 86.75 & 84.25 & 84.73 & 86.75 & 11 & 56 & 146 & 131 \\ \hline
**Average** & **86.98** & 82.78 & 84.01 & 85.38 & **10.94** & 49.31 & 130.3 & 109.1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Test Accuracy over MTL-16 datasets. “Test Time” means millisecond per batch on the test set (batch_size is 128). “Len.” means the average sequence length on the test set.

 

nal state of satellite nodes \({\bf H}^{T}\) to classify the label in each position. Since we believe that the complex neural network could be an alternative of the CRF, we also report the result without CRF layer.

As shown in Tab-5, Star-Transformer achieves the state-of-the-art performance on sequence labeling tasks. The "Star-Transformer + Char" has already beat most of the competitors. Star-Transformer could achieve such results without CRF, suggesting that the model has enough capability to capture the partial ability of the CRF. The Star-Transformer also outperforms the standard Transformer on sequence labeling tasks with a significant gap.

### Ablation Study

In this section, we perform an ablation study to test the effectiveness of the radical and ring connections.

We test two variants of our models, the first variants (a) remove the radical connections and only keep the ring connections. Without the radical connections, the maximum path length of this variant becomes \(O(n)\). The second variant (b) removes the ring connections and remains the radical connections. Results in Tab-6 give some insights, the variant (a) loses the ability to handle long-range dependencies, so it performs worse on both the simulation and real tasks. However, the performance drops on SNLI and CoNLL03 is moderate since the remained ring connections still capture the local features. The variant (b) still works on the simulation task since the maximum path length stays unchanged. Without the ring connections, it loses its performance heavily on real tasks. Therefore, both the radical and ring connections are necessary to our model.

## 6 Conclusion and Future Works

In this paper, we present Star-Transformer which reduce the computation complexity of the standard Transformer by carefully sparsifying the topology. We compare the standard Transformer with other models on one toy dataset and 21 real datasets and find Star-Transformer outperforms the standard Transformer and achieves comparable results with state-of-the-art models.

This work verifies the ability of Star-Transformer by excluding the factor of unsupervised pre-training. In the future work, we will investigate the ability of Star-Transformer by unsupervised pre-training on the large corpus. Moreover, we also want to introduce more NLP prior knowledge into the model.

\begin{table}
\begin{tabular}{l c c c} \hline \hline Model & SNLI Acc & CoNLL03 MSE \\  & Acc & Acc & MSE \\ \hline Star-Transformer & 86.0 & 90.93 & 0.0284 \\ variant (a) -radical & 84.0 & 89.35 & 0.1536 \\ variant (b) -ring & 77.6 & 79.36 & 0.0359 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Test Accuracy on SNLI dataset, CoNLL2003 NER task and the Masked Summation \(n=200,k=10,d=10\).

\begin{table}
\begin{tabular}{l|c c|c|c c} \hline \hline \multirow{2}{*}{Model} & \multicolumn{2}{c|}{JOS} & \multicolumn{2}{c}{NER} \\ \cline{2-5}  & Adv Tech & PTB & CoNLL2003 CoNLL2012 \\ \cline{2-5}  & char & CRF & Acc & F1 & F1 \\ \hline Ling et al. [20] & ✓ & ✓ & 97.78 & - & - \\ Collobert et al. [20] & ✓ & ✓ & 97.29 & 89.59 & - \\ Huang et al. [20] & ✓ & ✓ & 97.55 & 90.10 & - \\ Chiu and Nichols [20] & ✓ & ✓ & - & 90.69 & 86.35 \\ Ma and Hovy [20] & ✓ & ✓ & 97.55 & 91.06 & - \\ Nguyen et al. [20] & ✓ & ✓ & - & 91.2 & - \\ Chiu and Nichols [20] & ✓ & ✓ & - & 91.62 & 86.28 \\ Zhang et al. [20] & ✓ & ✓ & 97.55 & 91.57 & - \\ Akhundov et al. [20] & ✓ & ✓ & 97.43 & 91.11 & 87.84 \\ \hline Transformer & & & 96.31 & 86.48 & 83.57 \\ Transformer + Char & ✓ & & 97.04 & 88.26 & 85.14 \\ Star-Transformer & & & 97.14 & 90.93 & 86.30 \\ Star-Transformer + Char & ✓ & & 97.64 & 91.89 & 87.64 \\ Star-Transformer + Char + CRF & ✓ & ✓ & **97.68** & **91.98** & **87.88** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results on sequence labeling tasks. We list the “Advanced Techniques” except widely-used pre-trained embeddings (GloVe, Word2Vec, JMT) in columns. The “Char” indicates character-level features, it also includes the Capitalization Features, Suffix Feature, Lexicon Features, etc. The “CRF” means an additional Conditional Random Field (CRF) layer.

 



## Acknowledgments

We would like to thank the anonymous reviewers for their valuable comments. The research work is supported by Shanghai Municipal Science and Technology Commission (No. 17JC1404100 and 16JC1420401), National Key Research and Development Program of China (No. 2017YFB1002104), and National Natural Science Foundation of China (No. 61672162 and 61751201).

## References

* Akhundov et al. (2018) Adnan Akhundov, Dietrich Trautmann, and Georg Groh. 2018. Sequence labeling: A practical approach. _CoRR_, abs/1808.03926.
* Ba et al. (2016) Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. _CoRR_, abs/1607.06450.
* Battaglia et al. (2018) Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. _arXiv preprint arXiv:1806.01261_.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In _EMNLP_, pages 632-642. The Association for Computational Linguistics.
* Bowman et al. (2016) Samuel R. Bowman, Jon Gauthier, Abhinav Rasatogi, Raghaw Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In _ACL (1)_. The Association for Computer Linguistics.
* Cheng et al. (2016) Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long short-term memory-networks for machine reading. In _Proceedings of the 2016 Conference on EMNLP_, pages 551-561.
* Chiu and Nichols (2016a) Jason Chiu and Eric Nichols. 2016a. Sequential labeling with bidirectional lstm-cnns. In _Proc. International Conf. of Japanese Association for NLP_, pages 937-940.
* Chiu and Nichols (2016b) Jason P. C. Chiu and Eric Nichols. 2016b. Named entity recognition with bidirectional lstm-cnns. _TACL_, 4:357-370.
* Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _Proceedings of EMNLP_.
* Choi et al. (2018) Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In _AAAI_, pages 5094-5101. AAAI Press.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2493-2537.
* Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In _EMNLP_, pages 670-680. Association for Computational Linguistics.
* Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. _CoRR_, abs/1901.02860.
* Dehghani et al. (2018) Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal transformers. _CoRR_, abs/1807.03819.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. _CoRR_, abs/1810.04805.
* Gilmer et al. (2017) Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In _ICML_, pages 1263-1272.
* Hashimoto et al. (2017) Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In _EMNLP_, pages 1923-1933. Association for Computational Linguistics.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. _CoRR_, abs/1508.01991.
* Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In _Proceedings of ACL_.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on EMNLP_, pages 1746-1751.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _CoRR_, abs/1412.6980.
* Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. _arXiv preprint arXiv:1609.02907_.
* Lei et al. (2015) Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2015. Molding cnns for text: non-linear, non-consecutive convolutions. In _EMNLP_, pages 1565-1575. The Association for Computational Linguistics

Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard H. Hovy. 2015. When are tree structures necessary for deep learning of representations? In _EMNLP_, pages 2304-2314. The Association for Computational Linguistics.
* Lin et al. (2017) Zhouhan Lin, Mo Feng, Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. _arXiv preprint arXiv:1703.03130_.
* Ling et al. (2015) Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015. Finding function in form: Compositional character models from vocabulary word representation. In _EMNLP_, pages 1520-1530. The Association for Computational Linguistics.
* Liu et al. (2018a) Pengfei Liu, Shuaichen Chang, Xuanjing Huang, Jian Tang, and Jackie Chi Kit Cheung. 2018a. Contextualized non-local neural networks for sequence learning. _arXiv preprint arXiv:1811.08600_.
* Liu et al. (2018b) Pengfei Liu, Jie Fu, Yue Dong, Xipeng Qiu, and Jackie Chi Kit Cheung. 2018b. Multi-task learning over graph structures. _arXiv preprint arXiv:1811.10211_.
* Liu et al. (2017) Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In _ACL (1)_, pages 1-10. Association for Computational Linguistics.
* Liu et al. (2016) Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional LSTM model and inner-attention. _CoRR_, abs/1605.09090.
* Ma and Hovy (2016)Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _ACL (1)_. The Association for Computer Linguistics.
* Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. _Computational Linguistics_, 19(2):313-330.
* Mou et al. (2016) Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In _ACL (2)_. The Association for Computer Linguistics.
* Nguyen et al. (2016) Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2016. J-NERD: joint named entity recognition and disambiguation with rich linguistic features. _TACL_, 4:215-229.
* Nie and Bansal (2017) Yixin Nie and Mohit Bansal. 2017. Shortcut-stacked sentence encoders for multi-domain inference. In _RepEval@EMNLP_, pages 41-45. Association for Computational Linguistics.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, pages 1532-1543. ACL.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _NAACL-HLT_, pages 2227-2237. Association for Computational Linguistics.
* Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In _EMNLP-CoNLL Shared Task_, pages 1-40. ACL.
* Radford et al. (2018)Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
* Tiong Kim Sang and De Meulder (2003)Erik F. Tiong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In _CoNLL_, pages 142-147. ACL.
* Shen et al. (2018a) Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chenguzi Zhang. 2018a. Disan: Directional self-attention network for rnn/cnn-free language understanding. In _AAAI_, pages 5446-5455. AAAI Press.
* Shen et al. (2018b) Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. 2018b. Reinforced self-attention network: a hybrid of hard and soft attention for sequence modeling. In _IJCAI_, pages 4345-4352. ijcai.org.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _EMNLP_, pages 1631-1642. ACL.
* Tai et al. (2015) Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In _ACL (1)_, pages 1556-1566. The Association for Computer Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NIPS_, pages 6000-6010.
* Yang et al. (2016)Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of NAACL_, pages 1480-1489.
* Yoon et al. (2018)Deunsol Yoon, Dongbok Lee, and SangKeun Lee. 2018. Dynamic self-attention : Computing attention over words dynamically for sentence embedding. _CoRR_, abs/1808.07383.
* Zhang et al. (2018) Yue Zhang, Qi Liu, andinfeng Song. 2018. Sentence-state LSTM for text representation. In _ACL (1)_, pages 317-327. Association for Computational Linguistics.

 

Xiao-Dan Zhu, Parinaz Sobhani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In _ICML_, pages 1604-1612.