Transforming Recurrent Neural Networks with Attention and Fixed-point Equations
Authors: Zhaobin Xu
Baotian Hu
Buzhou Tang
Venue:  (2021)
Keywords: Fixed-point
Attention
Feed Forward Network
Transformer
Recurrent Neural Network
Deep Learning

Abstract: Transformer has achieved state of the art performance in multiple Natural Language Processing tasks recently. Yet the Feed Forward Network(FFN) in a Transformer block is computationally expensive. In this paper, we present a framework to transform Recurrent Neural Networks(RNNs) and their variants into self-attention-style models, with an approximation of Banach Fixed-point Theorem. Within this framework, we propose a new model, StarSaber, by solving a set of equations obtained from RNN with Fixed-point Theorem and further approximate it with a Multi-layer Perceptron. It provides a view of stacking layers. StarSaber achieves better performance than both the vanilla Transformer and an improved version called ReZero on three datasets and is more computationally efficient, due to the reduction of Transformer's FFN layer. It has two major parts. One is a way to encode position information with two different matrices. For every position in a sequence, we have a matrix operating on positions before it and another matrix operating on positions after it. The other is the introduction of direct paths from the input layer to the rest of layers. Ablation studies show the effectiveness of these two parts. We additionally show that other RNN variants such as RNNs with gates can also be transformed in the same way, outperforming the two kinds of Transformers as well.

Under review as a conference paper at ICLR 2021
TRANSFORMING RECURRENT NEURAL NETWORKS WITH ATTENTION AND FIXED-POINT EQUATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Transformer has achieved state of the art performance in multiple Natural Lan- guage Processing tasks recently. Yet the Feed Forward Network(FFN) in a Trans- former block is computationally expensive, In this paper, we present a frame- work to transform Recurrent Neural Networks(RNNs) and their variants into self- attention-style models, with an approximation of Banach Fixed-point Theorem. model, StarSaber, by solving set of
Within this framework, propose equations obtained from RNN with Fixed-point Theorem and further approximate it with a Multi-layer Perceptron. It provides view of stacking layers. StarSaber achieves better performance than both the vanilla Transformer and an improved version called ReZero on three datasets and is more computationally efficient, due to the reduction of Transformer's 's FFN layer. It has two major parts. One is a way to encode position information with two different matrices. For every position in a sequence, we have a matrix operating operating on positions after it. The other is the introduction of direct paths from the input layer to the rest of layers. Ablation studies show the effectiveness of these two parts. We additionally show that other RNN variants such as RNNs with gates also be transformed in the same way, outperforming the two kinds of Transformers as well.
positions before it and another matrix
INTRODUCTION
Recurrent Neural Network, known RNN, has been widely applied various tasks in the last decade, such as Neural Machine Translation (Kalchbrenner & Blunsoo 2013 Sutskever et al.) 2014). Text Classification (Zhou et al. 2016). Name Entity Recognition (Zhang & Yang) 2018 Chiu & Nichols 2016). Machine Reading Comprehension (Hermann et al. 2015 Kadlec et al. 2016) and Natural Language Inference (Chen et al., 2017] Wang al.) 2017)) Models applied these tasks are not the vanilla RNNs but two of their famous variants, Gated Recurrent Unit (Cho et al.)
2014), known as GRU, and Long Short Term Memory (Hochreiter Schmidhuber 1997), known as LSTM, in which gates play an important role. RNNs are hard to be computed paralleily. They are not bidirectional either, meaning that a word cannot utilize the information of words coming after it. A general way to alleviate this problem is to reverse the input sequence and combine results given by two different RNN encoders with operations like concatenation and addition.
However, Transformer (Vaswani et al. 2017) has provided better solution. It is based on purely attention mechanism, which has been widely used in Neural Machine Translation since Bahdanau et al.] (2014). Models based on self-attention mechanism are mostly Transformer and its variants, such as Transformer-y XL (Dai al. 2019), Universal Transformer (Dehghani et al. 2019) and Stansfraasformer (Guo et al., 2019). Compared with recurrent units such as GRU and LSTM, self-attention-stye style models can computed parallelly, which means they suit better large-scale training. But each of these Transformers has an FFN layer with a very high vector dimension, which still is the bottleneck to improve the computation efficency.
In this paper, we present a new framework based on Banach Fixed-point Theorem to trans- form the vanilla RNN and its variants with self-attention mechanism. StarSaber, one of such transformed models, outperforms both the vanilla Transformer and ReZero (Bachlechner et al. 2020) in our experiments with less parameters and thus less computational power. To start with,

---

Under review as a conference paper at ICLR 2021
need a different view of attention. Attention is way to build relation graph between words, and the vanilla RNN is nothing but a model with a relation graph as a chain. This graph is in fact represented with an adjacent matrix, which is computed by mapping each pair of positions to positive real number and normalizing the numbers related to each position, which are just those in the same row of the adjacent matrix, that they sum up to one.
The vanilla RNN updates hidden states through a chain, that is, the hidden state for each po- sition only depends on that in the previous position. However, if we have this relation graph, the hidden state for each position depends on hidden states for all other positions in a sequence. This is where we obtain equations. In our opinion, a bidirectional RNN is defined by some equations and Banach Fixed-point Theorem inspires us to iterate according to them. When we fix the number of iterations and specify distinct weights for each of them, self-attention-stylee model is then constructed.
In Transformer, Position Embedding(PE) as a way to capture word order information in lan- guage by adding matrix to the input, is indispensable. But in StarSaber, position encoding is done in the aggregation step after the construction of relation graph. For each position, we sum up linear transformations of hidden states in all positions with the corresponding weights in the relation matrix in order to get an attention vector. In the calculation of such a vector, we specify different linear transformation weights for the "future" and the "past". Then the hidden vector for position is computed with the corresponding attention vector and an input vector, which turns into a direct path from the input layer to each hidden layer. And we directly drop the FFN layer in Transformer achieving still competive and even better results with much less parameters on three datasets provided by CLUE (Xu et al. 2020): the AFQMC dataset of Sentence Similarity, the TNEWS dataset of Text Classification and the CMNLI dataset of Natural Language Inference. More importantly, our derivation of StarSaber shows universal way to transform different RNNs, such as LSTM and GRU discussed in the following content, providing possibilities other than Transformers for self- atffeatention models.
2 RELATED WORK
Gates were first introduced into recurrent networks in LSTM and were rediscovered and simplified in GRU. Gate mechanism is an operation that multiplies an output by a single sigmoid layer of the input and is often approaches which addresses this problem should achieve similar results to LSTM and GRU. In this paper, we show by experiments that StarSaber which doesn't 't have such a problem, gates can also help improve the performance.
approach to address the gradient vanishing issue. But if only so, other
Attention in sequence modeling is weighted sum of the output in each position of se- quence, which simulates the way man distributes his attention to all its parts. Weights this sum are given by certain function of some inputs. And self-attention is an approach computing both the weighted sum and weights on the same sequence without any other inputs. There are different types of attention like multi-head attention and scaled dot product attention in Transformer, attention based on addition in Bahdanau et al.] (2014), and bilinear attention in Luong et al. (2015). Our model applies the bilinear attention in the construction of word relation graph.
Residual Connection was proposed by He et al. (2015). It alleviates the problem of training deep neural networks. In Natural Language Processing, Residual Connection alleviates both the gradient vanishing problem and the degration problem of deep networks. Our model uses a weighted residual connection (Bachlechner et al.] 2020) which further alleviates the degration problem. Another similar idea is the highway connection (Srivastava et al., 2015). In this paper, we inspect the gate mechanism in our eeff-ateeniion-style model. Note that the highway connection can also fit into our framework, which is a fixed-point generalization of GRU.
Pretraining has proved to be extremely useful since Embeddings from Language Models(ELMO) (Peters al. 2018). Many works that follow such as BERT (Devlin al.) 2018), ALBERT (Lan et al. 2020). XLNET (Yang et al.) 2019) have outperformed humans. Pretraining is a training pattern which trains language model, usually extremely large, enormous dataset with one
2

---

Under review as a conference paper at ICLR 2021
unsupervised tasks and fine-tunes it other datasets and tasks. There are two types of language models in general, known as auto-regressive models(e.g.. XLNET) and auto-encoder large dataset requires resources. We show in this paper that only pretraining on a dadtaset formed by collecting the training, development and test inputs together can well improve the performance, revealing the significance of pretraining tasks.
ones(e.g,, BERT). However, pretraining
MLM the unsupervised task utilized by BERT pretrain. It randomly masks some pieces of a sentence and train the model to predict what has been masked. In this way, knowledge is gained and the model is initialized for downstream tasks. However, experiments show that even not pretrained on a large dataset, using MLM to pretrain dataset formed by collecting the training, development and test inputs can still improve the performance, inspiring us that more flexible and task-related pretraining method is beneficial.
MODEL ARCHITECTURE
3.1 RECURRENT NEURAL NETWORKS AND BIDIRECTIONAL EQUATIONS WITH SELF-ATTENTION
This section follows the intuition we have discussed before. The vanilla RNN formulas are listed below:
h, = Lan Wx,)
self-attention, we don't just utilize the hidden state from the previous position but hidden states from all positions, to compute hidden vector for position n. To encode information of relative order, we specify distinct linear transformation weights. For simplicity, we ignore all bias terms in the following derivation. Following the bilinear self-attention mechanism, have:
hn = tanh( An + Vxn)
An = Σ Gnoule +
Σ
GngUTight he!
isn
i2n
G. softmax(g, -1)
Jni
(2)
Σ Jnj
1. Wh )2
In. exp(
Vd
What we have done here is replacing An with an attention vector A,, Notice a fact that in the first equality, h appears on both the left-hand side and the right-hand side(we use h to compute An), turning it into equation. This means bidirectional model is defined by set of equations, because the word relation graph constructed by attention is not free of loops. Moreover, introducing equations can be we view the non-linear function on the right hand side as an updating operation and the hidden vector we obtain in each position semantic representation, it simply means that when the model "reads" the whole sentence again based on the current understanding, it should produce the same representation, meaning that has "fully understands" the whole sentence and makes no changes on the hidden vectors.
constraint to obtain stable representation of sentence. Intuitively, if
3.2 GENERALIZE EQUATIONS WITH FIXED-POINT
Now we have an equation to solve, which extremely complex and difficult. But Banach Fixed- point theorem shows way.
Theorem 3.1 (Banach Fixed-point Theorem) For any real-valued function flx), if 1 < 1, then iteration process 1n+1 (z.) converges and lim In = x", where x = f(x*).
n
If not stated, sum without a limit is to sum over all possible values.
2 All these U, V, Ws are matrices that satisfy rules of the matrix-vector product. The hyperparameter d here the hidden size. The attention here is scaled for faster convergence.
3

---

Under review as a conference paper at ICLR 2021
The equation above is equation of an iterative pattern, and this Theorem just tells us that as long as we keep iterating, we will obtain a root of the equation if its jacobian matrix satisfies some conditions. The iterative pattern is given follows:
h [+1 tanh A Vsn))
A Σ Uleute + Σ
G ht
22
G softmarls -1)
Fri
(3)
Σ Inj
(ha) ( Vd
9.. exp(
We can then iterate till it converges. Similar ideas are in Bai et al. (2019), where the authors solve the fixed-point directly with very high computational cost. Sometimes cannot even converge to fixed- point, since the convergence condition is quite strict. A sufficient condition for convergence is that all parameter matrices are strictly orthogonal, making the optimization problem hard. Therefore, if want to obtain faster and more stable model, we approximate it with Multi-layer Perceptron (MLP)) and relax the condition convergence. In addition, we allow our model to assign different weights for different layers. The reason why don't 't reuse parameters in each layer is that iterating with the same set of parameters without a constraint orthogonality often diverges. Even we fix the number of iterations, is still hardly possible converge to the correct fixed-point. In this case, specifying different weights for each layer allows our model to learn a better fit for the whole iteration process. Therefore, we have
1+1 = tanh(A + V'xa)
4
Σ Ghau' Σ Q'
i<a
12n
I's
G softmar .1)
(4)
Σ gnj
S'4 exp( (h,,) (F
Vd
Here also need initial point to start the iteration. In model, we choose the input sequence itself to be the initial value, that to set h² = vy. In more general cases, the initial value may be linear transformation of the input just some fixed vector like zero one.
3.3 RESIDUAL CONNECTIONS
Since we decide to approximate the iteration process with an MLP, Residual Connection then indispensable in for helps to alleviate the problem degration. However, its magnitude, which is the fixed scaling number, needs to be tuned mannually. If we allow it to be automatically tuned by our model, the whole model can be written as follows:
(5) The rest of formulas are the same above. The a' here is crucial weight initialized to be one or zero in every layer. In Bachlechner et al. (2020) Transformer. But in our experiments we don't it train extremely deep networks with thousand or more layers. Thus we initialize it to one since we find that it speeds up convergence.
[+1 hi + + a''tanh(A + V'xx,)
initialized to be zero in order stabilize
3.4 MODEL SUMMARY
The derivation above has demonstrated how to transform the vanilla RNN into self-attention-ssylle model. To summarize, the structure of StarSaber can be described by Figure [ The Attention Graph here is the relation construction process returning matrix G. It is exactly what happens in the last two formulas shown above. Masked Attention here is how we implement Position Encoding. And is the weight for Residual Connection.
4

---

Under review as a conference paper at ICLR 2021
softmax
a xxaanh
Masked Attention
Nx
Linear Transform
Attention Graph
Linear Transform
Linear Transform
Layer Input
Input Embeddings
Figure 1: The model structure llayer-Starber Starber
4 EXPERIMENTS
4.1 A SPECIAL TYPE OF PRETRAINING: EFFECTIVENESS OF MASKED LANGUAGE
Although we don't 't pretrain on an enormous dataset, we can still improve the performance by utiliz- ing the common pretraining task. In practice, we pretrain model collection of all the inputs without their labels from the training, development and test set. We use dynamic mask (Liu et al.] 2019]. which is to mask different positions of sample every epoch. The reconstruction loss is computed on all positions of a sequence instead of only those masked ones. Masking probability in every position set to be 0.3. Different from BERT, we don't have MASK symbol. Instead every masked position is replaced with a word uniformally selected from the whole vocabulary.
Table 1: Model Configurations
Configurations
AFQMC
TNEWS CMNLI
BiLSTM
ReZero
4
Layer
Transformer
4
4
StarSaber
4
Werd2Vec
BiLSTM
1.45M
1.45M
1.45M
ReZero
3.97M
3.97M
3.97M
Parameter
Transformer
3.97M
3.98M
4.35M
StarSaber
2.89M
2.89M
2.89M
ReZero
6
12
Transformer
6
6
12
Layer
StarSaber-1
12
12
StarSaber-2
6
6
12
Pretrained
ReZero
18.19M
19.8IM
36.37M
Transforner
18.20M
19.82M
36.75M
# Parameter
StarSaber-1
13.47M
15.09M
27.30M
StarSaber-2
7.17M
8.79M
14.69M
5

---

Under review as a conference paper at ICLR 2021
4.2 SETTINGS
Experiments are conducted on three datasets from Xu et al. (2020). namely the AFQMC(Sentence Similarity) dataset, the TNext(Text Classification) dataset and the CMNLI(Natural Language In- ference) dataset. For the two text matching tasks, we concatenate the two imput sentences with seperation. The Adam optimizer (Kingma & Baj 2014) used. The learning rate is set to le-3 in pretraining and le-4 in fine-tuning for ReZero and StarSaber. A learning rate of le-4 in both pretraining and fine-tuning is applied to the vanilla Transformer. Random seed set to 10. We compare StarSaber with BiLSTM and two Transformers. For StarSaber and the two Transformers, we have two versions each for every dataset, namely pretrained and not pretrained ones. For LSTM, only have not-pretrained version. In such versions, utilize word embeddings trained from Word2Vec (Mikolov et al. a) with data collected from Wikipedia provided by Li et al. (2018) and finetune the embeddings the pretraining dataset for each task. In training process, we freeze these embeddings. Our BiL.TTM concatenates features encoded by two LSTM encoders of oppo- site directions. In ReZero Layer Normalization (Lei Ba et 2016) and the warmup procedure are dropped. Both Transformer and ReZero have heads and take 4 Hidden-size as the size for the FFN layer. For models using Word2Vec, the hidden size in every layer and the input size are all set to 300. For models pretrained, they are set to 512. Early Stopping is used and the loss function Cross-Ents pooss-Entropy
pretraining but Hinge Loss training. More details on model configurations are shown in Table 3 All results are submitted online to www c clueennchha rk com and test labels are not available. Due to the submission limit of 10 times in a month, we cannot try more configuration settings.
4.3
RESULTS AND ANALYSIS
Table 2: Dataset Statistics
Dataset AFQMC
# Training Sample
#Dev Sample
#Test Sample
# #Characters
34334
4316
3861
1.14M
TNEWS
53360
10000
10000
1.63M
CMNLI
391782
12426
13880
22.24M
Table 3: Accuracy(%) on datasets
AFQMC
TNEWS
CMNLI
Models
Dev
Test
Dev
Test
Dev
Test
BiLSTM
69.00
69.90
51.01
51.17
58.55
67.30
ReZero
69.05
69.90
49.20
50.03
58.82
67.30
Werd2Vec
Transformer
69.00
69.90
50.23
50.70
57.75
67.30
StarSaber
69.05
69.96
51.22
50.85
57.69
67.30
ReZero
69.97
70.27
52.59
52.69
69.44
69.43
Transformer
70.32
70.24
50.78
51.64
71.14
67.30
Pretrained
StarSaber-1
70.85
70.68
53.25
53.41
69.67
69.43
StarSaber-2
70.18
70.22
52.28
52.55
69.44
69.43
From Table 2 we that both AFQMC and TNEWS are datasets of medium size and CMNLI is larger dataset compared to the other two. AFQMC is dataset of sentence similarity, rep- resented in a binary classification task. TNEWS dataset of text classification which has 15 classes in total. We don't 't use the keywords provided in order to make comparision with re- sults in Xu et al.] (2020). And CMNLI is Natural Language Inference dataset containing classes for each sample. Our results are shown in Table 3 Results achieved by different large pretrained models are shown Table 4 can be seen that none of those large-scale models can achieve astonishing performance. This is due to the construction approach used by CLUE. They use a specific pretrained baseline to select all samples misclassified. Details can be found in https: github. com/ CLUEbenchmaaeee StarSaber, shows that even pretraining on a small dataset with less time and computational power
CLUE Given the results from BiLSTM and
'If not stated, parameter numbers computed with the size of embedding matrices.

---

Under review as a conference paper at ICLR 2021
can help improve the performance. Pretraining also allows to use deeper and larger models. The reason why models with Word2Vec are small is that larger models without pretraining can achieve much worse performance, for they indicate greater search space and are harder to optimize.
Table 4: Accuracy for large-scale pretraining models in Xu et al. (2020)
AFQMC
TNEWS
CMNLI
Models
Dev
Test
Dev
Test
Dev
Test
Bert-base
74.16
73.70
56.09
56.58
79.47
79.69
BERT-wwm-ext-baar
73.74
74.07
56.77
56.86
80.92
80.42
ERNIE-base
74.88
73.83
58.24
58.33
80.37
80.29
RoBERTa- -a-lage
73.32
74.02
57.95
57.84
82.40
81.70
XLNet-mid
70.73
70.50
56.09
56.24
82.21
81.25
RoBERTa-wwm--ex
74.30
74.04
57.51
56,94
80.70
80.51
RoBERTa-wwm-large-en
74.92
76.55
58.32
58.61
83.20
82.12
For AFQMC, all models with Word2Vee in fact output zero for every sample(class labels are zero and one). This may be due to distribution imbalance in such dataset. Only those pretrained ones can classify small fraction samples into the positive class. Thus an improvement of 0.41% 1% is uneasy achieve. Another insteresting phenomenon appearing in CMNLI is that the gap between the development set and the test set is surprisingly large. For models with Word2Vec, the gap reaches up 9.61%. For TNEWS, the input sentence is only the title of passage. In this dataset, StarSaber-1 outperforms ReZero by 0.72% while StarSaber-2 differs from ReZero by only 0. 1.14.
It can also be seen from the results of AFQMC and TNEWS that stacking more layers in fact helps improve performance for StarSaber. On the dataset of CMNLI, the fact that StarSaber-1 doesn't outperform StarSaber-2 is probably because 12 layers are enough or even redundant for StarSaber. The same logic can be applied to the fact that StarSaber-1 doesn't 't outperform ReZero. With enough data and enough model complexity, ReZero and Transformer can perform fairly well. Compared to them, StarSaber more efficient. In all three datasets, it achieves almost the same results as ReZero with the same number of layers, revealing simple fact that many parts such as Attention and the FFN layer are not necessary within our framework. We can drop all these computationally expensive parts.
ReZero is improved version of Transformer in Vaswani al. (2017). It adds trainable weight in front of the Residual Connection and leads to faster convergence. But the better perfor- mance of ReZero here doesn't I mean it always outperforms Transformer with Layer Normalization, since samples in all these datasets are selected using Transformer-based models. Such a conclusion drawn from these selected data may hold generally.
5 ABLATION STUDIES
5.1 EFFECTIVENESS OF GATES
In an RNN, gates allow gradients to flow back to the more distant past. But in our model, there has been weighted residual connection to solve such problem, which means that gates' function of adjusting gradient flows is no more important. Here, we incorporate gates different way. Formulas can be found in the appendix. Model configurations and results are in Table 5 Numbers of layers, hidden sizes and input sizes are the same as StarSaber-2.
Table 5: Results of StarSaber-gate Accuracy Tormat:Test aco(Dev acc)
AFQMC
TNEWS
CMNLI
# Parameter
13.47M
15.09M
27.29M
Accuracy(%)
70.27(70.62)
53.56(53.04)
69.43(69.95)
We can compare the results here with the results in Table With the number of layers, the hidden size and the input size fixed, gates certainly help improve the performance. But when parameters
7

---

Under review as a conference paper at ICLR 2021
equally many, that is our implementation of StarSaber-1 which has twice the number of layers, gates don't t show any superiority. For simplicity and compactness, we drop all gates. We may also want to drop the gates in LSTM and replace them with weighted residual connection instead, which is simpler and more efficient. And this weight itself can also he parametrized by a simple non-linear function of hidden vectors.
5.2 COMPARISION OF TWo WAYS FOR POSITION ENCODING
We conduct experiments on our proposed methods for position encoding. We at first replace the two matrices representing distinct directions with one and add position embedding matrix made of cosines and sines to the input. The number of parameters is increased by doubling the number of layers. From 6 we can observe that after replacing our implementation of position encoding with the PE matrix in Transformer, performance is even worse than StarSaber-2 with less parameters, especially in CMNLL It means that the PE in Transformer is not consistent with StarSaber. We may give an intuitive explanation: Because of Transformer's s FFN layers, the PE matrix added to the input in fact be recognized in hidden layers. But StarSaber doesn't have Feed Forward Network, therefore cannot seperate such mixed information.
Table 6: Results of StarSaber implemented with PE. Accuracy format:Test aco acc)
AFQMC
TNEWS
CMNLI
# Parameter
11.89M
13.5TM
24.14M
Accuracy(%)
70.06(70.06)
52.05(52.13)
67.30(66.67)
5.3 EFFECTIVENESS OF DIRECT PATHS
Direct paths seem unecessary in StarSaber since we already have a residual connection. However, from the perspective of fixed point, if we drop these direct paths in each layer, the model will finally converge to the same fixed point for whatever inputs. In order to check whether these direct paths practical not, remove all of them and again increase the number of layers equalize the number of parameters. From the results of CMNLI in Table 7 we can clearly see the benefits they bring.
Table 7: Results of StarSaber without direct paths. Accuracy format: aco(Dev acc)
AFQMC 10.32M
TNEWS
CMNLI
Paaameter Accuracy(%)
11.94M
20.99M
69.75(70.16)
52.76(52.31)
67.30(69.00)
CONCLUSION
This paper proposes a framework to transform RNN-based models to attention-based ones. With the perspective to view attention as way construct word relation graph, transform the vanilla RNN to StarSaber, by defining a set of equations. Other variants of RNN can also be transformed in the same way, such as LSTM and GRU discussed above. In this way, we reduce the number of parameters in Transformer by dropping the FFN layer. Experiments on three datasets and the ablation study show the effectiveness our model and framework.
REFERENCES
Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Garrison W. Cottrell, and Julian McAuley. ReZero All You Need: Fast Convergence Large Depth. arXiv e-prints, art. 2XXi:22003.04887 March 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning Align and Translate. arXiv e-prints, arXiv:1409.0477 September 2014.
8

---

Under review as a conference paper at ICLR 2021
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Deep equilibrium models. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garett (eds.), Advances in Neural Information Processing Systems 32, pp. 690-701. Curran Associates, Inc., 2019. URL
http:/ /tst:://paper.... mipss cc wwwpppppp...aeeecccoocoooeeeeeee 88358 deeear/ee--dddepepppee dewp-equiluueeeeeeeeee for um-ummmdee s. pdd
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computaaioonll Linguistics (Volume 1: Long Papers), pp. 1657-1668, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/117-11522 URL https: / 7/www. wwwwwb.ww..... org/ httgg///ncccoccc///1/111155
Jason P. C. Chiu and Eric Nichols. Named entity recognition with bidirectional Istm-cnns. Trans. As- soc. Comput. Linguistics, 4:357-370, 2016. URL htttee///eaannclllroooo////aaaas wtt.://s///ooee//o.
Chp/tacl/article///ieee
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder- decoder for statistical machine translation. In Proceedings of the 2014 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar, Oc- tober 2014. Association for Computational Linguistics. doi: 10.3115/1//14-1119 URL https: / www. . ac web.org aetgg/athol..o www.thooogy/yiiiooo pgy/D14-111
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. -ra Attentive language models beyond fixed-length context. In Proceedings the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/PH/ 1/P19-128 URL https:/ // www http://wwwwwoom wwt.://wwwwe...og/aaoooooo////// 111
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal
transformers. In International Conference on Learning Representations, 2019. URL https:
//openzeview. http://weww..tttttoooooooolllaaa
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv e-prints, art. 18100088, October 2018
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. In Proceedings of the 2019 Conference of the North American Chapter of the Associ- ation for Computationnll Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1315-1325, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10. 10.18653//1/N19-11133 URL https::/p//......... /top:://wwww.. /wwww//wwwwoo ac web.. www.sathooaaoooooooooooooooooo ogy/ NI9 1133
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. arXiv e-prints, art. arXiv:1512.03385, December 2015.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In C. Cortes. N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Pro- cessing Systems 28, pp. 1693-1701. Curran Associates, Inc., 2015. URL CTRL/htp::// /ttp:://pppers. nips. (esaaperr5545--technng http://ppppssss--eacccaaaaaaaaaa htaerr--toerra-aaddd---ccceeeeee
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 98): 1735-1780, November 1997. ISSN 0899-7667. doi: 10. 10.1162/neco.1997.....o
URL
https://doi.orgg 1100..110 12/ neco. 1997. 9. 8 1735
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 908-9 11-9188 Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1..11665/v11PPP 1086. URL https: 7/www. c web. wew.booo///athhooooo//-------000

---

Under review as a conference paper at ICLR 2021
Nal Kalchbrenner and Phil Blunsom. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1700-170000 1709, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL hetps:
// //www. a web. org/ ho htthhollgyy/111111//1111111 1-7666
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv e-prints, December 2014.
art. arXiv:1412.660
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
net. foroforuuu forum?
id=#1eA7AEtvS
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv e-prinis, art. arXiv: 1607 507.06450, July 2016.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du Analogical reasoning on chi- nese morphological and semantic relations. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pp. 143. Association for Computational Linguistics, 2018. [ttt:
org/ wttp./////toooooo///1----222222-
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv e-prints, art. 197.:1997.11192. July 2019.
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, Lisbon, Portugal, September 2015. Association
for Computational Linguistics. doi: 10. 10.186533/1/DD15-1166 URL (tRLhtt:::/ www. beb.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Repre- sentations in Vector Space. arXiv e-prints, art. arXiv:1301.3781, January 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Represen- tations of Words and Phrases and their Compositionality. arXiv e-prints, art. arXiv:1110.4546, October 2013b.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, URL
June 2018. Association for Computational Linguistics. doi:
https: www acw.ebacllbb.. http:://www...c........orraattaa wttp:///aa.oooggg//..----------.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway Networks. arXiv e- prints, art. arXiv: May 2015.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 3104-3112. Curran Associates, Inc., 2014. URL /attp://papers.r www.sperrs.iiss..o cctpaper/ 5346 wecc---euueee--teeeee to wetoo-sseuueeeeeee liee-learniin wew.-n--a------el--e----a- new.ura..nttwwos..r... pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998 6008. Curran Associates, Inc., 2017. URL http:/ /appers nips. CC/ www.c//appee/cee
all yoal--yyo--- you-need.eeeo pdf
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral perltiiperspectvvee matching for natu- ral language sentences. In Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, IJCAI-17, pp. 4144 4150, 2017. doi: 10.24963ficcai221117777. URL
http://d.ooogg..../ www.oo/......./...... http://331iaaaa..1171..... 2017/579
10

---

Under review as a conference paper at ICLR 2021
Liang Xu, Xuanwei Zhang, Lu Li. Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, Yiming Cui, Cong Yu, Qianqian Dong, Yin Tian, Dian Yu, Bo Shi, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang. He Zhou, Shaoweihua Liu, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, and Zhenzhong Lan. CLUE: A Chinese Language Understanding Evaluation Benchmark. arXiv e-prints, art.
urXiv:2004.05986 April 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. XI.Net: Generalized Autoregressive Pretraining for Language Understanding. arXiv e-prints, art. arXiv:1906.08237,, June 2019.
Yue Zhang and Jie Yang. Chinese NER using lattice LSTM. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1554- 1564, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.
18653/v1/P18- 1144. URL https::tp///tc wttp://wwww. www. ac http::/wwwwwb...///aattooooooo/.
Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. Text classification
improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING 2016, the 26th International Conference Computational Linguistics: Technical Pa-
pers, pp. 3485 -395, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.
URL https::////...... www acl web www...oo//atttooogo/CCCCa /-1//
A FORMULAS TO INCORPORATE GATES
The formulas to incorporate gates mentioned in the ablation study are listed below:
2 = 1/, + a' tanh(r" n
Ar, + +i, (V'xn))*
4 E..u + Σ Q'..
i<n
12n
// (W A/. + vT(ln))
d = (W A', + Villsr)))
I'z
G, = softmar(s -1) = Σ gaj
(h4,) (
g.. = exp(
Vd
Note that this is also a demonstration how to transform a recurrence- based model into an attention based model in our framework.
to denotes the clement-wise product.
11


Reviews:
[{"review_id": "HqkD-RT1Mre", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a way of \"transforming\" RNNs into attention based models. The basic idea is to simply allow every cell to read the from all other cells instead of just the one immediately preceding.\n\nThe authors present the interesting of idea of defining the model as a system of equations to solve instead of as a set of update rules. This approach is interesting but not novel. It was presented in \"Deep Equilibrium Models\" (Bai et al. (2019)).\n\nThe authors of this paper however cite \"Deep Equilibrium Models\" but then seem to abandon the idea and instead decide to abandon root-finding for the system of equations and go back to a pretty ordinary MLP. They claim that the MLP \"approximates\" root-finding, but, as far as I can tell, the authors don't even try to check whether the \"approximation\" they created is actually approaching a solution for the system of equations or is doing something else altogether.\n\nThe authors additionally report some gains by having a more complex position encoding and by adding residual connections.\n\nIn addition the authors propose a quick pretraining strategy based on masked language modeling on the concatenation of training, dev and test sets from the target tasks. It is not surprising that this helps, since they don't do large scale pretraining, but it's also dangerously close to cheating to include the test set in pretraining, even if the test labels are not available.\n\nOverall the paper is promising and I would like to see more work on this, but right now there is a lot of space for improvement:\n* the authors should actually try to solve the system of equations that expresses their model, even if they use a different technique from Bai et al. (2019) -- otherwise just reframe your paper as a modification to the transformer architecture and don't mention the fixed-point approach.\n* the authors shouldn't pretrain on the test set.\n* experimental results would stronger if they were compared directly to results from other papers. If the authors don't want to run large-scale pretraining they might want to find a low resource setting with competitive results from other papers.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hohxffg_GCG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper aims to incorporate the attention mechanism into recurrent neural networks by using fixed point equations. In particular, the authors define a bidirectional RNN with attention by a fixed point equation and then transform it to a variant of the Transformer block. The proposed model StarSaber is shown to be more parameter efficient than the Transformer model and achieve competitive performance on three CLUE datasets.\n\n\nPros: \n+ This paper proposes a framework of transforming RNNs to Transformer-style models by incorporating the attention mechanism.\n+ Several ablation studies are conducted to support the design choices.\n\n\nCons: \n-\tThis paper aims to transform the RNN from the perspective of fixed point equations. But in Eq. (4), different weights are assigned to different layers, which means the model is actually iterating different functions at each timestep (layer). Thus, indeed it’s not a fixed point equation and the proposed model is actually a variant of the Transformer model with a modified structure, so the whole story does not make too much sense to me.\n-\tThe difference between the proposed StarSaber block and the Transformer block needs to be highlighted and discussed more clearly in the main content.\n-\tThe dataset and the pretraining task used for experiments are not commonly used and small-scale. It would be more convincing to include experiments on common and larger-scale benchmarks (e.g., SQuAD and GLUE).\n-\tThe paper is not very well-written and is a bit tedious, for example:\n    1)\tThe related work section is not well organized and includes some unnecessary contents like the discussions about pretraining and MLM. Some discussions about related works on implicit deep learning models could be included, e.g., Deep Equilibrium Models [1], Invertible Residual Networks [2], and Implicit Deep Learning [3].\n    2)\tEq. (2) – Eq. (4) are three similar sets of equations and rewriting could be avoided. \n    3)\tThe experiment section includes some experimental details that are not very important (e.g., the dataset statistics) which could be moved to the appendix.\n\n\nAdditional Comments/Questions: \n1.\tAs in the StarSaber model, the linear transformation weights for all the future positions and all the past positions are shared ($U^{right}$ and $U^{left}$ respectively). It seems that it incorporates weaker positional information than the positional embedding of the standard Transformer model since it could not directly distinguish between those future positions (and past positions). Therefore, I feel surprised about the ablation study regarding the positional encoding. Could you provide more details about this experiment and explain more about the result?\n2.\tIt would be great to include some comparison of the computational cost of the StarSaber model and the Transformer model.\n3.\tHow do you tune the learning rate for all the models?\n4.\tTheorem 3.1 is for real-valued function, it would be better to use a vectorized version as the functions considered in the paper are all vector-valued.\n5.\tTypo: dadtaset - > dataset\n\n\n\n[1] https://arxiv.org/abs/1909.01377\n[2] https://arxiv.org/abs/1811.00995\n[3] https://arxiv.org/abs/1908.06315\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "G0JD4iWp1U", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to make use RNN cell to replace the connection between continuous layers in Transformer. Although the proposed method makes use RNN cell to replace the heavy MLP layer after self-attention. I still think there's no significant difference between vanilla Transformer. The authors also propose to use position aware bi-directional attention mechanism. While it's hard to say whether it's better than multi-head self-attention in Transformer and the authors don't have an ablation study on this. Moreover, I would like to see some model comparisons on more popular datasets such as GLUE, SQuAD, etc.\n\nCons:\n1. It is hard to tell the benefits of the proposed method compared to vanilla Transformer. The motivation for integrating RNN cell is not clear enough for me. If Feed Forward Network is a big concern on speed, there should have some fair comparison on speed.\n2. If the authors care more about final performance. It should be evaluated on more popular datasets like GLUE/SQuAD. Pre-training a base model on Wikipedia should not take too much time. I would appreciate it if the authors can have this experiment.\n3. The paper needs some further proofread.\n\nMinor comments:\n1. improve the computation efficency. (second paragraph in Intro) --> efficiency\n2. third paragraph in page 2, in Transformer achieving still competive -->competitive\n3. first paragraph in page 3, retraining on a dadtaset --> dataset\n4. first paragraph in page 6,we concatenate the two input sentences with a seperation --> separation or special token?\n\n###update###\n\nAs no author response, I would like to keep my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1AwvMrwwrx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors state that they present a framework to transform Recurrent Neural Networks(RNNs) and their variants into self-attention-style models, with an approximation of Banach Fixed-point Theorem. Within this framework, they proposed StarSaber, by solving a set of equations obtained from RNN with Fixed-point Theorem and further approximate it with a Multi-layer Perceptron.\n\nMajor comments-----------------------------\n1. In equation (3), there is a weight matrix V, while it becomes V^l in equation (4). I am wondering which model is used in reality? Changing V to V^l is irrelevant to the fixed point scheme. I believe that based on the motivation, the weight matrix V should be shared.\n\n2. Many people believe that MLP in transformers is used for universal approximation. Dropping MLP in StarSaber makes me concerned about even approximation capability of the proposed model.\n\n3. I am confused about how the model shown in Figure 1 is related to Equation (5)? Where the masked attention and linear transformers come from based on your derivation in Equations (3)-(5)? \n\n4. All the experiments seem to have been done at quite small scales. I would like to see the experiments on some more challenging NLP tasks, e.g. the experiments in this github repository https://github.com/kimiyoung/transformer-xl, where the baseline code has been provided.\n\n5. I want the authors to report the computational and memory cost of the proposed model over the baselines.\n\n6. The paper claims that weighted residual connection (Bachlechner et al., 2020) which further alleviates the degradation problem. I want to see the evidence to support this claim.\n\n\n\nMinor comments------------------------------\n1. computation efficency -> computation efficiency.\n\n2. degration -> degradation.\n\n3. dadtaset -> dataset.\n\n4. What is the name of the proposed model? StarSaber or Starber?\n\n5. There are many other typos, which raises my concerns about how serious the authors are in writing this paper and conducting experiments. It makes me feel that the authors did the work in a very short time and wrote the paper very rushly and carelessly.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new computational block called "StarSaber" which is a self-attention based block derived from RNN fixed-point approximations.

All the 4 reviewers and the authors agree that the work is not ready for publication. While the motivation of the authors is interesting, some reviewers have raised concerns about the validity of the derivation. All the reviewers agree that authors need to clarify the difference between StarSaber and Transformer, compare with Transformer in large scale pre-training experiments, compare the compute speed, do a lot of ablations to validate the design choices.

I recommend the authors to incorporate all the reviewers' comments and make a stronger submission to a future conference!
Decision: Reject