PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
Authors: Chao-Hong Tan
Qian Chen
Wen Wang
Qinglin Zhang
Siqi Zheng
Zhen-Hua Ling
Venue: ICLR 2022 Poster (2022)
Keywords: Transformer
Efficient Transformers
Token Mixing
Pooling
Linear
Long Range Arena
Transfer Learning
BERT
GLUE

Abstract: Transformer-based models have achieved great success in various NLP, vision, and speech tasks. However, the core of Transformer, the self-attention mechanism, has a quadratic time and memory complexity with respect to the sequence length, which hinders applications of Transformer-based models to long sequences. Many approaches have been proposed to mitigate this problem, such as sparse attention mechanisms, low-rank matrix approximations and scalable kernels, and token mixing alternatives to self-attention. We propose a novel Pooling Network (PoNet) for token mixing in long sequences with linear complexity. We design multi-granularity pooling and pooling fusion to capture different levels of contextual information and combine their interactions with tokens. On the Long Range Arena benchmark, PoNet significantly outperforms Transformer and achieves competitive accuracy, while being only slightly slower than the fastest model, FNet, across all sequence lengths measured on GPUs. We also conduct systematic studies on the transfer learning capability of PoNet and observe that PoNet achieves 95.7 percent of the accuracy of BERT on the GLUE benchmark, outperforming FNet by 4.5 percent relative. Comprehensive ablation analysis demonstrates effectiveness of the designed multi-granularity pooling and pooling fusion for token mixing in long sequences and efficacy of the designed pre-training tasks for PoNet to learn transferable contextualized language representations.

Published as a conference paper at ICLR 2022
PONET: POOLING NETWORK FOR EFFICIENT TOKEN MIXING IN LONG SEQUENCES
Chao-Hong Tan'; Qian Chen", Wen Wang", Qinglin Zhang", Siqi Zheng², Zhen-Hua Ling' National Engineering Laboratory for Speech and Language Information Processing,
University of Science and Technology of China
Speech Lab, Alibaba Group
lcht an@main ustc. edu. en, zhLinglustc.edu edu. en [ttanging. eq, w.wang, qinglin. 2ql, zsql 7
com
ABSTRACT
Transformer- basse models have achieved great success in various NLP, vision, and speech tasks. However, the core of Transformer, the self-attention mechanism. has a quadratic time and memory complexity with respect to the sequence length, which hinders applications of Transformer-basee models to long sequences. Many approaches have been proposed to mitigate this problem, such as sparse attention mechanisms, low-rank matrix approximations and scalable kernels, and token mixing alternatives to self-attention. We propose novel Pooling Network (PoNet) for token mixing in long sequences with linear complexity. We design multi-granulle cratiirranlllrtty pooling and pooling fusion to capture different levels of contextual information and combine their interactions with tokens. On the Long Range Arena benchmark, PoNet significantly outperforms Transformer and achieves competitive accuracy, while being only slightly slower than the fastest model, FNet, across all sequence lengths measured on GPUs. We also conduct systematic studies on the transfer learning capability of PoNet and observe that PoNet achieves 95. 95.7% of the accuracy of BERT on the GLUE benchmark, outper- forming FNet by 4.5% relative. Comprehensive ablation analysis demonstrates effectiveness of the designed multi-granularity pooling and pooling fusion for token mixing in long sequences and efficacy of the designed pre-training tasks for PoNet to learn transferable contextualized language representations. Our implementation is available at / http:///t...b..ccocchcttttaaaaaa com//xcctaan/PoNNe
INTRODUCTION
Transformer (Vaswani et al. (017) has become the oatte-of-thh-art (SOTA) architecture for sequence modeling in a wide variety of fields, including natural language processing (NLP), computer vision, speech processing, applications genomies data, etc. The key reason for Transformer's success is its self-attention mechanism, which computes dot-product between input representations for each pair of positions in the full sequence. Proved be greatly effective in learning contextualized representations, Transformer becomes the backbone for dominant pre-trained language models (PLM) in NLP, such as BERT (Devlin et al. 2019) and RoBERTa (Liu al. 2019). These PLMs demonstrate strong transfer learning capabilities and have achieved SOTA widely on NLP tasks. However, self-attention has quadratic time and memory complexity to the input sequence length (Vaswani et al. (017)). which becomes the bottleneck for applying the vanilla Transformer to long sequence modeling tasks and for scaling up Transformer- based PLMs.
A broad spectrum of approaches have been proposed to address the efficiency problem of self- attention, as summarized in (Tay et al.) 2020). One major direction approximates the dense full self-attention using techniques such as introducing sparsity (Child al.] 2019; [Beltagy et al. 2020 Zaheer et al. 2020; Zhang et al.) 2021a] Yang et al. [021]] Zhang et al.) 20212),))) low-rank
approximations of the softmax attention matrix (Katharopoulos
al. [2200] Wang et al. 2020a
Zhu & Soricut 2021 Xiong et al.) 2021 Peng et al. 2 Choromanski et al. 2022] and locality
Work is done during the internship Speech Lab, Alibaba Group.

---

Published as a conference paper at ICLR 2022
sensitive hashing (Kitaey et al. 2020). These approximation approaches exploit observations that token interactions have strong locality. hence the importance and in turn the attention should decrease with the increase of the distance between query token and key token. Several of these works achieve O(N) theoretical complexity. However, these models often require selecting relatively large local regions for fine-grained attention in order approximate full self-attention, hence the scaling constants hidden by ((N) are often large and hinder significant improvements in speed and memory usage. is observed that performance of the approximation approaches usually inversely related to their speed (Beltagy et al., 2020 Zaheer et al. 2020p. Another major direction replaces the self-attention structure with more efficient structures, such as MLP-Mixer (Tolstikhin) et al. 2021]. FNet (Lee-Thorp et al.) 2021), AFT (Zhai et al. 2021), and Fastformer (Wu et al., 2021.. Despite significant accelerations gained by efficient transformers, they are rarely evaluated both on effectiveness of the inductive bias and the transfer learning capability, except BigBird, FNet, and Nyströmformet as we understand. BigBird- -Base (Zaheer et al. (020) outperforms BERT-Base on the GLUE benchmark (Wang et al. 2019) without significant speedup (slowdown for input length < 3 Tay et al. 2021). FNet-Base trains 80% faster than BERT on GPUs at 512 input lengths but only achieves 92% of BERT-Base's accuracy (Lee-Thorp et al. (021))
In this work, we propose a novel Pooling Network (PoNet). aiming to simultaneously advance long sequence modeling capacity and transfer learning capabilities while improving speed and memory efficiency. PoNet replaces the O(N2) complexity self-attention with a O(N) complexity multi-granularity pooling block. We design multi-gran granularity pooling and pooling fusion to model different levels of token interactions. Multi-granulann granularity pooling incorporates three types of pooling, from coarse-graine gransee fine-grained, in each sublayer. Global aggregation aggregates information of the entire sequence into single token. Segment max- poollig captures the paragraph or sentence level information. Local max- max-pooling captures the more important local information. These three poolings are fused to produce the output feature of the multi-granularic pooling block. Then through the residual connection, this output feature is further aggregated into each token.
The contributions of this paper are summarized as follows:
We propose a novel PoNet architecture as a drop-in replacement for self-attention in Transformer, achieving linear time and memory complexity. We propose multi culli-graaularityy pooling and pooling fusion to capture different levels of contextual information and comprehensively model token interactions. To the best of our knowledge, our work is the first to explore the full potential of the simple pooling mechanism for token mixing and modeling long-range dependencies.
Extensive evaluations show that PoNet achieves competitive performance on the Long Range Arena benchmark (Tay et al. 202T) and significantly outperforms Transformer by +2.28 absolute (+3.9% relative) accuracy. with efficiency up to 9 times faster and 10 times smaller than Transformer on GPU. Also, PoNet demonstrates competitive transfer learning capabilities, with Base reaching 95.7% of the accuracy of BERT-Base on the GLUE benchmark. Ablation analysis further proves effectiveness of designed multi-granularity pooling and pre-training tasks.
2 RELATED WORK
Efficientt Transformer Variants Among the models approximate full self-attention, Long- former (Beltagy et al. 2020) (O(N)) sparsifies the full self-attention into three attention patterns of sliding window, dilated sliding window, and global attention. BigBird (Zaheer et al. 2020 (O(N) combines global attention, local attention, and random attention. Poolingformer (Zhang et al.) 1021ah (O(N) uses a two-level attention schema, with the first level using a smaller sliding window to aggregate local information and the second level using larger window with pooling attention to reduce time and memory cost. Focal Transformer (Yang et al. 2021) (O(N)) uses both fine- ane-grained local interactions and coarse-grained global interactions to balance efficiency and effectiveness of capturing short- and long-range dependencies. Transformer-LS (Zhu et al. 2021) (O(N)) approximates the full attention by aggregating long-range attention via dynamic projections and short-term attention via segment-wise sliding window. H-Transformer- 11D (Zhu & Soricut 2021) (O(N) exploits a matrix structure similar Hierarchical Matrix. AdaMRA (Zhang et al. 20216) (O(N)) leverages a multi-resolution multi-head attention mechanism and kernel attention. Luna (Ma et al. 2021) (O(N) introduces an additional fixed length sequence served as query to
Nyströmformer was evaluated only on subsets of GLUE Xioog al.
2021
2

---

Published as a conference paper at ICLR 2022
attend to the original input while the output is served as key and value to attend the original input.
Apart from the sparse attention models, other approximation approaches explore locality sensitive
hashing and matrix upproximation methods. Reformer (Kitaev et al., 20220 O(Nlog replaces
self-attention with locality sensitive hashing. Performer (Choromanski et al. 2020: 2021) (O(N))
approximates softmax attention by leveraging random features. Linformer (Wang et al. 2020a)
approximates the self-attention matrix with a low-rank factorization. Nyströmformer (Xiong et al, 2021) (O(N) approximates the softmax attention with the Nyström method by sampling subset of columns and rows. However, these approximation methods have strengths on certain tasks and may cause accuracy degradation on many other tasks.
Our work is in another line of research on replacing self-attention with more efficient token mixing mechanisms. MI.P-Mixer (Tolstikhin et al. (021) (O(N2)) applies two separate linear transformations on the hidden state dimension and the sequence dimension. FNet (Lee-Thhorp al. 2021) (O(Nlog.N) replaces the self-attention sublayer with 2D-FFT mixing sublayer. AFT- local/conv (Zhi al., 2021) (O(sN), < N) first combines the key and value with set of learned position biases and then combines the query with this result via element-wise multiplication. Fastformer (Wu et al.
2021) (O(N) first models global context via additive attention then models interactions between global context and input representations through clement-wiss product. Shared Workspace (Goyal al. 2021) (O(N) proposes the idea of using shared bottleneck to tackle the problem of quadratic dependence in attention.
Pre-training Tasks It has been observed that both underlying model architecture and pre-training are crucial to performance of PLMs. BERT (Devlin et al., (019) with a Transformer encoder is pre- trained with masked language modeling (MLM) and next sentence prediction (NSP) tasks on large- scale unlabeled text corpora including the English Wikipedia and BooksCorpus. MLM predicts the masked token from context. NSP predicts whether a sentence pair is contiguous or not in the original source. Many approaches proposed to improve these two tasks and show that more challenging pre-training tasks may help PLMs learn better and more transferable language representations.
Whole word masking (WWM) (Devlin al. 2019 Cui et al. 2019) and SpanBERT (Joshi et al. 2020) outperform BERT on many tasks. WWM simultaneously masks all WordPiece tokens belonging to the same word and forces the model to recover complete whole word. SpanBERT randomly samples contiguous spans inside of individual tokens and augments MIM with a new task to predict the entire masked span. RoBERTa (Liu al. 2019) reports ineffectiveness of NSP and removes it from pre- pre-training. ALBERT (Lan et al.) 2020) replaces NSP with a sentence-order prediction (SOP) task predict whether two consecutive sentences are in the right order or not, for learning fine-grained inter-sentence coherence. StructBERT (Wang et al., 2020b) extends SOP to a new sentence structural objective (SSO) as a ternary classification on two sentences S2) to decide whether S precedes or follows S₂ or the two sentences are noncontiguous. More challenging tasks for learning inter-sentence relations and document/discourse structures (Iter et al., 2020 Lee et al. 2020 Ding et al. 2021) show promising performance improvements on PLMs.
3 MODEL
Our work is inspired by the External Attention (EA) approach proposed in (Guo et al. 2021 for visual tasks. An input sequence of tokens x = {I1....,. IN. are mapped to an embedding matrix denoted by HiE RNxdd
(hi...., h,,,, ). where N is the sequence length and d is the hidden dimension. EA uses two linear layers to implement external and shared memories, which facilitates learning correlations across all samples and hence serves strong regularization to and improves generalization of the attention mechanism with linear complexity. We simplify EA into multi-layer perceptron and softmax, and observe that by infusing the sequence-level information into each token through the denominator term Σx= e softmaz provides context modeling capabilities. However, softmax involves calculations of exponents, which is still slow. Consequently, we consider using pooling as an alternative to capture contextual information with significantly reduced complexity. We propose a Pooling Network (PoNet) as a drop-in replacement for the self-attention sublayer in Transformer, as shown in Figure I PoNet models different levels of contextual information through multi-granularity pooling (MP) block consisting of three components, namely, global aggregation (GA), segment max- pooliigo (SMP), and local max- pooling (LMP). These pooling features are then aggregated through pooling fusion.
3

---

Published as a conference paper at ICLR 2022
element-wise product
Ourout
Pooling Fusion (PF)
Output Proyection
Danse
Ado Norm
Second Stage
Max-pooling
Food Foward
(LMP)
NX
Attenton
Stage
SMP
GA
LMP
Segment Max-pooling (3MP)
Query
Key
Value
Postion Embedding
AVG
Input Embadding
f Input
Global Aggregation
Figure 1: The illustration of the PoNet model architecture. The right enlarged view shows multi- granularity pooling (GA, SMP, LMP) and pooling fusion (Section
3.1 MULTI-GRANULARITY POOLING
In order to capture contextual information different levels, we design multi--ranularrty pooling. First, different linear projections are applied on the input for poolings at different granularities:
H. HW. + b.,
(1)
where represents Qg: Kg. Vg for GA, 8 for SMP, for LMP, and for pooling fusion, in total six W.. W. Raxd and b. IRd parameters to be learned. The different H. then used for different poolings.
3.1.1 GLOBAL AGGREGATION
Global Aggregation (GA) module carefully designed aiming both capture the most important global information for each token and also to guarantee an overall linear computational complexity. We calculate the first stage value g for GA by averaging the sequence level:
ha E RRd
9
Note that g is only a rough representation of the sequence. Beltagy et al. (2020); Zaheer et al. (2020) introduced global attention mechanism, which adds a set of global tokens with randomly initialized values to always attend to the whole sequence. Inspired by these works, we perform cross-attention on the first stage value for GA. The first stage value is used to perform query on the input sequence compute the second stage value g' for GA, follows:
g' Attention H Hv,
(3)
The cross-attention on g enables each token to attend to the whole sequence and hence the resulting second stage value g' provides a more accurate sequence representation, compared to g. Note that since g is a single token with the length equaling 1, the computational complexity of attention in Eq. is O(N). Theoretically, using average average-pooling for generating the rough representation of the sequence, could keep more information for the next step cross-attention and generating the second stage value for GA, instead of discarding most information and only focusing salient features as the max- pooling function does. This hypothesis verified as we observe better model performance from using average-pood ppooling for generating g than max- maxx-ooling. In contrast, the outputs of SMP and LMP are directly used as final representations, hence we choose max-pooling for SMP and LMP, which is also empirically verified produce better model performance.
the most
4

---

Published as a conference paper at ICLR 2022
3.1.2 SEGMENT
The information loss from compressing long sequence into a single global token could be enormous and hence become detrimental to the sequence modeling capacity. We introduce an intermediate level between tokens and the global token by segmenting an input sequence into segments, exploring prior knowledge of structure in the data when available, and introduce Segment Max-pooling (SMP). The use of structural information for segmentation adjustable for different tasks, which is described in detail in Section each segment at each dimension j E [1.d]. as follows, where K denotes the number of segments:
E As explained earlier, max-pooling is performed on
= max(
(4)
(st ..y sg) E ERd,
(5)
= {8) [ E Rxxd
(6)
3.1.3 LOCAL MAX-POOLING
Many prior works (Zhu & Soricut 2021 1 Yang et al. 2021 demonstrate the crucial role of capturing local information for the sequence modeling capabilities. We introduce Local Max-pooling (LMP), designed as standard max-pooling over sliding windows, to capture contextual information from neighboring tokens for each token. Different from GA and SMP, the window for LMP is overlapped. Similar to GA, LMP is also applied at the sequence level and the left and right boundaries of the input sequence are padded to ensure that the output length equals the original input length. The LMP values L E TR N are computed. The size and stride of the sliding window are set to 3 and in all our experiments unless stated otherwise.
3.1.4 POOLING FUSION
First, we model the interactions between GA and each token by computing G, through the element- wise product between the second stage value of GA g' and each token, follows:
Gn g' Hon:
The reason for conducting Eq. 7 insstedd of directly using g' as the output is to avoid all tokens from sharing the same global token. Otherwise, it will cause the token representations converge the same global token in the subsequent addition fusion layer. This effect, in turn, will make the token representations become more homogeneous, and consequently degrade performance as sentence-pair classifications. The SMP token S is shared by all tokens in each segment. Hence, for the same rationale of mixing the global token with each token, the same operation is conducted to mix the SMP token with each token and to compute S' as:
tasks such
S = Sk(n) Hons
(8)
where k(n) denotes the segment index of the n-th token. The above three features are added up as the final output of our multi-granularin pooling block, as illustrated in Figure] I
P G + S' + L,
where Pis is used to replace the original self elff-attention output of Transformer. We compare PoNet to related models, including Fastformer (Wu et al. (02T). Luna (Ma et al. 2021). Longformer (Beltagy al. 2020) and BigBird (Zaheer et al. 2020). in detail in Appendix [
3.2 COMPLEXITY ANALYSIS
We only analyze the computational complexity of the proposed multi- granuiirrnullrrtty pooling block, since we only replace the self-attention sublayer in Transformer with this block and keep other modules in Transformer unchanged. The six HW. in Eq. where represents Qg Kg: Vg for GA, for SMP. 1 for LMP, and for pooling fusion, require 6. computations. GA requires 2Nd ops (Eq. 3 SMP and LMP have no matrix multiplication, and Pooling Fusion requires 2Nd ops (Eq. and Eq. g. Hence, the total number of multiplication ops is 6. + 4Nd. We further simplify computations. By switching the order of Eq. and Eq. into first performing the average pooling then the affine function, computation can be reduced from Na to d² Adding g' and S
5

---

Published as a conference paper at ICLR 2022
Model
ListOps(2K)
Text(4K)
Retrieval(4K))
Image(1K)
Pathfinder(IKK)
AVG.
Transformer(1
36.37
64.27
57.46
42.44
71.40
54.39
Longformer (1)
35,63
62.85
56.89
42.22
69.71
53,46
BigBird (1)
36.05
64.02
59.29
40.83
74.87
55.01
Performer (1)
18.01
65,40
53.82
42.77
77.05
51.41
Transformer(22
36.06
61.54
59.67
41.51
80.38
55.83
Linear (2)
33.75
53.35
58.95
41.04
83.69
54.16
FNet (2)
35.33
65.11
59.61
38.67
77.80
55.30
Transformer(3)
37.10
65.02
79.35
38.20
74.16
58.77
Performer(3)
18.80
63.81
78.62
37.07
53.63
Reformer(3)
19.05
64.88
78.64
43.29
69.36
55.04
Linformer(3)
37.25
55.91
79.37
37.84
67.60
55.59
Nyströmformer(3)
37.15
65.52
79.56
41.58
70.94
58.95
FNet
37.40
62.52
76.94
35.55
FAIL
53.10
PoNet (Ours)
37.80
69.82
80.35
46.88
70.39
61.05
Results on the Long Range Arena (LRA) benchmark (AVG: average accuracy across all tasks). Results with (1) are cited from (Tay et al.) (021), with (2) are from (Lee-Thorp al.) [202T), with (3) are from (Xiong et al.) 2021). We implement our PoNet and re re-implement FNet based on the Py Torcch codebase from (Xiong et al.) 2021) and use the same experimental configurations to ensure fair comparison. For each group, the best result for each task and AVG are bold-faced.
Table I:
first and then performing element-win product can reduce 2Nd Nd, compared conducting Eq. and Eq. @sseparttey After the simplifications, the total number of multiplication ops is (5N 1)d 3Nd. The multi-granularity pooling block hence has linear time and memory complexity with respect to the input length.
EXPERIMENTS
We first evaluate PoNet on the Long Range Arena (LRA) benchmark (Tay et al., (021) and compare PoNet the vanilla Transformer and series of efficient transformer variants on accuracy, training speed, and memory usage. Next, we study the transfer learning capability of PoNet in the commonly used paradigm pre-training followed by fine-tuning. We evaluate the fine-tuning performance on the GLUE benchmark (Wang et al., 2019) as well as set of long-text classification tasks. All baseline models and PoNet use the same "Base" model configuration BERT- Base (Devlin al., 2019). PoNet-Base has 124M parameters (see the first paragraph in Appendix A for more details). More experimental details, results and analyses, and comparisons other models are Appendices.
4.1 LONG-RANGE ARENA BENCHMARK
Comparison on Accuracy The LRA benchmark is designed to assess the general capabilities of capturing long-range dependencies. LRA consists of six tasks spanning structural reasoning (ListOps), similarity reasoning (Byte-level Text classification and document Retrieval, Image classification), and visual-spatial reasoning (Puthfider)). We the PyTorch codebase from (Xiong et al. |021 to implement FNet and our PoNet and evaluate on LRA after first replicating the results from Nyströmformer in (Xiong et al., 2021). We keep the same hyperparameter setting as used by (Xiong et al.) 202T) for all of our LRA evaluations and report results on five tasks in Table F All the baseline models are summarized in Section 2 and the Linear variant Thhor et al. 202T) denotes replacing the self-attention sublayer with two linear projections, one applied to the hidden dimension and one applied the sequence dimension. Note that due to different code implementations, results for same models could differ across groups in Table [ (see Appendix [ for details). important to point out that all models in the third group are implemented with the same codebase from (Xiong et al., 202T) and the same experimental configurations to ensure fair comparison within this group. As shown in Table [ compared to the first and second groups and the cited results in the third group marked with (3), PoNet achieves competitive performance LRA. PoNet outperforms the vanilla Transformer by +2.28 (61.05 over 58.77) and Nyströmformer by +2.10 on the average accuracy and consistently accomplishes better performance on all tasks
We exclude the Path-X task since all the evaluated models failed in the Path-X task, prohably due to its very long 16K sequence length (Tay al. 2027

---

Published as a conference paper at ICLR 2022
Seq. length
512
1024
2048
4096
8192
16384
Training Speed (steps/s)t
Transformer
45.1
19.4
6.3
1.8
OOM
OOM
Performer
39.4(0.9x)
25.0(1.3x)
14.3(2.3x)
7.8(4.3x)
4.0
2.0
Nyströmformer
39.1(0.9x)
30.3(1.6x)
20.0(3.2x)
11.5(6.4x)
6.1
3.1
FNet
83.4(1.8x)
61.3(3.1x)
38.1(6.0x)
21.4(11.9x)
11.0
5.4
PoNet (Ours)
50.4(1.1x)
x0.1(2.1x)
27.8(4.4x)
16.2(9.0x)
8.7
4.5
Peak Memory Usage (GB)
Transformer
2.5
6.7
23.8
OOM
OOM
Performer
1.5(1.1x)
2.1(0.8x)
3.1(0.5x)
5.4(0.2x)
9.8
18.7
Nyströmformer
1.2(0.8x)
1.5(0.6x)
1.9(0.3x)
2.8(0.1x)
4.5
8.2
FNet
1.1(0.8x)
1.2(0.5x)
1.4(0.2x)
1.7(0.1xx)
2.3
3.8
PoNet (Ours)
1.1(0.8x)
1.3(0.5x)
1.7(0.2x)
2x4(0.1x)
3.6
6.5
Comparison of GPU training speed (in steps/s, the higher the better) and peak memory consumption (in GB, the lower the better) various input sequence lengths the LRA text classification task (using the same hyper-parameter setting for this task as in (Xiong et al.) (02T), with speed-up and memory-saving multipliers relative Transformer shown parentheses. The best results are fold-faced with the second-best results underlined.
Table 2:
compared to Transformer, Performer, Reformer, Linformer, FNet, and Nyströmformer, except slightly weaker than Transformer on the Pathfinder task. It is reasonable to conclude that PoNet outperforms BigBird on LRA since the margin +2.28 from PoNet over Transformer in the third group significantly larger than the margin +0.62 from BigBird over Transformer in the first group. To the best of our knowledge, PoNet achieves very competitive accuracy on LRA against Transformer and recent efficient transformers, only lower than 63.09 from AdaMRA (Zhang al. 2021b 61.95
from Luna-256 (M et al. 2021 and 61.41 from H- HrTrassfommrr- ID (Zhu & Soricut 202
Comparison on Speed and Memory Consumption Table 2 compares the GPU training speed and peak memory consumption of PoNet to Transformer, Performer, Nyströmformer, and FNet on a single NVIDIA V100 chip, on input sequence lengths from 512 up to 16384. We observe that PoNet is the second fastest model and consumes the second smallest memory footprint in the group, consistently on all sequence lengths, much faster than Transformer, Performer, and Nyströmformer and lighter than them, and only slightly slower and heavier than FNet. Also, the speedup from PoNet over Transformer escalates on longer input sequence lengths.
4.2 TRANSFER LEARNING
The paradigm of pre-training followed by fine-tuning has been extensively applied and accomplished SOTA results in wide variety NLP tasks. Therefore, it is critical to evaluate the transferability of PoNet. We perform pre-training on PoNet and evaluate the fine-tuning performance on the GLUE benchmark and set of long-text classification benchmarks.
To facilitate fair comparison on the transfer learning ability, we pre-train BERT, FNet, and PoNet with the same MI.M (Devlin et al. 2019) and sentence structural objective (SSO) as in StructBERT (Wang et al. 2020b) on the English Wikitext-103 (100M words) and BooksCorpus (800M words) d (in total 5GB data). The total pre-training loss L = LMLM + Lsso- All three models are base-uncased and pre-trained using the same configuration (Appendix A.2 Figure 2 illustrates validation accuracy of the same MLM and SSO tasks from BERT, FNet, and PoNet pre-training. MLM accuracy from PoNet is only slightly worse than that from BERT while the gap on SSO accuracy between them is bit larger. PoNet achieves significantly better MLM and SSO accuracy than FNet, consistent with its better sequence modeling capability shown LRA.
AdaMRA /hhng et al. 2021b) also re-implemented BigBird with the same codehase from Xiong et al. 2021 1) and reported AVG 59.43 from BigBird, which is worse than PoNet.
Luna-256 Ma al. 2021 outperforms their reimplemented Transfomer by +2.71, while PoNet achieves
+2.28 gain over Transformer implemented based on the same codebase and using the same configurations.
httpss//huggingfacc..ouaasssss
7

---

Published as a conference paper at ICLR 2022
0.6
0.80
0.5
0.75
0.4
0.83
950
0.63
0.50
0.45
5k
53k
105K
155c
2054
235k
325k
355K
5k
59k
105k
1538
203k
25.5
305k
359c
Steps
Steps
(a) MLM Accuracy
(b) SSO Accuracy
Figure 2: MLM and SSO validation accuracy against the numbers of training steps from BERT- Base, FNet-Bart Base, and PoNet-Base All models are uncased.
Model
MNLKm//m)
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
AVG.
BERT-Base
81.35/80,98
88.89
88.01
91.17
47.66
87.83
86.66
69.31
80.21
FNet-Base
13/73/33.66
85.75
80.50
88.65
40.61
80.62
80.84
$7.40
73.46
PoNet-Base (Ours)
76.99/77.21
87.55
84.33
89.22
45.36
84.57
81.76
64.26
76.80
Table GLUE Validation results from PoNet, BERT, and FNct. All models are uncased and pre- trained with the same configurations (Appendix A.2) with 340K steps. We report the best GLUE results for each model from multiple pyyee--prammetr configurations (Appendix A..)). We report the mean of accuracy and F1 for QQP and MRPC, matthews correlations for CoLA, spearman correlations for STS-B, and accuracy for other tasks. MNLI(m/mm) means match/mismatch splits.
Results on GLUE The GLUE benchmark covers a diverse range of challenging natural language understanding tasks and is widely adopted for evaluating transfer learning models. The tasks can be split into two groups, single-sentence tasks including CoLAA and SST-2, and sentence-pais -pir tasks including MRPC, QQP, STS-B, MNLI, QNLI, and RTE The special token "[SEP]" used as the segment separator. For fine-tuning PoNet on GLUE, inputs to single-sentence tasks include three segments as "[CLS]" Sentence-1 "[SEP]": whereas inputs to sentence- -aincce--rrrrra tasks include five segments "[CLS]" Sentence-1 "[SEP]" Sentence-2 "[SEP]" These segments are used for computing SMP (Section 3) Table ssows the results for the best base learning rate (no early stopping) on the GLUE Validation split (see Appendix A.3 for more details), providing fair comparison since all three models are pre-trained and fine-tuned with the same pre-training data (5GB data//takkshyypr- parameters with 340K steps. Table 3 shows that PoNet achieves 76.80 AVG score, reaching 95.7% % of the accuracy of BERT on GLUE (80.21) and outperforming FNet by 4.5% relatively. These performance comparisons are consistent with the pre-training accuracies shown in Figure 2 The results also prove that PoNet is equally competitive in both single-sentence and sentence- tasks. When pre-trained on the same 16GB data used for the official BERT pre-training with MSO tasks up to M steps, PoNet also achieves 95.9% % of BERT's 8 accuracy on GLUE (see Appendix B.2
Results on Long-text Classification We also evaluate the fine-tuning performance of the pre- trained PoNet on four long-text classification datasets. including Hyperpartisan news detec- tion (HND) (Kiesel et al. 2 IMDb (Maas et al. 2011). Yelp-5 (Zhang et al. 2015 and Arxiv-11 11 (He et al. (2019). As can be seen from Table 4 PoNet-Base outperforms BERT-Base on HND (+8.2 Fj) and Arxiv (+0.75 F2) and reaches 99% of BERT SorT-BBae'ss F, on IMDb and Yelp-5.
ABLATION ANALYSIS
We conduct ablation analysis on contributions from multi-granularity pooling and pre-training tasks. By applying leave-one-out PoNet components, we create variants PoNet w/o Second Stage
FFollowing (Devlin ct al. 2019 Lec- Lec-Thorp et al. 2021 we exclude WNLI.
We use the train/validation/ess division provided
Beltagy al. 2020
8

---

Published as a conference paper at ICLR 2022
Model
IIND(F1)
IMDb(F//Ace)
Yelp-S(F1)
Arxiv(F.)
#Example (#Classes)
500 (2)
25000 (2)
650000 (5)
30043 (11)
#Wordpieccs avg. (95thpetl.)
734 (1,974)
312 (805)
179 (498)
16,210 (32,247)
RoBERTa-Base (aaheer et al. 2020
87.8
95.3/95.0
71.75
87.42
Longformer (Beliagy al.
2020
94.8
95.7/-
BigBird (Zaheer et al 2020
92.2
195.2
72.16
92.31
BERT-Basc
88.0
94.1/94.1
69.59
85,36
FNet-Base
86.3
90.4/90.5
65.49
79.90
PoNet-Base (Ours)
96.2
93.0/93.0
69.13
86.11
Fine-tuning results (in F1 and Acc) on long-text classification datasets. For the third group of results, we use the official checkpoints of BERT-Base and FNet-Base (see Appendix A.4
Table 4:
Pre-trained tasks
Downstream tasks
Model
MLM
SST
CoLA
STS-B
PoNet(340K steps)
59.44
80.75
45.36
84.57
PoNet w/o SS-GA
59.33
76.92
46.18
78.38
PoNet w/o GA
56.64
74.36
49.51
64.61
PoNet w/o SMP
56.96
78.41
44.21
84.89
PoNet w/o LMP
56.53
80.27
41.44
85,55
PoNet w/o (SMP&LMP)
43.61
76.72
11.36
84.93
PoNet using AMN
62.53
79.28
50.91
75.32
PoNet using Lom
63.11
51.26
69.83
Table 5: Results of ablation study as accuracy for pre-training MLM and SST (Sentence Structure Task) tasks, matthews correlations for CoLAA, and spearman correlations for STS-B. SST denotes NSP when using LaN and the SSO task otherwise. All pre-training experiments run 340K steps.
GA (SS-GA), w/o GA, w/o SMP, w/o LMP, and w/o (SMP&LMP). We also pre-train PoNet with two weaker losses, LMN LMLM + LNSP and Lsom = LMLM. where LNsP is the NSP loss in BERT. We report pre-training task validation accuracy and GLUE validation scores on the single- sentence CoLA and sentence-pair STS-B tasks in Table 5 Since [CLS] from PoNet w/o GA cannot capture information of the whole sequence, as using (CLS) for classification fails on SST (Sentence Structure Task), CoLA, and STS-B, we use max- max--oooing of sequence for classification.
Removing GA (PoNet w/o SS-GA, PoNet w/o GA) significantly degrades accuracy on SST and STS-B, B, showing that sentence- pair tasks heavily rely the global information. MLM accuracy degrades only slightly from PoNet w/o SS-GA but more significantly from PoNet w/o GA, indicating that MLM also depends on the global information, but when the rough global information is available (PoNet w/o SS-GA), SMP can compensate sufficiently and PoNet w/o SS-GA improves CoLA. Removing GA enhances SMP and LMP learning thus improves CoLA since CoLA relies on SMP and LMP much more than GA. Different from the conclusions on BERT (Devlin et al. 2019 Liu et al. 2019), we find fine-tuning performance of PoNet on sentence-pan tasks highly relies on SST pre-trate tra-nrannng tasks. Weakening SST loss (LMN. LOM) weakens GA representation learning while enhancing SMP and LMP learning, causing a significant degradation in STS-B accuracy but significant gain in CoLAA. Similarly, removing SMP or LMP enhances GA representation learning and hence improves STS-B accuracy while degrading CoLA accuracy. PoNet w/o SMP&LMP (i.c. GA only) shows a drastic degradation MLM and CoLA accuracy (45.36 to 11.36). These results confirm that all three poolings are important for the modeling capabilities of PoNet.
6 CONCLUSION
We propose a novel Pooling Network (PoNet) to replace self-attention with a multi-granullrityy pooling block, which captures different levels of contextual information and combines them for a comprehensive modeling token interactions. Extensive evaluations demonstrate that PoNet achieves both competitive long-range dependency modeling capacity and strong transfer learning capabilities, with linear time and memory complexity. Future work includes further optimization of model structure well applying PoNet to broader range of tasks including generation tasks.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All data used in the experiments in this paper are open source. Readers can refer to the original papers for details of the datasets, which are cited in our paper. Information on data access can be found in Appendix [ Experimental details are also described in Appendix A. Our implementation
available at wttpt::://// httpp:://ggttub.../ http:///hib...cm////ctaa/////aaa
ACKNOWLEDGEMENTS
This work was supported by Alibaba Group through Alibaba Research Intern Program.
REFERENCES
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long- documeet transformer. CoRR, abs/2004.05150, 2020. htthttps: /https://arxxi.....m wtwzxiv.org/ab//000000 05150
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python. O' O'Reilly, 2009, ISBN
URL http:://ww..rrrllllyy. wttpeellllye/cttlllooooo
9780596516499/Ineee
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/ 1904. 10509, 2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Jared Davis, Tamás Sarlós, David Belanger, Lucy J. Colwell, and Adrian Weller. Masked language modeling for proteins via linearly scalable long-context transformers. CoRR, abs/2006.03555, 2020. URL
httts://arxiv. http://xxxiv..r//aaaaaaaaaea/aaa /00e 03555
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021, URL /tLL:ttt////eeeeereiew..... net/
Eorum?
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. Pre-training with whole word masking for chinese BERT. CoRR, abs/1906.08101, 2019. URL
http://arxii>> httt:://arxiv.or/// wwwg/as//////oooooo/////oo//eaaa abs/ 1906 08101
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. MAACL-HLT 2019, Minneupolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: (0.18653/v1/119-444 1423. URL 14223
ht https://dol www.ooggg//00...
Siyu Ding. Junyuan Shang. Shuohuan Wang. Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Emic- doc: retrospective long-document modeling transformer. Chengqing Zong Fei Xia. Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp- 2914-2927, Association for Computational Linguistics. 2021. doi: 10. 10.118/33v1/222 1tt.1/2.......longg....... 227. URL wttph:ttp///mm /ttp::////oooooo /rrg/1000 10600.18663100 / vl. / acl Long. 227
Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. CoRR, abs/2103.01197, 2021. URL https://a
http::://rraiv.orr////////11111/ wttps/o.rrr//////....////// 01199 01197
10

---

Published as a conference paper at ICLR 2022
Meng- -ao Guo, Zheng-Ning Liu, Tai-Jiang Mu, and Shi-Min Hu. Beyond self-attention: External attention using two linear layers for visual tasks. CoRR, fbss/2105.0335, 2021. URL https:
//rxivv.rr// abs/ 2005. 02358
Jun He, Liqun Wang. Liu Liu, Jiao Feng. and Hao Wu. Long document classification from local word glimpses via recurrent attention learning. IEEE Access, 7:40707 2019. doi: 10. 109/ ACCESS,2019
URL /tt.///do.i/// org 111./...1119/1..55 201 2907992
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. Pretraining with contrastive sentence objectives improves discourse performance of language models. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), ACI., pp. 4859 4870, 2020. doi: 10.18653/v1/200 acl mmai.nann33
acl-main.439, URL htttps:///oi....
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:64-77, 2020. URL https:: wrwpaecoooorrooooooo wwt.ooo/g/oooooooooooooooo phpttac1/
article/view/165
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5156 5156-5165, PMLR, 2020. URL httt::/procceel: ngs mlr. wwr.ppees/vlll http//v1119kkatt wtw.ataaaaopuullaaaaaaaaaaaaaaaa 20 html
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David P. A. Corney, Benno Stein, and Martin Potthast. Semeval-2019 task Hyperpartisan news detection. In Jonathan May. Ekaterina Shutova, Aurélie Herbelot, Xiaodan Zhu, Marianna Apidianaki, and Saif M. Mohammad (eds.), Proceedings of the 1 13th International Workshop on Semantic Evaluation, SemEval @ NAACL-HLT 2019, Minneapolis, MN, USA, June 6-7, 2019, pp. 829- 839. Association for Computational Linguistics, 2019. doi: 10.18653/11/119224455 URL https::/ddi.orgg// http://///....1.////////////111/
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net. forum?
VK
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for sulf-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. WRLLhttps:/ http::s///ppeeevviie net/ / forum?ine MleA7ABEVV.
Haejun Lee, Drew A. Hudson, Kangwook Lee, and Christopher D. Manning. SLM: learning discourse language representation with sentence unshuffling. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 1551-1562. Association for Computational Linguistics, 2020. doi: 10.18653/w120200.o 10002000mnlppmmi.... 120. URL https:://do www.oorg/..... 11w.:/...115111111 2020 wmw.ppmmain.......mmm.......mmmm 120
James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. FNet: mixing tokens with fourier transforms. CoRR, abs/2105.03824, 2021. URL
03824
Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Yacine Jernite, Abhishek Thakur, Lewis Tunstall, Suraj Patil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Teven Le Scao, Victor Sanh, Kevin Canwen Xu, Nicolas Patry, Angelina McMillan-Major Philipp Schmid, Sylvain Gugger, Steven Liu, Nathan Raw, Sylvain Lesage. Théo Matussière, Lysandre Debut, Stas Bekman, and Clément Delangue. huggingfacce/datastss:: 1.12.1, September 2021. URL actps: :/tps:://oi... httpa////.....g/11....o org/10. 5281/
zenodoo 5510481
11

---

Published as a conference paper at ICLR 2022
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.116992 2019.
11692
Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zeut Luna: Linear unified nested attention. In Advances in Neurul Information Processing Systems. Curran Associates, Inc., 2021.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeling of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
7/www. www.waclwwb.oom hwwp://rgggttttteeoe 2tts///anoooooyy111-11111
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A. Smith, and Lingpeng Kong, Random feature attention. In 9th International Conference Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:
//openzeviww. /est/fooumm -tl-dd--tttTt IKI
Yi Tay, Mostafa Dehghani. Dara Bahri, and Donald Metzler. Efficient Transformers: A survey. CoRR, 0bs/2009.067333 2020. paattps://axxx. org/ abs 2009. 06732
Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao. Liu Yang, Sebastian Ruder, and Donald Metzler. Long Range Arena : A benchmark for efficient transformers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htthhtttt:://peerevieee net./ forum? Forum?id=qu La=--dde-gyeerer-
Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MIP-Mixer: an all-mlp architecture for vision. CoRR, abs/2105.01601,
2021. https:/s:
org/ abs//ab////// /2005.10..2 01601
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL httpps:s/pprccceengss.... neurips 3ff5e223344dde BFI2e243Bfeee11do0scccaaaa-a---2
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https::/openrer/// httpr//wwwweweeeeeeeeeeeeeeeeeee wwt. wwwt/eert//ieiooc FJ4km2R5E7
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. CoRR, 22//200660476.. 2020a. URL httpp:://axiv..rrg///// 2006.04765
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. StructBERT: incorporating language structures into pre-training for deep language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020b. URL bttps::/openrevieet net forum?
ld BJ-B@g141SFHH
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin I Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural
12

---

Published as a conference paper at ICLR 2022
language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguisties. URL https: /www http://wwwwwwee...oo//ottttoooo.
emm 1 1--1lp-dd..... 6
Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Fastformer: Additive attention can be all you need. CoRR, abs/2108.09084. 2021. URL /tLLLttpee//aaxxv.oo////////////
[9084]
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystrômformer: A nyström-based algorithm for approximating self-attention. In Thirty-Fifth AAAI Conference on Artificial Intelligence. AAAI 2021. Thirty-Thize Conference on Innovative Applications of Artificial Intelligence. IAAI 2021. The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021. 14138-14148, AAAI Press, 2021. URL httph:tttttt... /ttpps:://ss... http:////sssaaa.......... www.ai.orgaaao index.ndxx.ee/ee/ pphp/ AAAT AAAT/article/view/111
Jianwei Yang, Chunyuan Li, Pengchuan Zhang. Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. CoRR, (obs/2107.0064))
2021. htttpt:ss////xxtt org/ 02000 00641
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big Bird: transformers for longer sequences. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-, 2020, virtual, 2020. URL
neurips. cc/ www.aaattbbtttttttt.o.m.. httll
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh M. Susskind. An attention free transformer. CoRR, abs/2105. 14103, 2021. URL https:
7/arxiv. wwwxivvorr//ao org. abs 220/21155.. 14103
Hang Zhang, Yeyun Gong, Yelong Shen, Weisheng Li, Jiancheng Lv, Nan Duan, and Weizhu Chen. Poolingformer: Long document modeling with pooling attention. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, , 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning PMLR, 2021a. URL httt::/pproceednngg/// www. press//
Research, pp.
v139/ 2hagg/haag2/1 11.hmml
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 649-657, 2015. URL httpttt://pc/ceedd//// ngs newrips....... cct. .act.
caper/20115/ash/2222 (tt.aa/e...........7.e3333eeeee3
Yao Zhang, Yunpu Ma, Thomas Seidl, and Volker Tresp. Adaptive multi-resolutiionn attention with linear complexity. CoRR, abs/2108.0496. 2021b.
04962
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in Neural Information Processing Systems, 34, 2021.
Zhenhai Zhu and Radu Soricut. H-Transformer- 1D: fast one-dimensional hierarchical attention for sequences. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 3801-3815 Association for Computational Linguisties, 2021. doi: 10.18653/v 000/S33/1122211 laa.lllloooo Iol..o....29
URL https: /000://011/00/101118////
2021 2021.acl-a -la.ccllooo..00
13

---

Published as a conference paper at ICLR 2022
Appendices
A EXPERIMENT DETAILS
Model Size For all experiments for PoNet in this paper, parameters WK,, bx, and WV, by, in Equation ] are shared reduce the calculations and we observe no performance degradation. After this sharing of W WK, bx, and Wv, by, in Equation 1 PoNet-Base has 124M parameters. This is comparable to the 110M parameters for BERT-Bas Base but still still a bit larger in the number parameters. Our ideas for further reducing the number of parameters are summarized in Appendix F.1
A.1 LONG-RANGE ARENA BENCHMARK EXPERIMENTAL DETAILS
Implementations and Hyperparameters We use the Pytorch codebase from Xiong et al. [2021 to implement our PoNet and re-implement FNet and conduct all LRA evaluations to facilitate fair comparison. We use exactly the same experimental configurations provided by Xiong et al. 2021 Note that due to different code implementations, as shown in Table the results from our re- re-implemented FNet achieve 53.10 average score, lower than 55.30 reported in the original FNet paper -Lee-Thorp et al., 2021) where the original FNet is implemented implementation also causes
JAX/Flax. The Pytorch difference in results for other models compared to their LRA results reported in the original LRA paper (Ta et al. 2021). which are implemented in JAX/Flax.
For each task, the input sequence is truncated evenly into K segments (as the K in Equation 4 We find that 32 segments for the Image task, 64 segments for the ListOps and Retrieval tasks, 2048 segments for the Text task produce the best results. However, PoNet fails on the Pathfinder task under all three segment configurations. Hence we remain computing SMP on the whole sequence (i.e., only segment) for the Pathfinder task.
Additional LRA Results The standard deviations from three runs of our PoNet model LRA for each task are: ListOps (3.98e-3), Text (4.09e-3), Retrieval (5.89e-3), Image (2.50e-3), and PathFinder (3.2e-3). The standard deviations from three runs of our FNet implementation LRA for each task are: ListOps (4.66e-3) Text (3.25e-3), Retrieval (2.99e-3), Image (1.57e-2), and all the three runs on the Pathfinder task failed.
Note that we use the PyTorch codebase from Xiong et al. 2021 to implement FNet and our reported FNet results on LRA in Table [ and Table This implementation adds a Linear layer after the attention, as done for the BERT series but not in the original FNet paper Lee-Thorp et al. (202T) nor in its official Google research code and the HuggingFace implementation. We experiment with removing this Linear layer in FNet and notice significant performance degradation LRA from FNet, with the AVG score dropping from 53.10
based on our FNet implementation.
in Table to to 50.65.
Speed and Memory Comparison The training speed and peak memory consumption compar- isons are conducted on the LRA classification task on single NVIDIA Tesla V100 GPU, The input sequence lengths are set from 512 to 16384. Note that since the original LRA githut 10 also provides the full length datasets, hence we could truncate the sequences in the LRA text classincation task into 8K and 16K lengths. The dyper-paramedt the hidden size is set 64, the intermediate size to 2, the number of layers is set to 2, and the batch size is set to 32.
are the same as in (Xiong et al.) (2021), that is, 128, the number of attention heads is set
A.2 PRE-TRAINING DETAILS
For feasible experiment tum-arromm with our computation resources, we first use the English Wikitext-103 (100M words) and the BooksCorpus (800M words datasets for pre-training,
http:///giiibboooo/////ee////to/ rangee--rnnn "https//huggingffac.. coldattasets/vikitext
12 /htpss//uuginggface http:///wwunnngaaaa.ooaaaa/cc/nn
14

---

Published as a conference paper at ICLR 2022
in total 5GB data size. For PoNet, natural paragraph segmentations in the datasets, which marked by ""n", are treated as segments for the SMP computation. For the MLM task, the masking probability is set to 15%. 80% of the masked positions replaced by "[MASK]", 10% are replaced by randomly sampled words, and the remaining 10% are unchanged. For the SSO task, a long sequence containing several paragraphs is truncated into two subsequences random positions, with 1/3 probability of replacing one of the subsequences with another randomly selected subsequence, 1/3 probability swapping the two subsequences, and 1/3 probability unchanged. These three cases are assigned three different labels for the ternary classification. All input sequences are truncated to a maximum sequence length of 512, and to accommodate sentences of different lengths, some input sequences are truncated shorter with probability of 0.1. The datasets were duped 5 times to alleviate overfitting of SSO tasks, which were also applied by Devlin al. (2019 13 Since the selection of masked positions and sentence pairs are done according to probabilities, we obtain 5 times more training sample pairs. The pre- pre-training implementation based on the Pytorch codebase from Wolf' et al. (2020), with the hyper-parameee shown in Table 6 Each pre-training experiment run on NVIDIA Tesla V100 GPUs and takes about 9 days.
Pre-training
GLUE
Long-text Tasks
Max Steps
750K
Max Epochs
4 [3e-5, 5e-5, le-4, 3e-4)
10 or 2 (3e-5, 5e-5}
Learning Rate
le-4
Batch Size
192
{128, 64, 32, 16, 8}
32
Warm-up Steps
5K
0 128 Linear
0
Sequence Length
512
4096
Learning Rate Decay
Adam
le-8
Adam (31, .22)
(0.9, 0.999)
Clip Norm
1.0
Dropout
0.1
The value is used for the high-resource task Yelp-5, and 10 for the other tasks.
Table 6: Detailed hyperparameter settings for the pre-training and fine-tuning experiments. For rows with a single hyperparameter value, the value is used across pre-training and fine-tuning on GLUE and long-text classification tasks.
A.3 FINE-TUNING ON THE GLUE BENCHMARK
The GLUE datasets can be accessed from Lhoest al. (2021). The special token "[SEP]" is used as the segment separator. Inputs to single-sentence tasks are processed as "[CLS] S [SEP]"; whereas inputs to sentence-pait tasks as "[CLS] S1 [SEP] S2 [SEP]". We implement all fine- tuning code based the Pytorch codebase from Wolf et al. (2020). We run 20 sets of hyper- parameter configurations based on Table (6) for fine-tuning BERT, FNet, and our PoNet and report the best GLUE results in Table 3 Note that Table 3 provides fair comparison between BERT, FNet, and PoNet as all three models are pre-trained with the same pre- pre-training data of English Wikitext- 103 (100M words) and BooksCorpus (800M words) dataset [5| the same pre- mre-training tasks of MLM+ the same pre-training configurations (Appendix A.2) (all trained with 340K steps), and fine-tuned GLUE with the same configurations.
"hatps//ggihuu..comm// wtw.:/oooooeeeeeeeeeeeiooooe 14 "htrs://huggingfacc. 15 https:/hugginggacc.
http://gggggggggcoooo//as//////a coldatascts/glue
http:///uuuggnngaaacccttttttssss co/datasets
15

---

Published as a conference paper at ICLR 2022
Model
MNLKm//m)
QQP
QNLI
SST-2
CoLA
STS-B
MRPC
RTE
AVG.
BERT-Base(1)
84/81
93
83
83.3
Linear-Base(1)
74/75
94
67
69
77.0
FNet-Base(1)
72/73
95
63
76.7
BERT-Basc(2)
85/85
89.77
91.78
92.66
58.88
89.28
89.31
70.76
83.52
FNct-Basc(2)
75/76
86.72
83,23
90.13
35.37
81.43
80.34
59.92
74.23
PoNet-Base(Ounss(2)
78/78
87.76
85.17
89.00
47.24
85.86
83,39
63.53
77.54
BERT-Base(3)
83/83
89.48
90.65
91.74
51.19
89,28
88.73
67.51
81.63
FNet-Base(3)
75/76
86.17
82.52
88.42
40.57
83.64
80,90
61.73
74.99
3oNet-Base(Ours((3 79/78
87.92
86.31
89.79
45.18
87.17
84.27
66.43
78.29
Table 7: Extra GLUE Validation results. BERT-Base and PoNet- PoNet-Base are uncased whereas FNet- Base is cased (since the official FNet checkpoint is cased). We report the mean of accuracy and Fl scores for QQP and MRPC, matthews correlations for CoLA, spearman correlations for STS-B, and accuracy scores for other tasks. The MNLI(m/mm) means the match/mismatch splits. Results with (1) are from (Lee-Thorp et al.] 2021). Results with (2) and (3) are the best results from searching 20 sets of hyper-parameter configurations based Table 6 for fine-tuning the pre-trained models. For BERT-Base(2) and (2)))Baae(22, we use the official checkpoints provided by authors while for PoNet-Base(2), we pre-train the PoNet model on 5GB data (Wikitext-10) and BooksCorpus), For a fair comparison on model capacity by pre-training with more data, BERT-Base()) FNet-Base(3), and PoNet-Base(3) are all pre-trained on the same 16GB Wikipedia and BooksCorpus data used for pre- vre-training official BERT checkpoints, trained with MLM+SSO tasks for 1M steps.
A.4 FINE-TUNING ON THE LONG-TEXT CLASSIFICATION TASKS
The HND dataset can be acquired followed the guide in (Beltagy et al. 2020 16 The IMDb datase T7
and Yelp-5 dataser are from Lhoest et al. 2022 The Arxiv-11 dataset from He et (2019
The max sequence length for all long-text classification experiments is 4096. Since there is no natural paragraph segmentation for these data, we use the NLTK toolkit (Bird et al. (2009) to segment the input into sentences for SMP computation. Note that our PoNet model was pre-trained with max sequence length 512, to able to fine-tune on 4096 input lengths, following [Blltgy et al. (2020), add extra position embeddings initialized by copying the pre-trained 512 position embeddings recurrently. We implement all the tne-tunnng code based on the Pytorch codebase from Wolf et al. (0200) with the hypee--asameters shown in Table 6
For BERT-Base, we fine-tune using the HuggingFace BERT- trained on Wikipedia and BooksCorpus. in total 16GB data size). We fine-tune FNet- Base (Lee- Thorp et al. 2021) (pre-trained on 700GB C4 data) by converting the official FNet checkpoints using the tool from htts://gtthub.com/eescccchhtt
checkpoints (pre-
be loadable by the Pytorch codebase for fine-tuning For PoNet-Base Base, we fine-tune our pre-trained PoNet-Base- cuase-d with pre-training pyper-parameters shown in T that is, PoNet-Base pre-trained with 750K steps.
B ADDITIONAL GLUE RESULTS
B.1 COMPARE TO OFFICIAL BERT AND FNET CHECKPOINTS
Earlier studies show that due to its model capacity, Transformer-based PLMs benefit from larger pre-training data Liu et al. (019). Hence, we also show additional GLUE validation results. First, compare the performance between PoNet pre-trained on 5GB data and the official checkpoints of BERT pre-trained on 16GB Wikipedia and BooksCorpus data and FNet pre-trained on 700GB C4 data, shown in Table 7 The results with (1) in Table 7 are from (Lee-Thorp al. (02T). The results with (2) in Table 7 show the best GLUE results from searching 20 sets of
"huts://huggingfacc"
co/datasets/imdd htt.://gnnggaacoooaaaaaaassinnni
"hutss//huggingffce
http://nggnnncoollasssss//////ee co/datasets/yelp review full
19 "hhpps//github.com cwwp://..ooo....//..///////.cla. Stgun//l.ooo wowgdd-ommett-aateeee
"htps://uugginffcccco//iie co/hhen-hass--ucaseed
16

---

Published as a conference paper at ICLR 2022
0.00
0.5
0.75
0.70
950
0.65
0.3
0.60
0.55
0.59
3k
130K
2358
300K
5.54
630%
755k
850c
2005
5k.
150%
233k
350c
305k
633k
755K
B80k
1003K
Steps
Stepe
(a) MLM Accuracy 16GB Datasets
(b) SSO Accuracy 16GB Datasets
Figure 3: MLM and SSO validation accuracy 16GB datasets (Wikipedia and BooksCorpus) against the numbers of training steps from BERT-Base, FNet-Base BNse, and PoNet-Base. All models are uncased.
configurations based on Table/ōfoor fine-tuning the pre-trained models. For BERT-Base(2 and FNet- Base(2), we their respective pre-trained checkpoints provided by authors. For BERT-Base(2), we fine-tune on GLUE using the Huggingface BERT- BaRRT-BBse-uncseed checkpoints For (Net-Base(2), we re-evaluate FNet- Base-cased [Lee-Thorg al. 202T) on GLUE by converting the official FNet- Base-cased checkpoints Jssing the tool from by the PyTorch codebase for fine-tuning. For PoNet-Base(2), we fine-tune our pre-trained PoNet- Base-uncased with pre-training hyper-paramen parameters shown in Table/Bland pre-trained with 750K steps.
http:///uuu..oo/rssscc/eee/ pyttsschh pytorch to be loadable
is important point out that the GLUE results from official checkpoints of BERT-Bas and FNet- Base and our pre-trained PoNet-Base are not comparable since the pre-training data and pre-training tasks are different. BERT-Base was pre-trained on English Wikipedia (2.5B words) +BooksCorpus (800M words) with MLM+NSP, in total 20GB data size. FNet-Base data size) with MLM+NSP. In contrast, our PoNet-Base was pre-trained on Wikitext- 103 (100M Booksss-Bo (800M words) with MLM+SSO, in total only 5GB data size. Nevertheless, with the much smaller pre-training data, PoNet-Base still achieves GLUE AVG 77.54, reaching 92.84% of the accuracy of BERT GLUE (83.52) and outperforms FNet (74.23) by 4.5% relatively, and also better than 76.7 reported in the original FNet paper (they used different code implemented with JAX/Flax).
pre-trained
C4 700GB
We also compare other reported results to verify the correctness of our fine-tuning implementations. official checkpoints in our
Note that as reported in Table 7 fine-tuning the FNet-Base-cund Py -yToee--imleementee fine-tuning code achieved 74.23 AVG score on the GLUE Validation set. HuggingFace reproduced FNet-Base-cased with PyTorch and after pre-training FNet-base using the same pre-training data C4 and MIM and NSP tasks as the original FNet paper (Lee-Thorp et al.) 2021 and conducting GLUE fine-tuning, their FNet- Base-cased achieved 74.89 AVG score on the GLUE validation se which is comparable to 74.23 from our "Net-Base-casee fine-tuning results. These results verified the correctness our FNet fine-tuning implementation. HuggingFace also compares the GLUE validation results between their FNet-Bass Base re-implemented in PyTorch and the original FNet implemented in JAX/Flax, and also shows a significant gap from the PyTorch implementations Hence, think the difference between our FNet-Base GLUE score and the score reported in the original FNet paper (our 74.23 versus their 76.7) is also due to different code implementation (Py versus JAX/Flax) and different BERT-Base-uncaaeed checkpoints, we obtain the best GLUE validation results as AVG 83.52, which
settings. For the official
net Note that we exclude their WNLI accuracy from computing the AVG score GLUF, following the common practice from (Devlin et al.) 2019).
21
hhtts://huggingface. www../oll/ottttttooooooo.
hhtps:/huggingface wwwgoooll///eettbaaeeeeoeoo HuggingFace shows that the average score 9 tasks (including WNLI) from FNet-base(Py Torch) is 72.7 while it is 76.7 from FNet-base(Flax official).
17

---

Published as a conference paper at ICLR 2022
is better than the 82.62 AVG score HuggingFace reported from GERT-Base-caa 24 These results verified the correctness of our BERT fine-tuning implementations.
B.2 RESULTS FROM USING MORE PRE-TRAINING DATA
In order investigate the model capacity of PoNet, we also pre-train PoNet on the sufficiently large Wikipedia and BookCorpus dataset used for pre-training the official BERT model, in total 16GB data size. For a fair comparison, we pre-train BERT-Base, FNet-Base, and PoNet-Base with the same 16GB data, the same joint MLM tasks, up to IM steps considering the larger data amount. The hyper-parameters for pre-training on 16GB data are the same as shown in Table except for training 1M steps.
The MLM and SSO validation accuracies against the numbers of training steps are shown in Figure 3 Compared to the observations when pre-training BERT-Base, FNet-Base, and PoNet- Base on the 5GB data set, the SSO accuracy from PoNet is only slightly worse than BERT whereas the gap the MLM accuracy is bit larger. PoNet achieves significantly better MLM and SSO accuracy than FNet, consistent with its better sequence modeling capability shown on LRA.
GLUE validation results from BERT-Base, FNet-Base, and our PoNet-Base models pre-trained on the same 16GB data with MLM+SSO tasks up to 1M steps are shown as the group (3) in Table Compared with PoNet-Base(2) (pre-trained on 5GB data), PoNet-Base(3) improves the average score of GLUE by margin of 0.75 (77.54 to 78.29), which proves the potential of our model capture more knowledge with larger amount of pre-training data. Note that table 3 shows the GLUE fine-tune results when BERT-Base, FNet-Base, and PoNet-Base are all pre-trained on 5GB data with MLM+SSO tasks, where PoNet-Base reaches 95.7% of the accuracy of BERT (76.80 against 80.21). It is encouraging to observe here that in this fair comparison with all three models pre-trained on the much larger 16GB data, PoNet-Base(3) also reaches 95.9% of the accuracy of BERT-Base(3) on GLUE (78.29 against 81.63), demonstrating the competitive transfer learning capabilities of PoNet. Note that BERT-Base(3)) has a lower performance than the official BERT- Base(2), which is mainly due to the difference in batch size constrained by machine resource limitations. Although we conduct 1M training steps, the batch size is still smaller than that of the official BERT (98,304 versus 128,000 words/batch). This degradation is also caused by the data pre-processing scripts we use as well as slightly by the FP16 training strategy we adopt. Note that BERT, FNet, and our PoNet all suffer some performance degradation from these factors, and the performance comparison the group (3) in Table iis is fair comparison.
C VISUALIZATION
C.I STATISTICAL ANALYSIS
To study the importance of the three granularities of pooling, i.c., GA, LMP, and SMP, at different layers of PoNet, we take 4000 examples, calculate the mean of the L2-Norm of each type of pooling for the 4000 examples at each layer, as well as the average of the means for the three types of pooling. The resulting curves are shown in figure Specifically, the norm is calculated as follows,
LN
L2-Norm
Σ h.1
(10)
H
LN
(11)
D
where i, h, d denote the dim of example, attention head, text length, and hidden unit, respectively. We observe that
(1) All three curves corresponding to the three types of pooling are close to the average curve in the graph, indicating that all three types of pooling play significant role in PoNet.
18

---

Published as a conference paper at ICLR 2022
0.35
SMP
0.30
0.25
Nom
0.20
2
7
11
12
Layer
Figure 4: The L2-Norm of the three pooing GA, SMP, and LMP and their average at different layers of PoNet.
0.200
0.30
0.150
0.20
0.325
Deage
thoggh
0.05
D191
1500
10
32
30
11
1syers
laysrs
(a) GA Attention
(b) SMP Argmax Positions
Figure 5: The GA attention map and SMP argmax positions for the example "The word had filled his head as though the girl had whispered directly into both ears."
This observation is consistent with the ablation results shown in Table 5 which shows that removing GA, removing SMP, or removing LMP in PoNet all significantly degrade the performance on downstream single-sentence and sentence-pan pair tasks.
(2) All three curves corresponding GA, SMP, and LMP share similar trend of going down, up. down, slowly rising and going down. From Layer to 8, the norm values of SMP are greater than those of LMP and GA. Higher than Layer 9, the norm values of LMP turn out to be greater than those of SMP. These observations indicate that the significance of different granularities of pooling changes across layers.
(3) The value of GA is relatively low compared that of SMP and LMP. On the other hand, as shown in the ablation results in Table removing GA from PoNet (PoNet w/o GA) degrades performance on SST and STS-B significantly, demonstrating that the sentence- pair tasks heavily rely on the global information.
C.2 CASE STUDY
To analyze how PoNet works, we loaded a pre-trained PoNet and selected an example, "The word had filled his head though the girl had whispered directly into both ears.", for visualization. For
19

---

Published as a conference paper at ICLR 2022
GA, average the attention weights of all heads as the attention weights of each layer. For SMP, count the number of times all hidden layer dimensions were taken as the segment maximum value. The resulting GA attention map and SMP argmax positions across layers are shown in Figure The tokens "[CL_S]" and "[SEP]" are excluded in the SMP visual map since they belong to a single word segment. To sharpen the display, the maxima in the figure are truncated to 0.2 0.3 accordingly. On the bottom layer, we observe that GA first focuses on some of the more important words, such as "whisperedd at Layer I and then attends the rest words at Layer 2. This complementary attention allows the model to capture more comprehensive information about the sequences after multiple layers. We observe that SMP is more inclined to capture information about punctuation and pronouns at the bottom level and then some keywords, e.g., "whispered", also begin to receive attention at the higher level, especially at Layer 9-11 based on the SMP argmax positions.
D COMPARISON BETWEEN PONET AND OTHER MODELS
D.1 DIFFERENCE BETWEEN PONET AND FASTFORMER
First, our PoNet consists of three components, i.c., global aggregation(GA)), segment max- Po)) and local MaX-P).linggLMMP). The importance of SMP and LMP is demonstrated in Section 5 Second, the GA component in PoNet is different from Fastformer (Wu et al., 2021) due to different motivations and implementations, follows.
In the motivation aspect, our GA is inspired by the global attention in Longformer and BigBird, but is also different from them (see Appendix D.2) In contrast, Fastformer uses the additive attention mechanism to model global contexts.
In the aspect of implementation, firstly, comparing GA in PoNet with Fastformer, the global query is computed differently. In Fastformer, the global query vector is a weighted sum of all query vectors; whereas, in PoNet, is an average pooling of all query vectors.
Secondly, the global query is mixed with key and value in different ways. In Fastformer, element- wise product performed between the global query vector q and each key vector k; obtain a set of vectors p;. A set of learnable vectors are introduced for additive attention to compute weights for pi- After softmax normalization of the weights, the sum of weighted p; is the global key (GK). Then, element- wise product is performed between GK and each value vector to obtain set of vectors U. After that, U passes through a linear transformation layer to obtain its hidden representations R. And the hidden representations R are element-wise summed with Q to obtain the final output of Fastformer.
In our GA, as described in Section 3.1.1 the first stage value for GA, denoted by y in the paper, serves as a query token to interact with the input sequence through cross-attention. There are several differences from Fastformer. First, standard scaled Not-production is performed matrices, and the weight of the scalar is obtained directly. Then, softmax is applied on the scaled dot- product results weights for V. The weighted sum of V is computed the global representation, denoted by q' in the paper, and there is no process of global key (GK). Finally, as described in Section 1.4.4 we perform element-wise product between g' and Hon to compute the final global representation for pooling fusion, where Hon is from projecting input embedding of each token through another linear transformation specific for pooling fusion. Different from Fastformer, there is no linear transformation layer nor summation with Q
Q and K
The above comparisons show that the GA in our PoNet quite different from Fastformer. In addition to algorithm comparisons, we also experimented with replacing the GA module in our PoNet with Fastformer, and observed that after the same pre-training and fine-tuning configurations, replacing GA with Fastformer has lower performance compared to our PoNet: On CoLA the performance dropped from 45.36 to 43.28; on STS-B, the performance dropped from 84.57 to 84.46.
D.2 DIFFERENCE BETWEEN PONET AND LONGFORMEE/BIG
Our design of the multi granulartttu pooling and pooling fusion is inspired by previous works such as Longformer and Bird Our design considers the importance of capturing context information
20

---

Published as a conference paper at ICLR 2022
from different granularities and combining the captured information. However, design is different from previous works.
Longformer uses window-based self-attention to capture local context and task-specific global attention to encode inductive bias about the task. For example, for classification tasks the global attention attends to the [CLS] token and for question-answering tasks, the global attention attends to all question tokens. In contrast, GA in PoNet is task-agnostic.
BigBird combines global, windowed local, and random attentions. BigBird chooses subset from existing tokens as global tokens or adds additional global tokens such as [CLS]. These global tokens attend to all existing tokens. The GA in PoNet is computed differently from BigBird (see Section 3.1.I and Section 3.1.4 for details). Also, PoNet uses GA, segment max-pooling and local max-iooling. We experimented with replacing our windowed local max-pooling with random pooling in an early PoNet structure with a small number of pre-training steps, and observed that this change weakened PoNet's capability of capturing local context and significantly degraded the accuracy of downstream single-sentence CoLA task (from 42.57 to 31.44) while causing a slight gain on the sentence-pait pair STS-B task (from 80.29 to 80.48). Hence, we did not use random attention in the final structure of PoNet.
D.3 DIFFERENCE BETWEEN PONET AND LUNA
The mechanism in Luna Ma et al. 2021> is also somewhat similar to our GA module. Luna introduces additional "global" sequence of fixed length L, which is used query cross- attend with the original input to obtain global representation of length L. Then, is served as key and value to perform the second cross-attention with the original input obtain the final output. In fact, Luna changes the parallel structure of the global attention in Longformer/BigBirr to a serial structure. Also, Luna does not pass the updated global tokens (P") through the FF layer as in BigBird; instead, Is is output directly.
For PoNet, instead of introducing additional tokens preform cross-attention, we use the results of average pooling as the global representation of the first stage of GA. Also, in the second stage of GA, the global representation is served as query in GA rather than key and value in Luna to calculate cross-attention. While Luna is able to directly obtain an output of the same length as the sequence, our GA obtains an output consistent with the length of the first stage global representation (i.e., I), so we further integrate this output into the original input by element- miienttwwieee multiplication.
D.4 DIFFERENCE BETWEEN PONET AND FNET
To the best of our knowledge, our work is the first to comprehensively explore the full potential of the simple pooling mechanism for token mixing and verify that built upon qulti-granularity pooling and pooling fusion, our PoNet is competitive in modeling long-range dependency with a linear complexity. The token mixing mechanism in PoNet, built upon pooling mechanisms, is completely different from Fourier transforms used by FNet.
As shown in Table [ the LRA benchmark, PoNet achieves accuracy of 61.05, which is significantly better than the accuracy from FNet (55.30 30 reported in the original paper with JAX/Flax implementation, 53.10 in our PyTorch implementation). PoNet achieves significantly better pre- training accuracy (with the same MLM+SSO tasks) than FNet, as shown in Figure 2 PoNet achieves significantly better transfer learning performance on GLUE than FNet, as shown in the updated Table 3 and significantly better transfer learning performance on long-text classification tasks than FNet, as shown in Table
E ADDITIONAL MODEL CONFIGURATIONS FOR PONET
We explore several additional ideas to improve PoNet.
E. TREE MAX-POOLING (TMP)
For different layers in PoNet, we apply different dilation sliding window max-pooling as another way to exchange the information between two different tokens. We compute the TMP value T
21

---

Published as a conference paper at ICLR 2022
R. Nxd The size (length) of the dilation windows is based on 2. that the I-th layer, ditatiiionn 2'-1. The lowest level, ie. Level 1, uses length of I for dilation, which is a normal sliding window max-pon max-pooling. Since each length of the dilation sliding windows can be represented by binary and the connection state of tokens can be represented as {0. 1 (not link or link), the distance between any two tokens can be reached by this structure. We can easily calculate that the longest distance that the structure could reach was 2 - 2. After adding TMP into pooling fusion for PoNet, we observed that the MLM validation accuracy improves in pre-training, but we did not observe performance improvements on the downstream GLUE tasks.
E.2 CONTEXTUAL DECISION MECHANISM
Since different tokens may need different levels of information, strictly adding the three pooling features together for all tokens, in Equation cannot meet this requirement. We consider another aggregation method, denoted Contextual Decision Mechanism, to replace the Pooling Fusion in Section 3 Following the idea of attention, each token conducts cross-attention with the 3 pooling features for contextual interactions, as follows:
M = Attention KMn VMnn E RR
(12)
where
QH = HW KH + ВKи.
(13)
K M = Ig'; Sn; W + .KKM
(14)
VMn Sn: W.i + .V
(15)
and W. E Rdxdd b. E Rd are parameters to be learned. Note that KM VMn E R3xd M' is then the final output of the multi-granularity pooling block. Similar to the TMP idea, when switching from the pooling fusion in Section 3 to this contextual decision mechanism for PoNet, we observed that the MLM validation accuracy improves in pre-training, but did not observe performance improvements on the downstream GLUE tasks.
F FUTURE UPDATES FOR PONET
F.1 REDUCING THE NUMBER OF PARAMETERS
Note that after sharing WK, bx, and WV, by, in Equation 1 our current PoNet model has total of 5 linear transformations from Equation [1 in the token mixing layer. Note that other transformations, such as WK, and W., are difficult for sharing, because the corresponding projection outputs from WK, and W, are used for different procedures of global cross-altention and segment max maxx-pooligg respectively. Sharing them will cause difficulty in parameter updating. For other parameters, since WQ, will only be applied on single token (global token) instead of on the whole input sequence, its computational cost can be ignored. The main computational cost comes from the remaining 4 transformations. To further reduce the number of parameters PoNet, we plan to also remove W.. This is based on the consideration that the original input sequence is not involved in the computations of GA, SMP, and LMP, hence it can be mixed directly with GA and SMP without transformations. This leaves with only 3 sets of linear transformation parameters that require the computations of the whole sequence, namely, shared WK, and WV,, W., and W. Then PoNet will have the same number of linear transformations BERT, which has a total of 3 linear transformations in the self-attention layer. We will also investigate sharing these linear transformations across different PoNet layers, which will further reduce the number of parameters. We plan to run more experiments to evaluate the effect on the performance of PoNet from these methods on reducing the number of parameters.
F.2 SUPPORT CAUSAL ATTENTION
The key issue for supporting causal attention is prevent information leakage from the moment t > T to the sequence t <= T. If the computational complexity of the complete sequence is O(N), to ensure that the newly added > T moments still maintain the O(N) complexity, the computation
22

---

Published as a conference paper at ICLR 2022
cost must be guaranteed to be (11) the introduction of the T + moments, i.c., it needs to be possible to recursively push out the results from <= T to the T + 1 moments.
Considering the three different pooling types in PoNet, firstly, since LMP and SMP do not actually introduce multiplication, the computational complexity is negligible. In order to guarantee that the complexity of the Max-pooling operation is O For LMP, it is sufficient to change the pooling window to a causal window without adding extra computation. For SMP, since max(St<<--1))
make the following improvements.
max (ST, max(S:<=T)), the additional token computation can be bounded to (1) by accumulating max, and the overall computation ((N) will not change. However, for GA, it's feasible since there are two variables involved in the calculation of attention in cross-attention, one of which is sequence of length T'. The T+1 moment inevitably introduces O(T) computation, which makes the overall computational complexity rise to O(N2). We consider weaker model scheme by removing SS-GA (PoNet w/o SS-GA) (also mentioned in section 5). As shown in 5 PoNet w/o SS-GA has no performance degradation over the original model on MLM and single-sentence classification tasks, i.e. CoLA. These two tasks reflect the ability of sentence encoding, which is important for auto-regressive decoding task. PoNet w/o SS-GA will only have additive calculation in the GA stage. and the computational complexity negligible. Similar to SMP, we can also simplify the computation by using the recursive property of mcan, mean (S:<=r++1) ("emea((S<c=7) Sr-1 Overall, this causal attention in PoNet has O(N) computational complexity.
T+1
23


Reviews:
[{"review_id": "lu2dQwnKPOF", "reviewer": "Reviewer_ssBH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Their method is easy to understand and highly motivated. \n2. Extensive experiments have been conducted and reasonable performance has been obtained.\n3. Ablation studies show the contribution of these pooling mechanisms. \n\nWeakness:\n1. PoNet Is suitable for language representation learning but to my understanding, it is impossible for PoNet to build generation models such as machine translation.\n2. I do not find the number of parameters for PoNet. Will these pooling methods introduce a lot of extra parameters?\n3. Some relevant works should be discussed, such as Luna: Linear Unified Nested Attention.\n4. More importantly, the authors introduce the SSO during the pertaining. Is the FNet pertaining also using this objective? If not, I think the comparison is not fair.\n\nMinor comments:\nIn the abstract, authors claim that \"PoNet significantly outperforms Transformer\" which should be in terms of memory and speed, right? Maybe here could be clearer.", "summary_of_the_paper": "In this paper, the authors aim to resolve the quadratic time and memory complexity of the standard attention mechanism. They introduce a multi-granularity pooling and pooling fusion that captures contextual information from various levels (global, segment, and token).  They conduct experiments on long sequence tasks and large-scale pre-training and fine-tuning. As a result, their model achieves great performance with speed-up. ", "summary_of_the_review": "The authors introduce a pooling-based module to replace the self-attention layer in the Transformer in order for fast training speed and low memory usage. Extensive experiments show the effectiveness of their methods. However, some details of their models should be further clarified in the paper (such as the number of parameters). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5xRisY-DQTU", "reviewer": "Reviewer_zcvh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The PoNet attention mechanism is very attractive as it intuitively makes sense with global, segment, and local inductive biases. PoNet also distinguishes itself with a low constant factor and performs well at even short sequence lengths. Experiments are very well done. Explanations and figures are clear.\n\nI would have liked to see more discussion of the inductive biases beyond the ablation experiments and visualization in the appendix. For instance, I'd be curious if the different granularities are more or less important at different layers. Based on the single sentence in the appendix, it's hard to get an idea what the attention mechanism is actually doing, too. Some tasks beyond classification that may better be able to use structure like question-answering, translation, or summarization may be interesting, too.", "summary_of_the_paper": "PoNet addresses the quadratic time and memory complexity of Transformer with a new attention mechanism that has only linear complexity. Theoretical (Section 3.2) and experimental results (Table 2) bear this out. Experimental results on standard datasets like Long Range Arena and GLUE are very competitive with SOTA. Ablations were done on the 3 components of the attention mechanism, too.", "summary_of_the_review": "My recommendation is to accept the paper. It builds upon much work about exploiting inductive biases to make a more efficient transformer. While not revolutionary, it is a well-written paper with good analysis and experiments. The idea is has good intuition behind it, and the fact that it is a drop-in replacement for self-attention encoders makes it attractive for ML practicioners, too.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4OF33_tB4xB", "reviewer": "Reviewer_Xvc7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The architecture of PoNet is simple and the paper is clearly written.\n\nHowever, there are some weaknesses:\n\n1. The proposed architecture of PoNet is not novel. The key component, which is the global aggregation module, is very similar to FastFormer. The other two components are also straight-forward and less important than GA.\n\n2. The experimental results are not entirely convincing, especially those on large-scale pre-training and fine-tuning. The BERT-base model in the second group of Table 3 is trained on the same data of the original BERT paper, with and advanced SSO loss and more update steps (125k in original BERT paper vs. 340k in this paper). However, the fine-tuning performance of the BERT-base in Table 3 on GLUE is even worse than that in the original paper. In the paper of Linformer (Wang et al., 2020, Table 2), they re-trained a BERT-base model with only the MLM loss and 250k update steps, but their scores on GLUE are also better than that in Table 3. \n\n3. The authors did not clarify if the PoNet can be used in auto-regressive attention.\n\n4. Personally speaking, some claims in this paper are not well-supported. For example, the last sentence in page 2 claims that the low-rank approximations used in Linformer and Nystromformer limit the ability of looking at the full sequence and hence degrade long-range modeling capabilities. I do not see why PoNet is better than these low-rank methods on modeling long-range modeling.\n\nQuestions:\n\n1. In the experiments of LRA, in Table 1 the authors said that they used the same setting as (Xiong et al., 2021). However, in Table 2 they said the setting is the same as (Tay et al., 2021). Are these two experiments using different settings?\n\n2. In table 2, Performer is 4.3 times faster than Transformer under 4K sequence length. However, in the original LRA paper (Tay et al., 2021), the number is 5.7. What is the difference?\n\n3. In the LRA paper, the authors clarified that the sequences in the text classification task are trunked to length 4K. Why in Table 2, there are two columns with 8K and 16K lengths?\n\nMissing references:\n\nThere are some relevant papers that are missed in the related work.\n\nZhu et al., Long-Short Transformer: Efficient Transformers for Language and Vision. NeurIPS 2021.\n\nMa et al., Luna: Linear Unified Nested Attention. NeurIPS 2021.\n", "summary_of_the_paper": "This paper proposed PoNet, which is an efficient architecture to replace self-attention in Transformer-based models. \nPoNet consists of three components, which are called multi-granularity pooling block.\n\nThe first pooling component is the global aggregation module, which is very similar to the additive attention in FastFormer. The different is that in FastFormer, the global query vector is computed by a weighted sum of the query vectors, while in PoNet it is the average of the query vectors. The segment max-pooling and local max-pooling modules are straight-forward, which are max-pooling operations on pre-defined segments and on local sliding windows, respectively.\n\nExperiments were conducted on Long-range Arena (LRA) and large-scale pretraining and fine-tuning.\nThe authors also conducted ablation experiments to analyze the contributions of these three multi-granularity pooling components.", "summary_of_the_review": "To sum up, the idea in this paper is incremental and more empirical evidences are needed to demonstrate the effectiveness of the proposed PoNet architecture.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1Je4SmFaAHQ", "reviewer": "Reviewer_vM28", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. This paper accelerates calculation and retains accuracy in common benchmarks. \n2. The paper is clear.\n3. Although the idea of each module in multi-granularity pooling is not entirely new, the combination is novel and comprehensive.\n\nWeaknesses: \n\n1. The improvements are not significant over the previous token-mixing method, the FNet. Moreover, their experimental settings are different, and the FNet gets lower performance in this setting. As to the originality, FNet does token-mixing, the contribution of Ponet is incremental.\n2.  The paper relies on empirical results but does not report error bars.\n3. The paper claims it designs a new pretraining objective in their abstract, but they do not validate this claim well.\n4. They do not report their speedup on the GLUE benchmark (e.g., reduces xx% GPU/TPU time) but only report that they lose 4% accuracy than the Transformer baseline. \n5. This paper lacks theoretical evidence or understandings.\n\nMinor: \nThe right subfigure of Figure 1 is a bit confusing.\n", "summary_of_the_paper": "This paper proposes PoNet, an efficient model with linear complexity for modeling long sequences. This model replaces the self-attention layer of the Transformer model with its pooling network. \n\nTo aggregate information from different levels, their pooling network consists of three pooling strategies: \na. Global aggregation that aggregates information from all tokens in a sequence. This method is a bit like BERT CLS, and the difference is that they use average representations rather than the representation of the first token.\nb. Segment Max-pooling that performs max-pooling in each sentence to capture sentence-level information. \nc. Local Max-pooling that performs max-pooling in each window to capture local information.\nTheir fusion method is the addition operation.\n\nAs to the training objective, they not only use the MLM, but also use the sentence structural objective as in StructBERT.\n\nThis paper contributes to a well-defined but highly-influential problem: Efficient modeling for long sequences. The main contribution of this paper is the introduction of a multi-granularity pooling that is more efficient than self-attention and brings higher accuracy than previous efficient methods.\n\nEmpirical studies are performed to show the superiority of PoNet over previous SOTA FNet. Their method outperforms FNet in terms of accuracy in their experiment settings but slightly lags regarding time efficiency.\nResults are shown on the task of (1) Long-range Arena (2) GLUE by fine-tuning pre-trained networks.\n", "summary_of_the_review": "The paper is good but not enough. It empirically improves the accuracy of classification and language modeling. But the significance and the originality are not high, and they do not bring us much new knowledge. Their comparisons to baselines are not sufficient, e.g. they can be benefited from reporting GPU/TPU time. Since it does not meet the high standard of ICLR, I tend to reject this paper.\n\n---------------Post Rebuttal-------------\n\nThe authors have clarified my concerns about performance, I decide to raise the score from 5 to 6.  I still recommend authors to provide code, report error bars, speedup on GLUE. Moreover, the combination of StructBERT and MLM can not be a significant contribution. Finally, the challenge of combining these pooling methods and pertaining objectives should be discussed and well-evidenced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work reduces the time and memory complexity of Transformer for long sequences by using multiscale pooling to reduce attention from quadratic to linear complexity. Theoretical and experimental results show good results and are very competitive with the state-of-the-art. The paper is well written and experiments are thorough. The additional results in the rebuttal also helped reduce some of the reviewers' concerns. Though the work is somewhat incremental and the experimental settings for the baselines are different, the thoroughness of the experiments and the good results make this a good addition to the conference. For the final version, the authors should provide code, provide error bars and details of the speedup of GLUE.
Decision: Accept (Poster)