Efficient Classification of Long Documents via State-Space Models
Authors: Peng Lu
Suyuchen Wang
Mehdi Rezagholizadeh
Bang Liu
Ivan Kobyzev
Venue: EMNLP 2023 Main (2023)
Keywords: Long Document Classification
State Space Models
Efficient NLP

Abstract: Transformer-based models have achieved state-of-the-art performance on numerous NLP applications. However, long documents which are prevalent in real-world scenarios cannot be efficiently processed by transformers with the vanilla self-attention module due to their quadratic computation complexity and limited length extrapolation ability. Instead of tackling the computation difficulty for self-attention with sparse or hierarchical structures, in this paper, we investigate the use of State-Space Models (SSMs) for long document classification tasks. 
We conducted extensive experiments on six long document classification datasets, including binary, multi-class, and multi-label classification, comparing SSMs (with and without pre-training) to self-attention-based models. We also introduce the SSM-pooler model and demonstrate that it achieves comparable performance while being on average 36\% more efficient. Additionally our method exhibits higher robustness to the input noise even in the extreme scenario of 40\%.

Efficient Classification of Long Documents via State-Space Models
2 ;uuuuhenn Wang 22,, Mehdi Rezagholizadeh', Bang Liu", Ivan Kobyzev' HHawwei Noah's Ark Lab, Canada
Mehhi
2 Department of Computer Science and Operations Research, Université de Montréal
(peng.lu, suyuchen.wang bang.liu) @ @umontreal.ca
rmehhii.ezaghhoizzaeee,, woan.kobyeev] @huawei.com
Abstract
fication for transformers (Dai al., 2022). One of the common approaches to simply truncate long texts to a pre-defined length, c.g. 512, which makes pre-train pre-trained models to be applicable for them. Some work demonstrated this technique is not sufficient for long documents due to the missing of important information (Dai et al., 2022).
Transformer-based models have achieved state- of-the-art performance on numerous NLP ap- plications. However, long documents which are prevalent in real-world scenarios cannot be efficiently processed by transformers with the vanilla self-attention module due their quadratic computation complexity and limited length extrapolation ability. Instead of tack- ling the computation difficulty for self-attention with sparse or hierarchical structures, in this pa- per, we investigate the use State-Space Mod- els (SSMs) for long document classification tasks. We conducted extensive experiments on six long document classification datasets, including binary. multi-class, and multi-label classification, comparing SSMs (with and with- out pre-training) self-attention-balt based models. We also introduce the SSM SSM-pooler model and demonstrate that achieves comparable perfor- mance while being average 36% more effi- cient. Additionally our method exhibits higher robustness the input noise even in the ex- treme scenario of 40%.
Another sort of technique attempts to reduce the computation overhead of attention-basee sys- tems. This problem has several relevant solutions, e.g. Sparse Attention models (Beltagy al., 2020) and Hierarchical Attention models (Chalkidis al., 2022). One of the important sparse attention meth- ods is Longformer, which leverages local and global attention to reduce the computational com- plexity of the models and increases the input length up to 4096 tokens. Another popular sparse at- tention method is BigBird (Zaheer et al., 2020): besides the global and local attention, it includes extra random attention modules to attend pre- defined number of random tokens. Apart from designing sparse attention mechanisms, Hicrarchi- cal Transformers (HAN) like ToBERT (Pappagari et al., 2019) propose to construct systems on top of the conventional transformer (Chalkidis et al., 2022). Basically, the long text is first split into several chunks less than fixed number, e.g. 200. Next, every chunk is encoded by a vanilla trans- former one by one to form a collection of chunk representations and then another transformer pro- cesses the sequence of chuck representations.
1 Introduction
Since the emergence large-scale pre-trained lan- guage models such as BERT (Devlin et al., 2019) and GPT3 (Brown et al., 2020), these transformer- based models have become popular solutions for many text classification and generation tasks. How- ever, their benefit is constrained short-length inputs when the computation resource is limited be- cause attention module requires quadratic computa- tion time and space. More specifically, each token in a sequence of length N requires pairwise com- putation with all N tokens, which results in O(N2) complexity. Such limitation makes transformer- based models hard to process long sequential data efficiently. There are many works aiming to im- prove the performance Long Document Classi- Lab (Montreal).
Apart from the computation complexity issue, some works pointed out that the length extrapola- tion ability of self-attention-bane limited (Press et al., 2022; Ruoss et al., 2023), namely, the transformer models perform poorly during inference if the sequence length of test data is beyond the sequence length of training data. As an order-invariant encoding mechanism, the self- attention- based encoder heavily relies on the Posi- tion Embedding (PEs) to model input orders, how-
models is quite
"Rescarch done during internship in Huawci Noah's Ark

---

ever, these works demonstrate that the failure of transformers on long sequence is due the limited length generalization ability of these position em- bedding methods. This also encourages exploring alternative architectures for the challenge of long document classification problems.
Sankkekk Haar
Pood Loye
Space Nood
Namaliatoo
Recently, Gu et al. (2022) propose modeling sequential data with space models and show surprisingly good performance for comparing Transformers over long sequential data (Tay et al., 2021). However, this benchmark only consists of one single text classification task. It is still unclear about the efficacy and efficiency of state-space models for long document tasks com- pared to transformer-bant
Layer
O
benchmark
Pozing
oo
p
Bank Scace
Nomaliation Lawer
models.
Stato MoceiR
In this paper, we aim to fill this gap with comprehensive by conducting extensive periments and analysis on six long document clas- sification benchmarks. Besides, we develop an efficient long document system and show that the satt--pacce-basee models outperform self-attention- based models (including sparse attention and HAN) and yield consistent performance across datasets with much higher efficiency and more robustness to input noise.
Figure 1: The illustration of the architecture of the State- Space-Poolet model.
also be rewritten in a linear convolution manner facilitating the encoding speed.
K CC,, CAB
CA B).
k 21 L CA'B 1=
2 Background and Methodology
(3)
2.1 State Space Model
this section, we briefly introduce a recent long dependency modeling method utilizing State-space models to encode sequential data (Gu et al., 2022). The state-space model is defined by the following equations, which map the I-dimensional continu- input signal u(!) to an N-dimensional hidden state (f),, and this hidden state then projected to a 1-dimensional output y()):
where K RC is defined as the SSM kernel.
Given input sequence [r1,-... Ft E R the corresponding outputs y [V1, (.,...., ØL) R of the convolution y K * can be computed efficiently OL log L) time with the Fast Fourier Transform (FFT) (Cormen et al., 2009).
The computational complexity of the SSM algo- rithm is OLL L) in convolution mode, which enables it process significantly longer input quences with the same hardware resources com- pared to the attention mechanism O(L²). This re- sults in a substantial reduction in information loss from the input side. Next, instead of relying on a limited number of global attention mechanisms to capture long-term term dependencies in sparse atten- tion models, the SSM model is designed to model long sequences dynamic systems. By utilizing Hurwitz state matrix A (Goel al., 2022; Gu et al., 2023), it preserves long-term dependency in- formation in high-dimensional state. Such states enable SSM models strong ability to capture long dependency information.
a' zt()) Ax(t) + Bu(t), y(t) Ca(t) + Du(t),
(1)
where A, B, C, D are trainable parameters. A discrete sequence, like text, can be regarded as discretized data sampled from continuous signal with step size A, and the corresponding SSM in the recurrence manner is:
hg = Alu-1 + Bak
yk Chy Drok
A (I 2/2 - (I 1/2 A),
(2)
where A is the discretized state matrix and B, C, D share the similar formulas. The SSM can

---

Table Test Performance on six LDU datasets with or without pre-training. found SSM-based models achieve significant improvement over selfattiention-baee methods.
wlo pre-training
Models
ECUH
Hyperpartisan
20News
EURLEX
BOOK
AMZ
Avg.
Transformer (ólayer)
58.7
89.5
73.4
62.1
46.0
35.9
60.9
Longformer (6layer)
62.2
90.3
76.9
62.9
47.3
40.4
63.3
S4
64.4
93.8
76.2
72.6
49.7
44,9
66.9
S4-poeler
64.2
93.8
76.1
72.0
48.0
44.2
66.4
w/ pre-training
Models
ECIHR
Hyperpartisan
20News
EURLEX
BOOK
AMZ
Avg.
BERT
71.7
91.8
84.7
73.2
58.2
51.1
71.8
BERT+random
72.8
89.3
85.0
73.3
59.2
56.8
72.7
BERT+textrant
73,5
91.2
84.7
72.9
58.9
56.9
73.0
Longformer
81.5
93.7
83,4
71.5
58.5
56.4
74.2
HAN
77.2
89.5
85.5
69.6
57.3
54.6
72.3
H3
82,9
94.0
85.9
76.7
60.9
57.9
76.4
Ho--e
82.1
94.2
84.1
76.4
60.5
57.7
75.8
2.2 SSM-based Systems
quence level computed with the average of the last layer and then is fed to a fully-comnected dense layer with softmax or sigmoid function to output the prediction probability for multi-class or multi- label problems, respectively.
In practice, recent SSM-based systems, e.g. S4 (Gu al., 2022; Goel al., 2022) built up with block as shown on the left of Figure 1. This block shares a similar structure as the self-attention module with pre-Layer Normalization (Ioffe and Szegedy, 2015; Ba et al., 2016) but replaces the multi-head attention with a discretized state-space model. The Structured State Space sequence model (S4) (Gu et al., 2022) utilizes this structure and leverages the Cauchy kernel method to simplify the kernel computation of SSM. Building this idea, Hungry Hungry Hippo (H3) (Dao al., 2022) extends the mechanism by including an additional gate and short convolution obtained via shift SSM to improve the language modeling ability of -SM-based systems.
Experiments
In this section, we present experimental setup and results for multiple datasets.
3.1 Long Document Understanding (LDU)
Dataset. We first evaluated the effectiveness and efficiency on commonly-used long text clas- sification datasets including Book (Bamman and Smith, 2013), ECIHR (Chalkidis et al., 2021), Hy- perpartisan (Kiesel et al., 2019), 20News (Lang, 1995), EURLEX (Chalkidis et al., 2019) and Ama- zon product reviews (AMZ) (He and McAuley, 2016). For the AMZ dataset, we randomly sam- pled product reviews longer than 2048 words from the Book category. We report accuracy for binary and multi-class classification tasks (Hyperpartisan, 20News, and AMZ) and macro-F1 for the rest of the multi-label classification problems. The de- tailed dataset statistics are included in Appendix A.
2.3
PatteSSpace-Pollee
To further improve the system efficiency, we pro- pose modification of the system architecture which imposes a progressive reduction of the in- put length for deeper layers. Figure depicts our system architecture. The model progressively con- structs the representation from the token level to- ward the final representation level. By inserting max pooling layer between each SSM block. the model each level automatically extracts the im- portant information between nearby inputs and re- duces the input length to half of the previous layer, which further accelerates the speed training and inference. The final representation of the long se-
We compare methods with Trans- former (Vaswani al., 2017). Longformer and S4 with and without pre-training. More specifi- cally, for models without pre-training, we compare 6-layer S4 and S4-pooler models (1 parame- ters) with 6-layer Transformer (70m parameters)
Baselines.

---

and Longformer models (99m parameters). For models with pre-trained checkpoints', we choose BERT-base with truncated input length and its two variants BERT-random and BERT-textrank (Park et al., 2022), we choose one sparse attention model: Longformer (Beltagy et al., 2020) and one hicrar- chical transformer model (HAN): ToBERT (Pappa- gari al., 2019). The detailed model settings and hyperparameters are included in Appendix B.
Traihing Time for 1 Epoch
Transformer Longtormer
pooter
Traning Time tscooes
10a
Results. Table shows the results of different methods on six commonly used long text classifi- cation datasets. Among the models without pre- training, the Transformer models perform more poorly than all other models by large margin (3% 6%). S4 and S4-pooler perform significantly bet- ter than Transformer and Longformer models on all datasets except for 20news, which is probably be- cause the average length of this dataset is relatively short. On average, S4 and S4-pooler outperform the Longformer by 3.4% and 3.1%, respectively. For models tne-tuned with pre-trained checkpoints, although the BERT models with truncated inputs improve a lot compared transformers without pre-training, they still underperform other methods, which indicates the necessity of processing longer text for this task. Again, the H3 and H3-pooler models give better performance on average. Be- sides, the S4-pooler and H3-pooler models demon- strate comparable performance to the correspond- ing original systems with around 36% training time reduction. We discuss the training time in the anal- ysis section.
Euriex
Ecthr Case
Books
20news
AMZ
Datanetts
Figure 2: training time comparison of methods on five datasets. All models are trained single Nvidia V100 GPU for one epoch.
Table 2: Ablation study of different pooling structures.
Models
AMZ
ECIH
EURLEX
Book
$4
44.9
64.4
72.6
49.7
S4-pooler-1-2
64.2
48.0
S4-pooler-1-3
44.6
64.0
69.4
47.8
S4-pooler-15. 1-5
43.7
64.2
71.8
47.4
S4-pooler-2-2
43.1
63.4
72.3
24-pooler-2-3
63.1
71.5
50.3
S4-pooler-2-S
39.9
64.2
49.7
ciency compared transformer and longformer models. Even with an times greater maximum put length, S4 and S4-pooler models only took 42% and 27% of the training time transformer, respec- tively. They also only took around 8% and 13% of the longformer's training time. By introducing pooling layers, the training time of the S4-pooler is reduced to 64% of S4 models on average.
4 Discussion
this section, we investigate the training effi- ciency, the impact of different down- wown-sampling structures and the robustness to input noise of SSM- based models.
4.2 Ablation Analysis
The pooling layer down-samples the input gradu- ally, hence reducing the computation cost. More- over, changes the resolution of the input sequence by choosing different window sizes on different lay- ers. We conducted experiments to investigate the impact of the hierarchical structures of S4-pooler models. S4-pooler-x-y refers to placing the pool- ing layer every layer with window size y. The results are averaged over three random seeds. The performance of different tasks is influenced by the hierarchical structures. For example, enlarging the sown-sampling ratio from decreases the per- formance of AMZ, while having less impact on the
4.1 Time Analysis
Figure shows the training time of different mod- els on single Nvidia V100 GPU for one epoch The maximum input length set to 512 for trans- former models and 4096 for Longformer, S4, and S4-poole models. All models use the same batch size of 16 and layer number of 6. SSM-based mod- els have significant advantage in training effi-
'Our experiments were conducted with Huggingface toolkit
"The experiments of SSM models are adapted from hatp:/ghhhhh..ooooaoeeeeeeecaaae

---

66
References
S4 S4-parler
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
64
62
Mscro-F
David Bamman and Noah A. Smith. 2013. New align- ment methods for discriminative book summarization. CoRR, abs/ 139.1999..
80
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05
56
0
0.1
0.3
0.4
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Syxtems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Percentage of mask-out tokens
Figure 3: The performance of S4 and S4- 24-lorler models on ECtHR datasets under different noise levels.
ECtHR task.
4.3
Robustness Analysis
Next, we analyze the robustness of SSMs to in- put noise. We conduct experiments the ECtHR dataset under different input noise rates p. More specifically, we randomly mask out p percent of inputs and train S4 and S4-pooler models with the noisy data. Figure 3 shows the results of two mod- els. We can see the performance of S4 drops much faster than S4-pooler model with the increasing percentage of noise. This is reasonable because the S4-pooler system has max-pooling layers and imposes stronger information extraction effect.
Iliss Chalkidis, Xiang Dai, Manos Fergadiotis, Pro- dromos Malakasiotis, and Desmond Elliott. 2022. An exploration of hierarchical attention transform- ers for efficient long document classification. CoRR, abs/2210.05529
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, and Ion Androutsopoulos. 2019, Large-scale multi-label text classification on EU legislation. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 63 14 6322, Florence, Italy. Association for Computational Linguistics.
5 Conclusion
In this work, we conduct comprehensive evalua- tion of the SSM-based models and show their su- periority over self-attention--asee models on long document classification tasks terms of perfor- mance and training time. We further propose an ef- ficient SSM-based system by imposing input length reduction for deeper layers and show our method performs on par with the original SSM models while greatly improving time efficiency.
Taas Chalkidis, Manos Fergadiotis, Dimitrios Tsarapat- sanis, Nikolaos Aletras, lon Androutsopoulos, and Prodromos Malakasiotis. 2021. Paragraph-Ievel ratio- nale extraction through regularization: A case study european court of human rights cases. In Proceed- ings of the Annual Conference of the North Ameri- can Chapter of the Association for Computational Linguisties. Mexico City, Mexico, Association for Computational Linguistics.
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction Algorithms, 3rd Edition. MIT Press
Limitations
The model we proposed only focuses on the clas- sification of long documents, therefore, it may be extended other NLP tasks, like long document Question Answering tasks. Furthermore, we con- centrated on English datasets during the evaluation. the future, we plan to extend it to more tasks and languages.
Xiang Dai, Ilias Chalkidis, Sune Darkner, and Desmond Elliott. 2022. Revisiting transformer- models for long document classification. Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 7212-7230. Association for Com- putational Linguistics.

---

Tri Dao, Daniel Y. Fu, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. 2022. Hun- gry hungry hippos: Towards language modeling with state space models. CoRR, abs/2212.14055
Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba. Yishay Carmiel and Najm Dehak. 2019. Hierar- chical transformers for long document classification. In IEEE Automatic Speech Recognition and Under- standing Workshop, ASRU 2019, Singapore. Decem- ber 14-18, 2019, pages 838 338-844. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLT 2019. Minneupolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4188 Association for Computational Linguisties.
Hyunji Hayley Park, Yogarshi Vyas, and Kashif Shah. 2022. Efficient classification of long documents us- ing transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), ACL 2022. Dublin, Ireland, May 22-27, 2022, pages 702-709, Associa- tion for Computational Linguistics.
Ofir Press, Noah A. Smith, and Mike Lewis. 2022. Train short, test long: Attention with lincar biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OvenReview.net.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré. 2022. It's raw! audio generation with state-space models. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Mu- chine Learning Research, pages 7616-7633. PMLR.
Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. 2023. Randomized positional encodings boost length generalization of transform- ers. CoRR, abs/2305. 16843.
Albert Gu, Karan Goel, and Christopher Ré. 2022. Effi- ciently modeling long sequences with structured state spaces. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 29-29, 2022. OpenReview.net.
Yi Tay, Mostafa Dehghami, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang. Sehastian Ruder, and Donald Metzler. 2021. Long range arena A benchmark for efficient transformers. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.net.
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. 2023. How to train your HIPPO: state space models with generalized orthogonal hasis projections. In The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 2017, Long Beach, CA, USA, pages 5998-6008,
Ruining He and Julian J. McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Pro- ceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 15, 2016, pages 507-517. ACM.
Sergey loffe and Christian Szegedy. 2015. Batch nor- malization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 JMLR Workshop and Conference Pro- ceedings, pages 448-456. JMLR.org.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed, 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David P. A. Corney, Benno Stein, and Martin Potthast. 2019. Semeval- 2019 task 4: Hyperpartisan news detection. In Pro- ceedings of the 13th International Workshop on Se- mantic Evaluation, SemEval @ NAACL-HLI HLLT 2019, Minneapolis, MN. USA, June 6-7, 2019, pages 829- 839. Association for Computational Linguistics.
A Dataset
The dataset statisties is shown in Table 3. # class is the number of labels, the max, min, mean indi- cates the maximum, minimum and mean number of tokens with BERT tokenizer, respectively.
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In Machine Learning, Proceedings of the Twelfth International Conference on Machine Learn- ing, Tahoe City, California, USA, July 921 1995, pages 331-339. Morgan Kaufmann.
B Experimental setup details
We used Adam optimizer with default |3 and we searched the best learning rate from 5e-5, 3e-5,

---

Table 3: Dataset statistics of LDU datasets.
Dataset
# class
max
train
dev
test
ECUR
33
46,712
9,000
1,000
1,000
Hyper
2
5.538
516
64
65
20News
20
30,602
10.182
1.132
7,532
EURLEX
4,721
4,443
45,000
6,000
6,000
BOOK
227
14,165
10,230
1,279
1,279
AMZ
5
18,282
3,880
485
485
le-5 for one run of each baseline model and se- lected the best learning rate for the model. For the Transformer and Longformer trained from scratch, we used the BERT-base and Longformer base and truncated them to be 6 layers without using the pre- trained checkpoints. For all methods, we set the maximum epoch to 20 and selected the best model based on the performance metric the dev set the checkpoint of the last epoch. We reported the average results on the test set over three different seeds. For Experiments with S4, we followed the recommended setting (Gu et al., 2022). For H3 models, we used H3-125M which has a similar model scale as BERT-base and Longformer-be models.


Reviews:
[{"review_id": "CTVrvsuiEP", "reviewer": "Reviewer_Y6PV", "summary": "", "questions": "1. I'd like to see more discussions about variations of the hierarchical mechanism, and the theoretical connection between it and s4 model.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R**:  It is not a surprise that the state space model works on more datasets. And simply evaluating it on more datasets can not be a significant contribution.\n\n>**A**:  The SSM was mainly evaluated on the Long Range Arena benchmark only consisting of one single text classification task. It is still unclear about the efficacy and efficiency of state-space models compared to attention-based models for long document tasks covering more domains and length varieties. Our work not only filled this gap but also provided a more efficient solution.\n\n**R**:  The proposed method improves efficiency at the cost of performance.\n\n>**A**: The training time reduction for long text classification is around 36% at the cost of only around 0.5% performance drop on average. Compared to the significant efficiency improvement, we think the performance drop is acceptable.\n\n **Q**:  I'd like to see more discussions about variations of the hierarchical mechanism, and the theoretical connection between it and s4 model.\n\n> **A**: First, the pooling layer down-samples the input gradually, hence reducing the computation cost. Moreover, it changes the resolution of the input sequence by choosing different window sizes on different layers. We conducted experiments to investigate the impact of the hierarchical structures of S4-pooler models. S4-pooler-x-y refers to placing the pooling layer on every x layer with window size y. The results are averaged over three random seeds. The performance of different tasks is influenced by the hierarchical structures. For example, enlarging the down-sampling ratio from 2 to 5 decreases the performance of AMZ, while having less impact on the EcTHR task. \n|     **Models**    | **AMZ**  | **Ecthr**|**Eurlex**| **Book** | \n| :------------------- | :----------: | :----------: | :----------: | :----------: |\n| S4            |   44.9   |   64.4   |   72.6   |   49.7   | \n| S4-pooler-1-2 |   44.2   |   64.2   |   72.0   |   48.0   |\n| S4-pooler-1-3 |   44.6   |   64.0   |   69.4   |   47.8   | \n| S4-pooler-1-5 |   43.7   |   64.2   |   71.8   |   47.4   |\n| S4-pooler-2-2 |   43.1   |   63.4   |   72.3   |   49.1   | \n| S4-pooler-2-3 |   44.2   |   63.1   |   71.5   |   50.3   |\n| S4-pooler-2-5 |   39.9   |   64.2   |   70.3   |   49.7   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes a method to improve time efficiency over the state space model.\n\n2. This paper evaluates the state space model on more datasets and investigates robustness.", "reasons_to_reject": "1. It is not a surprise that the state space model works on more datasets.  And simply evaluating it on more datasets can not be a significant contribution.\n\n2.  The proposed method improves efficiency at the cost of performance.", "questions_to_authors": "1. I'd like to see more discussions about variations of the hierarchical mechanism, and the theoretical connection between it and s4 model.", "paper_topic_and_main_contributions": "This paper proposes an empirical method that combines hierarchical pooling and the state space model.  This paper claims a contribution in that they first demonstrate the effectiveness of s4 on more text classification tasks. Another contribution it claims is the additional pooling layer, which improves time efficiency while harming performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MUNUBXAD4T", "reviewer": "Reviewer_FpWq", "summary": "", "questions": "Did the authors experiment with pooling techniques for BERT-based models, i.e., instead of truncating to 512 tokens, would it help if the embeddings for different chunks were aggregated in some way to get a single embedding for a given long document for a fair comparison?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R**:  It would help to state the intuition on why state-space models perform better than transformer based models.\n\n>**A**: Thanks for your suggestion, more discussion and missing references will be added in the final version. \n\n>Firstly, the computational complexity of the SSM algorithm is $O(NlogN)$ in convolution mode, which enables it to process significantly longer input sequences with the same hardware resources compared to the attention mechanism $O(N^2)$. This results in a substantial reduction in information loss from the input side.\n\n>Next, instead of relying on a limited number of global attention mechanisms to capture long-term dependencies in sparse attention models, the SSM model is designed to model long sequences as dynamic systems. By utilizing a Hurwitz state matrix A, it preserves long-term dependency information in a high-dimensional state. Such states enable SSM models strong ability to capture long dependency information. \n\n**R**: Did the authors experiment with pooling techniques for BERT-based models, i.e., instead of truncating to 512 tokens, would it help if the embeddings for different chunks were aggregated in some way to get a single embedding for a given long document for a fair comparison?\n\n>**A**: Thanks for your suggestion. We had such experimental comparisons and will make this part more clear in the final version. In Tab. 1, we compared our method to a Hierarchical Attention method (ToBERT) [1] for fair comparison. For this method, the long input is first split into several segments of less than 200 tokens. Next, each segment is encoded by a BERT model sequentially. The representation of the first token of each segment is then aggregated with a 2-layer transformer and the outputs of the transformer are fed to a mean pooling layer before the classification head.  \n\n>[1] Hierarchical Transformers for Long Document Classification. Pappagari et al. IEEE ASRU 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ Conduct extensive experiments on 6 long document classification benchmark\n+ Demonstrate that state-space models outperform transformer-based self-attention model\n+ The proposed method is more efficient and more robust to input noises\n", "reasons_to_reject": "- It would help to state the intuition on why state-space models perform better than transformer based models.\n- The work might be considered slightly incremental.", "questions_to_authors": "Did the authors experiment with pooling techniques for BERT-based models, i.e., instead of truncating to 512 tokens, would it help if the embeddings for different chunks were aggregated in some way to get a single embedding for a given long document for a fair comparison?\n", "paper_topic_and_main_contributions": "This is a short paper that introduces State-Space Models for Long Document Classification Task. The authors conduct extensive experimentation on 6 long document classification task and demonstrate that state-space models outperform transformer-based self-attention models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L097: \"more comprehensive analysis by...."}, "extra_text": {"Missing_References": "1.  Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork. 2019. Semantic\ntext matching for long-form documents. In Proceedings of the World Wide Web Conference. 795–806.\n2. Jha, Akshita, et al. Supervised contrastive learning for interpretable long-form document matching. ACM Transactions on Knowledge Discovery from Data 17.2 (2023): 1-17.\n3. Liu Yang, Mingyang Zhang, Cheng Li, Michael Bendersky, and Marc Najork. 2020. Beyond 512 tokens: Siamese multidepth transformer-based hierarchical encoder for long-form document matching. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management. 1725–1734."}}, {"review_id": "nYddG6rxtr", "reviewer": "Reviewer_VUdY", "summary": "", "questions": "- Why SSM-based models are more effective than self-attention-based models?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**R**: -   Why SSM-based models are more effective than self-attention-based models?\n\n>**A**: Due to the $O(N^2)$ time and space complexity, it is common to truncate the long sequential data for attention-based models, which is likely missing crucial information for addressing different tasks [1]. The complexity of SSM is $O(NlogN)$ in convolution mode, thus it is able to process much longer input given the same resources.  \n\n>Apart from that, instead of modeling the long dependency with a limited number of global attention in the sparse attention, the SSM model is designed to model long sequences as a dynamical system, it preserves long dependency information in a high-dimensional state by utilizing a structured Hurwitz state matrix $A$. This enables SSM models to effectively capture long-term dependency information. \n>[1]Revisiting Transformer-based Models for Long Document Classification. Dai et al. EMNLP22.\n\n**R**: More in-depth analysis is lacked.\n>**A**: Thanks for your suggestion. We conducted more experiments to study the influence of different pooling strategies. All results will be included in the final version. The pooling layer changes the resolution of the input sequence via different window sizes. We conducted experiments to investigate the impact of the hierarchical structure of S4-pooler models. S4-pooler-x-y refers to placing the pooling layer on every x layer with window size y. The performance of different tasks is impacted by the resolution of the input sequence. For example, enlarging the down-sampling ratio from 2 to 5 decreases the performance of AMZ, while having less influence on the EcTHR task.\n|     **Models**    | **AMZ**  | **Ecthr**|**Eurlex**| **Book** | \n| :------------------- | :----------: | :----------: | :----------: | :----------: |\n| S4            |   44.9   |   64.4   |   72.6   |   49.7   | \n| S4-pooler-1-2 |   44.2   |   64.2   |   72.0   |   48.0   |\n| S4-pooler-1-3 |   44.6   |   64.0   |   69.4   |   47.8   | \n| S4-pooler-1-5 |   43.7   |   64.2   |   71.8   |   47.4   |\n| S4-pooler-2-2 |   43.1   |   63.4   |   72.3   |   49.1   | \n| S4-pooler-2-3 |   44.2   |   63.1   |   71.5   |   50.3   |\n| S4-pooler-2-5 |   39.9   |   64.2   |   70.3   |   49.7   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper provide comprehensive evaluation of SSM-based models.  The experiments are conducted on self-attention-based models and SSM-based models on six commonly-used long text classification dataset with and without pre-training. Then the training efficiency and robustness analysis are also given.\n- The proposed method is more efficient and robust than current methods.\n- The authors shows the  superiority of SSM-based models over self-attention-based models.\n", "reasons_to_reject": "- Their proposed method is simple pooling operation and lack of novelty.\n- More in-depth analysis is lacked.", "questions_to_authors": "- Why SSM-based models are more effective than self-attention-based models?", "paper_topic_and_main_contributions": "This paper investigate State-Space Models for long document classification tasks. This paper conduct a comprehensive evaluation of the SSM-based models and show their superiority over self-attention-based models. The authors propose state space pooler to impose a progressive reduction of the input length for deeper layers.\n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper extends State-Space Models to many longer benchmarks, and proposed a slightly adjusted way to make it more efficient. It's a clearly written paper. Its main merit is to further reassure the efficacy of SSMs. 

pros

1. A solid justification for SSM to certain longer text documentation.

2. Well written and easy to follow.

cons

1. The proposed adjustment of model is not significant. None reviewer finds it novel enough such that the reviewer won't to champion it.

2. To make it a really comprehensive study-style paper on efficacy of SSMs on long document, the analysis is not enough and it lacks many other references to compare with.
Decision: Accept-Main