Once is Enough: A Light-Weight Cross-Attention for Fast Sentence Pair Modeling
Authors: Yuanhang Yang
Shiyi Qi
Chuanyi Liu
Qifan Wang
Cuiyun Gao
Zenglin Xu
Venue: EMNLP 2023 Main (2023)
Keywords: information retrieval
text matching
embedding

Abstract: Transformer-based models have achieved great success on sentence pair modeling tasks, such as answer selection and natural language inference (NLI). These models generally perform cross-attention over input pairs, leading to prohibitive computational cost. Recent studies propose dual-encoder and late interaction architectures for faster computation. However, the balance between the expressive of cross-attention and computation speedup still needs better coordinated. To this end, this paper introduces a novel paradigm TopicAns for efficient sentence pair modeling. TopicAns involves a lightweight cross-attention mechanism. It conducts query encoding only once while modeling the query-candidate interaction in parallel. Extensive experiments conducted on four tasks demonstrate that our TopicAnscan speed up sentence pairing by over 113x while achieving comparable performance as the more expensive cross-attention models.

Once is Enough: A Lightweight Cross-Attention for Fast Sentence Pair Modeling
Yuanhang Yang'
Shiyi Qi'
Chuanyi Liu'
Qifan Wang"
Cuiyun Gao" Zenglin Xu
Harbin Institute of Technology, Shenzhen, China
Meta AI, CA, USA
[ysngkil, syqi12138} @ @gmail.com liuchuanyi @ hit.edu.en wqfer@ (efirr@fb.com fb.com gaaccuiyun xuzenglin| |
@ @hi..edu.cn
Abstract
computation costs, especially when the number of candidates is very large e.g., the interaction will be calculated N times there are N candidates). This computation cost, therefore, restricts the use of these cross-encoder models in many real-world applications (Chen et al., 2020).
Transformer- based models have achieved great success sentence pair modeling tasks, such as answer selection and natural language infer- ence (NLI). These models generally perform cross-attention over input pairs, leading to pro- hibitive computational costs. Recent studies propose dual-encoder and late interaction ar- chitectures for faster computation. However, the balance between the expressive of cross- attention and computation speedup still needs better coordinated. To this end, this paper in- troduces a novel paradigm MixEncoder for ef- ficient sentence pair modeling. MixEncoder involves a lightweight cross-attention mecha- nism. It avoids the repeated encoding of the same query for different candidates, thus al- lowing modeling the guery-candidate interac- tion in parallel. Extensive experiments con- ducted on four tasks demonstrate that our Mix- Encoder can speed up sentence pairing by over I 113x while achieving comparable performance as the more expensive cross-attention mod- els. The source code is available https: //gi thub. com/ http//snnki//xxnoood.
To tackle this issue, propose lightweight cross-attention mechanism, called MixEncoder, that speeds up the inference while maintaining the expressiveness of cross-attention. Specifically, the proposed MixEncoder accelerates the cross- attention by performing attention only from candi- dates to the query, involving few tokens and only at a few layers. This lightweight cross-attention avoids repetitive query encoding, supporting the processing of multiple candidates in parallel and thus reducing computation costs. Additionally, MixEncoder allows pre-compute the candidates into several dense context embeddings and store them offline to accelcrate the inference further.
We evaluate MixEncoder for sentence pair mod- eling on four benchmark datasets related to tasks of natural language inference, dialogue, and infor- mation retrieval. The results demonstrate that Mix- Encoder better balances the effectiveness and effi- ciency. For example, MixEncoder achieves a sub- stantial speedup of more than 113x over the cross- encoder and provides competitive performance.
1 Introduction
Sentence pair modeling, such as natural language inference, question answering, and information retrieval, is an essential task in natural language processing (Nogueira and Cho, 2020; Qu et al., 2021: Zhao et al., 2021). These tasks can be de- picted as procedure of scoring the candidates given a query. Recently, Transformer-basel mod- els (Vaswani et al., 2017; Devlin et al., 2019) have shown promising performance on sentence pair modeling tasks due to the expressiveness of the pre-trained crosseencoder. As shown in Figure 1(a), the cross-encoder takes a pair of query and candi- date input and calculates the interaction between them at each layer by the input-wide self-attention mechanism. Despite the effective text representa- tion power, the cross-encoder leads to exhaustive
Background
Extensive studies, including dual-encoder (Reimers and Gurevych, 2019) and late interaction models (MacAvaney et al., 2020; Gao et al., 2020; Chen et al., 2020; Khattab and Zaharia, 2020), have been proposed to accelerate the transformer inference on sentence pair modeling tasks.
As shown in Figure 1, dual-encoders process the query and candidates separately, allowing pre- computing the candidates accelerate online in- ference, resulting in fast inference speed. However, this speedup is built upon sacrificing the expres-

---

2020) extract global features from output of the encoder by attention mechanism (C-strategy). The default configuration is S-strategy as provides slightly better performance. The pre-computed con- text embeddings E' E IR. xxxx are cached for on- line inference, where N is the number of candi- dates.
0.5 Sanyy
14 Crosnwnsoder
30 Duakensoder
LINK HnMOCSOO
3.1.1 Query Encoding
Figure 1: Illustration of three popular sentence pair approaches. where N' denotes the number of candidates and denotes the relevance score of candidate-query pairs. The cache stores the pre-computed embeddings.
Since the cross-encoder performs N times of query encoding, which contributes to the inefficiency, straightforward way to accelerate the inference is to reduce the encoding times of the query. Here we encode the query without taking its candidates into account, thus requiring the encoding only once.
Chocker
To preserve the expressiveness of the cross- attention, the simplified cross-attention is per- formed at several interaction layers. As shown in Figure 2, the context embeddings Ej-1 of can- didates are allowed attend over the intermedi- ale token embeddings of the query, thus obtaining context-awar representations Ej and Hj for the query and its candidates.
Qarry
M
Figure 2: Overview of proposed MixEncoder.
Concretely, at cach interaction layer, the key and value matrices of the query are utilized by candi- dates in two ways. (1) Producing contextualized representations for the candidates:
siveness of cross-attention (Luan et al., 2021; Hu et al., 2021; Zhang et al., 2021). Alternatively, late-interaction models adjust dual-encoders by ap- pending interaction component, such as a stack of Transformer layers (Cao al., 2020; Nie et al.. 2020), for modeling the interaction between the query and the cached candidates. These approaches still suffer from the high costs of the interaction component (Chen et al., 2020).
(1) where K", V' are derived from the Ej-1 with linear transformation. Fj is supposed to con- tain semantics from both the query and candidates. (2) Compressing the semantics of the query into vector for each candidate:
Ej Aum((' (K,
3 Method
(2) where Q' E IR- is derived from Ey-1 by a pool- ing operation, H RRW stands for the candidate- aware query states and Ho is initialized as a zero matrix.
H = (V,,
In this section, we introduce the details of the proposed MixEncoder, which simplifies cross- attention by enabling pre-computation, reducing the times of query encoding, and reducing the num- ber of involved tokens and layers.
3.2 Prediction
3.1 Candidate Pre-computation
Let H and F. denote the query states and the can- didate context embeddings generated by the last interaction layer, respectively. For the i-th can- didate, its representation is the mean of the i-th row of E, denoted as e,, The representation of the query with respect to this candidate the i-th row of H, denoted as h4. The cosine similarity between Gj and h; used as the semantic similarity. Addi- tionally, we can pass e; and h3 to a classifier for classification tasks.
Given a candidate that is a sequence of tokens T. = [t..... ,t/], we experiment with two strate- gies to encode these tokens into & context embed- dings in advance, where Ao << 1: (1) prepending k special tokens {5.) to T, before feeding T; into the Transformer encoder (Vaswani al., 2017; Devlin et al., 2019), and using the output at these special tokens as context embeddings (S-strategy); (2) maintaining à context codes (Humeau et al.,

---

Table Time Complexity of the attention module. We use g, to denote the query and candidate length, respectively. d indicates the hidden layer dimension, .V indicates the number of candidates for each query and &s indicates the number of context embeddings for each candidate.
Model
Total (N =
Pre eee--omuutaionn (N
Online
Dual-BERT
d(c² + q¹) (c+
de +++d' +
dg d'
Cross-
Cross-BERT BERT
d(c q)² + dle
N + c)² Ll c))
MixEncoder d(e + q1q k) k²) + +(+
de d'c
du + d'g + N(k: + didk
3.3 Time Complexity
training models the other three datasets, we use in-batch negatives (Karpukhin al., 2020; Qu et al., 2021). Detailed settings are provided in A.I.
Table presents the time complexity of the Dual- BERT, Cross-BERT, and our proposed MixEncoder. We can observe that MixEncoder supports offline pre-computation reduce the online time complex- ity. During the online inference, the query encod- ing cost term (dq² + q) of MixEncoder does not increase with the number of candidates since con- ducts query encoding only once. Moreover, Mix- Encoder's query -andddda term N(k + q + d)dk can be reduced by setting k as small value, which can further speed up the inference.
Results
Table 2 shows the experimental results of baselines and three variants of MixEncoder. We measure the inference time of all the baseline models for queries with 1000 candidates and report the speedup.
5.1 Performance Comparison
Variants of MixEncoder. To study the effect the number of interaction layers and that of the number of context embeddings per candidate, we consider three variants, denoted as MixEncoder-a, -b, and -c, respectively. Specifically, MixEncoder-a and -b set le as 1. The former performs interaction at the last layer and the latter performs interaction at the last three layers. MixEncoder-c similar to MixEncoder-b but with 2.
4 Experiments
Datasets. We evaluate MixEncoder three paired-input tasks over four datasets, including MNLI (Williams al., 2018) for natural language inference, MS MARCO passage reranking (Bajaj et al., 2018) for information retrieval, and DSTC7 (Yoshino et al., 2019), Ubuntu V2 (Lowe et al., 2015) for utterance selection for dialogue.
Dual-BERd and Cross-BERT The performance of the dual-BERT and cross-BERT reported in the first two rows of Table 2. We observe that MixEncoder consistently outperforms the Dual- BERT. The variants with more interaction layers or more context embeddings generally yield more im- provement. For example, on DSTC7, MixEncoder- and MixEncoder-b achieve an improvement by 0.7% (absolute) and 1.6% over the Dual-BERT, re- spectively. Moreover, MixEncoder-a provides com- parable performance to the CrRST both Ubuntu and DSTC7. MixEncoder-b can even out- perform the Cross-BERT on DSTC7 (+0.6), since MixEncoder can benefit from a large batch size (Humeau et al., 2020). However, the effectiveness of the MixEncoder on MS MARCO is slight.
Baselines. (1) Cross-BERT is the original BERT (Devlin al., 2019). (2) Dual-BERT
(Sentence-BERT) proposed by Reimers et al. (Reimers and Gurevych, 2019). (3) Deformer (Cao et al., 2020) is a decomposed Transformer that utilizes lower layers to encode sentences separately and then uses upper layers to encode text pairs together. (4) Poly-Encoder (Humeau al., 2020) encodes the query and its candidates separately and performs light-weight late interaction. (5) ColBERT (Khattab and Zaharia, 2020) is late interaction model which adopts the MaxSim operation to obtain relevance scores. This operation prohibits the utilization of ColBERT on classification tasks. (6) VIRT (Li et al., 2022) performs the cross- crossaattention at the last layer and utilizes knowledge distillation during training.
We find that the difference in the inference time between the DuaT-BERT and MixEncoder is minimal, while Cross-BERT is orders of magnitude slower than these models.
Training Details. While training models on MNLI, we use the labels provided in the dataset. While

---

Table 2: Performance
Cross-BERT and three variants of MixEncoder on four datasets.
MNLI
Ubuntu
DSTC7
MS MARCO
Specdup Times
Space
Model
Accuracy
RI@ I0
MRR
R1@ 100
MRR
R 1000
MRR(dev)
GB
Cross-BERT
8..
la.7
89.4g.5
66.8u.
75.2
23.3
36.0
1.0x
Dual-BERT
75.
81.60.2
88.50.1
65.81.
74.20.7
20.3
32.2
132x
0.3
PoloEneeder-64
86.
82.30.5
88.96.4
66.41.5
74.80.9
20.3
32.3
130x
0.3
PolyEncoder-360
77.30.2
$1.8g.2
88.60.1
65.70.6
74.00.3
20.3
32.4
127x
0.3
ColBERT
x
82.9h.3
89.30.2
67.20.7
74.80.4
22.8
35.4
35.2x
8.6
VIRT
78.30.3
83. 10.2
89.40.2
66.50.7
74.90.2
21.5
32.3
33.3x
52.7
Deformer
82.00.1
83.20.4
89.5p.2
66.31.0
75.30.6
23.0
35.7
1.9x
52.7
MixEncoder-a
77.5n.4
83. 83.10.1
89.4g.
66.9u.s
74.90.2
20.4
32.0
113x
0.3
MixEncoder-b
77.80.2
83.2o.0
89.50.1
68. 28.2
75.80.5
20.7
32.5
89.6x
0.3
MixEncoder-c
78.40.4
83.30.1
89.5p.n
66.
74.8p.3
20.0
84.8x
Table 3: Ablation analysis for MixEncoder-a and -b.
Murtt
06/07
Ubuntu
DTSC7
Ubuue
DSTC7
Variants
-a
-b
-a
-b
15.5
89.0
Original
89.5
89.5
74.9
76.1
75.0
88.8
w/o H
88.9
89.1
74.0
73.9
10
1i
w/o E
89.2
89.3
74.8
75.2
a) Performing simplified cresi-attention the x-th loyers and above.
76.0
Late Interaction Models. From Table have the following observations. First, among all the late interaction models, Deformer that adopts a stack of Transformer layers as the late interaction compo- nent consistently shows the best performance on all the datasets. This demonstrates the effectiveness of cross-attention. In exchange, Deformer shows limited speedup (1.9x). Compared to the ColBERT and Poly- -ocoder, MixEncoder outperforms them on the datasets except for MS MARCO. Although CoIBERT consumes more computation than Mix- Encoder, shows worse performance than MixEn- coder on DSTC7 and Ubuntu. This demonstrates that the lightweight attention can achieve a better trade-off between efficiency and effective- ness. However, MS MARCO, MixEncoder and poly-encoder lag behind the COlBERT by a large margin. We conjecture that MixEncoder falls short handling term-level matching. We will elaborate on it in section A.4.
77.0
75.0
DSTC3
77.2
77.0
bl Model with a varing number of context vectors and different strategies.
Figure 3: Parameter analysis on the interaction layers and pre-computed context embeddings.
Interaction layers. Figure 3(a) shows the results when MixEncoder performs interaction at Transformer layers upper than r. Increasing interaction layers cannot continuously improve the ranking quality. On both Ubuntu and DSTC7, the performance of MixEncoder achieves peak with the last three layers utilized for interaction. More experiments are reported section A.6.
Context embeddings, We study the effect of the number of candidate embeddings and the pre- computation strategies with the last layer per- form the simplified cross-ane attention. From Figure 3(b), it observed that the S-strategy generally outperforms the C-strategy, and larger can lead to a better performance for the S-strategy.
5.2 Ablation Study
Representations. We conduct ablation studies to quantify the impact of two key components (E and H) utilized in MixEncoder. The results are shown gain in
in Table 3. All components contribute performance. It demonstrates that the simplified cross-attention can produce effective representa- tions for both the query and its candidates.
Table 4 shows the average time per example for different models. It is shown that MixEncoder con- sumes more time
increases. Nevertheless, the difference in timing between Dual-BERT and Mix- Encoder is rather minimal, whereas Cross-BERG

---

is significantly slower by two orders of magnitude.
References
Payal Bajaj, Daniel Campos. Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder. An- drew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song. Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: human gener- ated machine reading comprehension dataset.
Table 4: Query processing times with 1,000 candidates and the last layer utilizing simplified cross-attention.
Model
Time (ms)
Dual-BERT
7.2
Cross-BERT
949.4
MixEncoder (k=1)
8.4
Qingqing Cao, Harsh Trivedi, Aruna Balasubrama- nian, and Niranjan Balasubramanian. 2020. De- Former: Decomposing pre-trained transformers for faster question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4487-4497 Online. Association for Computational Linguisties.
MixEncoder (k=2)
9.1
MixEncoder (k=3)
10.0
MixEncoder (k=4)
11.5
MixEncoder (k=10)
24.3
Jiecao Chen, Liu Yang. Karthik Raman, Michael Ben- dersky, Jung-Jung Ych, Yun Zhou, Mare Najork, Danyang Cai, and Ehsan Emadzadch. 2020. DiPair: Fast and accurate distillation for trillion-scale text matching and pair modeling. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020. pages 2925-2937, Online. Association for Computa- tional Linguistics.
6 Conclusion
In this paper, we propose MixEncoder to balance the trade-off between performance and efficiency. It involves a lightweight cross-attention mechanism that allows us to encode the query once and pro- cess all the candidates in parallel. Experimental results demonstrate that MixEncoder can speed up sentence pairing by over 1 13x while achieving comparable performance as the more expensive cross-attention models.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
7 Acknowledgements
This work was partially supported by the Na- tional Key Research and Development Program of China (No. 2018AAA0100204), a key pro- gram of fundamental research from Shenzhen Science and Technology Innovation Commis- sion (No. JCYJ202001091 13403826), the Ma- jor Key Project of PCL (No. 20222D0115301),
Luyu Gao, Zhuyun Dai, and Jamie Callan. 2020. Mod- ularized transfomer-based ranking framework. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 4180-4199, Online. Association for Computa- tional Linguistics.
and an Open Research Project of Zhejiang Lab (N0.2022RCOAB04).
7he Hu, Zuohui Fu, Yu Yin, and Gerard de Melo. 2021. Context-aware interaction network for ques- tion matching. In Proceedings of the 2021 Confer- ence Empiricul Methods in Natural Language Pro- cessing, pages 3846-3853, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Limitations
Although MixEncoder has been demonstrated to be effective in cross-attention computation, we rec- ognize that MixEncoder does not perform well on MS MARCO. It indicates that our MixEncoder falls short of detecting token overlapping since it loses level features by pre-encode candidates into several context embeddings. Moreover, Mix- Encoder is not evaluated tion dataset, such as an end-to-end retrieval task, which requires the model to retrieve top-k candi- dates from millions of candidates (Qu et al., 2021; Khattab and Zaharia, 2020).
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architec- tures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference Learning Representations.
large-scale evalua-
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 7769-6781, Online. Association for Computational Linguistics.

---

Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Reirieval, page 39-48. Association for Computing Machinery.
3982-3992, Hong Kong, China, Association for Com- putational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Dan Li, Yang Yang, Hongyin Tang, Jiahao Liu, Qifan Wang, Jingang Wang, Tong Xu, Wei Wu, and Enhong Chen. 2022. VIRT: Improving representation- based text matching via virtual interaction. I Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 124, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
challenge corpus for sen-
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015, The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294, Prague, Czech Repub- lic. Association for Computational Linguistics.
Koichiro Yoshino, Chiori Hori, Julien Perez, Luis Fer- nando D'Haro, Lazaros Polymenakos, Chulaka Gu- nasekara, Walter Lasecki, Jonathan K. Kummer- feld, Michel Galley, Chris Brockett, Jianfeng Gao, Bill Dolan, Xiang Gao, Huda Alamari, Tim K. Marks, Devi Parikh, and Dhruv Batra. 2019. Dialog system technology challenge 7.
Yi Luan, Jacob Eisenstein, Kristina Toutanova. and Michael Collins. 2021. Sparse, dense, and attentional representations for text retrieval. Transactions of the Association for Computational Linguistics, 9:329 345.
Linhao Zhang, Dehong Ma. Sujian Li, and Houfeng Wang. 2021. Do it once: An embarrassingly sim- ple joint matching approach to response selection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4872-4877, Online. Association for Computational Linguistics.
Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Efficient document re- re-ranking for transformers by precomputing term representations. In Proceedings of the 43rd International ACM SI- GIR Conference on Research and Development in Information Retrieval, page 49-58. Association for Computing Machinery.
Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2021, SPARTA: Efficient open-domain question answering via sparse transformer matching retrieval. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 565-575, Online, Association for Computational Lin- guistics.
Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. 2020. De-bert: Decou- pling question and document for efficient contextual encoding. In Proceedings of the 43rd International ACM SIGIR Conference Research and Develop- ment in Information Retrieval, page 1829-1832. As- sociation for Computing Machinery.
Rodrigo Nogueira and Kyunghyun Cho. 2020. Passage re-ranking with bert.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. In Proceedings of the 202/ Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835-5847, On- line. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages

---

A More Details
A.1 Training Details
Ubuntu
180
For Cross-BERT and Deformer, which require ex- haustive computation, we set the batch size as 16 due to the limitation of computation resources, For other models, set the batch size 64. All the models use BERT (based, uncased) with 12 layers and fine-tune it for up to epochs with a learning rate of lc-5 and linear scheduling. All experiments are conducted on server with Nvidia Tesla A 100 GPUs, which have 40 GB graphic memory.
70.0
88.5
68.0
Batchh Size
Figure 4: Parameter analysis on the batch size.
A.5 Inference Speed
We conduct speed experiments to measure the on- line inference speed for all the baselines. Con- cretely, we sample 100 samples from MS MARCO. Each of the samples has roughly 1000 candidates. We measure the time for computations on the GPU and exclude time for text reprocessing and moving data the GPU.
A.2 Datasets
The statistics of datasets are detailed in Table 5. We accuracy evaluate the classification per-
formance MNLI. For other datasets, MRR and recall are used as evaluation metrics.
Table Time evaluate 100 queries with can- didates. The Space used cache the pre-computed embeddings for Ik candidates are shown.
Table 5: Statistics of experimental datasets.
MNLI
MS MACRO
DSTC7
Ubaam
*c quecies
392702
198.70
200.90
500000
Time (ms)
Space (GB)
Train
Ang length of queries
153
139
Model
condicbtees
1k
lk
of queries
Ff candibates per query
9,706
6,808
1006
1,00
100
50,000
10
Dual-BERT
7.2
0.3
PolyEncoder-64
7.3
0.3
Arg length of candidates
PolyEncoder-360
7.5
0.3
ColBERT
27.0
8.6
A.3 In-batch Negative Training
Deformer
488.7
52.7
Cross-BERT
949.4
We change the batch size and show the results in Figure 4. It can be observed that increasing batch size contributes better performance. Moreover, we have the observation that models may fail to di- verge with small batch sizes. Due to the limitation of computation resources, we set the batch size as 64 for our training.
MixEncoder-a
8.4
0.3
MixEncoder-b
10.6
0.3
MixEncoder-c
11.2
0.6
A.6 Interaction Layers
From Table 7, it observed that performing cross- attention at higher layers generally yields better performance. Since we use the output of the fi- nal interaction layers as the sentence embeddings, choosing low layers enables the carly exit mecha- nism.
A.4 Error Analysis
In this section, take sample from MS MARCO analyze our errors. We observe that MixEncoder falls short of detecting token overlapping. Given the query "foods and supplements to lower blood sugar", MixEncoder fails to pay attention to the keyword "supplements." which appears in both the query and the positive candidate. We conjecture that this drawback is due to the pre-computation that represents each candidate into & context em- beddings. It loses the token-level features of the candidates. On the contrary, ColBERT caches all the token embeddings the candidates and esti- mates relevance scores based on token-level simi- larity.
Table 7: Results (Recall @ of performing simplified cross-attention at two interaction layers on DSTC.
Layer
12
10
8
2
65.4
64.8
64.3
66.4
65.7
66.2
67.1
65.5
66.0
66.6
65.4
10
67.4


Reviews:
[{"review_id": "bMLN3nNa0Z", "reviewer": "Reviewer_7S85", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.\n\n**Re: \"The performance of the model is unstable.\"**\n\nIn our experiments, we have explored three variants of MixEncoder, namely MixEncoder-a, -b, and -c.  Our experiments on the MS MARCO dataset show that both MixEncoder-a and MixEncoder-b exhibit slight improvements over Dual-BERT in terms of R1@1000,  while MixEncoder--c lags behind Dual-BERT. We have discussed this observation in section A.5 and have provided a potential explanation for these results.\n\nIt is also worth mentioning that the baselines considered in our study have not been collectively evaluated on all four datasets (MNLI, Ubuntu, DSTC7, MS MARCO) in prior research. Specifically, Poly-encoder is designed for the Ubuntu and DSTC7 datasets, ColBERT is proposed for MS MARCO, and Deformer is evaluated on MNLI and other NLU tasks.  Given the overall effectiveness of MixEncoder across these four tasks, we believe that MixEncoder represents a general strategy for representation-based matching tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of this work is simple but effective.", "reasons_to_reject": "1. The performance of the model is unstable. On some datasets such as MNLI, Ubuntu, and DSTC7, this work achieves good performance. However, on MS MARCO, it's performance even weaker than the Dual-BERT.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, authors propose a lightweight cross-attention mechanism for sentence matching to preserve the performance and the inference speed of the model simultaneously. Firstly, it attempts to encode candidates into k (k<<l) vectors. When inference, this paper integrates a cross-attention module into some intermediate layers. Experiments on four datasets prove the advantage of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fr9tb8HSuX", "reviewer": "Reviewer_TknS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments.\n\n**Re: \"The time cost with different k should be presented\"**\n\nThank you for your suggestion. Following this, we have added experiments to discuss the time cost with different k. It is shown that MixEncoder consumes more time as k increases. Nevertheless, the difference in timing between Dual-BERT and MixEncoder is rather minimal, whereas Cross-BERT is significantly slower by two orders of magnitude. In our revised version, we will provide additional analysis to elaborate on these findings in Section 5.2 (Ablation Study).\n\nTable: Time to evaluate 100 queries with 1k candidates with the last layer performing the simplified cross-attention.\n\n|       Model       | Time(ms) |\n| :---------------: | :------: |\n|     Dual-BERT     |   7.2    |\n|    Cross-BERT     |  949.4   |\n| MixEncoder (k=1)  |   8.4    |\n| MixEncoder (k=2)  |   9.1    |\n| MixEncoder (k=3)  |   10.0   |\n| MixEncoder (k=4)  |   11.5   |\n| MixEncoder (k=10) |   24.3   |\n\n**Re: \"The methods of comparison are a bit outdated. Please include some recent works.\"**\n\nThank you for your valuable suggestions. We only considered the methods that improve Dual-BERT by modifying the model architecture before. Following your suggestion, we have added one orthogonal method VIRT [1] which leverages knowledge distillation to improve representation-based text matching. The table below briefly reports the results of this method in the first row. In the revised version, we plan to extend Table 2 within our paper and provide a more comprehensive analysis in Section 5.1.\n\n| Model        | MNLI (acc)   | Ubuntu (MRR) | DSTC7 (MRR)  | MS MARCO (MRR) | Speedup |\n| ------------ | ------------ | ------------ | ------------ | -------------- | ------- |\n| VIRT         | $78.3_{0.3}$ | $89.4_{0.2}$ | $74.8_{0.4}$ | $32.3$         | 40.5x   |\n| MixEncoder-a | $77.5_{0.4}$ | $89.4_{0.1}$ | $74.9_{0.2}$ | $32.0$         | 113x    |\n| MixEncoder-b | $77.8_{0.2}$ | $89.5_{0.1}$ | $75.8_{0.5}$ | $32.5$         | 89.6x   |\n\n[1] [VIRT: Improving Representation-based Text Matching via Virtual Interaction](https://aclanthology.org/2022.emnlp-main.59) (Li et al., EMNLP 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis work is simple but effective. The paper is easy to follow, and the contributions are clear.\n2.\tThe experimental result on multiple datasets shows that the method is both efficient in time and space. \n3.\tThe analysis of strength and limitations is convincing.\n", "reasons_to_reject": "1.\tThe time cost with different k should be presented, as “a larger k can lead to a better performance for the S-strategy.”\n2.\tThe methods of comparison are a bit outdated. Please include some recent works.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about accelerating cross-attention mechanism in the scenario of sentence pairing. The main contribution is to propose a lightweight paradigm MixEncoder to speed up by over 113x and maintaining good performance for sentence pairing.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4zI3btjp6U", "reviewer": "Reviewer_go6q", "summary": "", "questions": "MixEncoder only considers BERT-base and does not do experiments on large language models to verify its effectiveness. How does MixEncoder perform on large language models?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful comments\n\n**Re: \"MixEncoder only considers BERT-base and does not do experiments on large language models to verify its effectiveness.\"**\n\nThank you for this feedback. Following [1] , our research primarily focuses on the application of BERT-base in our experiments. We agree that experiments on large language models may benefit our work. To this end, we consider a larger variant: BERT-large. Due to time constraints, the additional experiments are conducted on DSTC7. The table below reports the results. It is found that MixEncoder continues to enhance the performance of Dual-BERT even after the model size is scaled up. We plan to incorporate this table into the appendix for further reference.\n\n| Method       | BERT-base    | BERT-large   |\n| ------------ | ------------ | ------------ |\n| Cross-BERT   | $75.2_{0.5}$ | $77.0_{0.2}$ |\n| Dual-BERT    | $74.2_{0.5}$ | $75.9_{0.4}$ |\n| MixEncoder-a | $74.9_{0.2}$ | $76.3_{0.4}$ |\n| MixEncoder-b | $75.8_{0.5}$ | $76.5_{0.3}$ |\n\n[1] Humeau, S., Shuster, K., Lachaux, M., & Weston, J. (2019). Poly-encoders: Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring. *International Conference on Learning Representations*.\n\n\n\n**Re: \"The references need to be cleaned up.\"**\n\nWe appreciate your valuable suggestions. We will clean up the references to enhance the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes MixEncoder, which simplifies cross-attention by enabling pre-computation, reducing the times of query encoding, and reducing the number of involved tokens and layers.\n2. This paper conducts lots of experiments.", "reasons_to_reject": "1. MixEncoder only considers BERT-base and does not do experiments on large language models to verify its effectiveness.\n2. The references need to be cleaned up.", "questions_to_authors": "MixEncoder only considers BERT-base and does not do experiments on large language models to verify its effectiveness. How does MixEncoder perform on large language models?", "paper_topic_and_main_contributions": "In this paper, the authors propose MixEncoder to balance the trade-off between performance and efficiency. MixEncoder uses a lightweight cross-attention mechanism to encode the query once and process all the candidates in parallel. Experimental results demonstrate that MixEncoder can speed up sentence pairing by over 113x while achieving comparable performance as the more expensive cross-attention models.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers have agreed that the idea in this paper is simple yet effective.

However, all the reviewers have concerns on whether the results are convincing enough or not. During the rebuttal period, the authors have addressed some of these concerns.

Generally, the merits of this paper outweigh its flaws.
Decision: Accept-Main