Cross-Probe BERT for Efficient and Effective Cross-Modal Search
Authors: TAN YU
Hongliang Fei
Ping Li
Venue:  (2021)

Abstract: Inspired by the great success of BERT in NLP tasks, many text-vision BERT models emerged recently. Benefited from cross-modal attention,  text-vision BERT models have achieved excellent performance in many language-vision tasks including text-image retrieval.   Nevertheless,  cross-modal attentions used in text-vision BERT models require too expensive computation cost when solving text-vision retrieval, which is impractical for large-scale search. In this work, we develop a novel architecture, Cross-Probe BERT.  It relies on devised text and vision probes, and cross-modal attentions are conducted on text and vision probes.  It takes lightweight computation cost, and meanwhile effectively exploits cross-modal attention.  Systematic experiments conducted on two public benchmarks demonstrate state-of-the-art effectiveness and efficiency of the proposed method.

Under review as a conference paper at ICLR 2021
CROSS-PROBE BERT FOR EFFICIENT AND EFFECTIVE CROSS-MODAL SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Inspired by the great success of BERT NLP tasks, many text-vision BERT mod- els emerged recently. Benefited from cross-modal attentions, text-vision BERT models have achieved excellent performance in many language-vision tasks in- cluding text-image retrieval. Nevertheless, cross-modal attentions used in text- vision BERT models require too expensive computation cost when solving text- vision retrieval, which is impractical for large-scale search. In this work, we de- velop a novel architecture, cross-probe BERT. It relies on devised text and vision probes, and the cross-modal attentions are conducted on text and vision probes. It takes lightweight computation cost, and meanwhile effectively exploits cross- modal attention. Systematic experiments conducted on two public benchmarks demonstrate state-of-the-art effectiveness and efficiency of the proposed method.
INTRODUCTION
Tradition text-to-image retrieval tasks are tackled by joint embedding methods. They map the text queries and reference images into the same feature space that queries and images can be compared directly. Basically, they adopt a two-tower architecture, in which one tower extracts features of text queries, and the other tower extracts features of reference images. In the training phase, the parameters of two towers are optimized so that query and its relevant images are close in the feature space whereas the distance between the query and its irrelevant images is large. Since two towers independently generate image and query features, image features can be extracted offline and cached in the database. In the search phase, the cached image features can be directly compared with the query's 's feature, and the retrieval is efficient. Due to high efficiency, joint embedding methods based on the two-tower structure have been widely used many large-scale cross-modal retrieval.
Inspired by great success achieved by self-attention mechanism of Transformer (Vaswani et al. (017) and BERT (Devlin et al.] (2019) in NLP tasks, several text-vision BERT models (Lu et al. 2019 Li et al. (2020)) emerge. They take the query-image pair as input and extend the original text-modal self-attention to the multi-modal self-attention. The text-vision BERT effectively models the interactions between image features and query features, provides contextual encoding for both image features and query features, and achieves a significantly better retrieval accuracy compared with its two-tower counterpart. Despite the high effectiveness achieved by text-vision BERT, the extremely high computation cost brought by pairwise input limits its practical usefulness, especially for large-scale cross-modal retrieval in industrial applications. Given query and N reference im- ages, it needs to feed N query-image pairs to text-vision BERT for N relevance scores, That is, it requires to repeatedly encode the query for N times. In a large-scale cross-modal retrieval task, N is extremely large, making text-vision BERT prohibitively slow for obtaining relevant scores with all reference images. In contrast, a two-tower encoder only needs to encode the query for one time, and N reference image features can be pre-computed offline and cached in the database. Thus, it obtains relevance scores between the query and reference images in a very efficient manner by computing the cosine similarities between the query feature and pre-computed reference image features.
Though the inefficient pairwise attention limits the usefulness of the text-vision BERT large-scale cross-modal retrieval, there are few works to speed up the text-vision BERT. In fact, the inefficiency caused by pairwise input is general problem which is also encountered in other retrieval tasks such -oceumet retrieval (Humeau et al. (2020)). question answering (Cao al.) (2020): Zhang al. (2020)). In these tasks, similarly there are two mainstream encoders for obtaining the relevance score. The first type of encoder, Bi-encoder (Dinan et al. (2019) Mazare al. (018 )) is based

---

Under review as a conference paper at ICLR 2021
the two-tower architecture. Since the query/question and document are independently encoded, the document features can be pre-computed and cached. In this case, the relevance between the query and each document can be determined by the cosine similarity between the querylquestion's 's feature and the document's cached feature. It achieves a high efficiency but a relatively low retrieval accuracy. In contrast, Cross-encoder (Urbanek et al.) takes a question-answe document pair as input, exploiting the cross-attention like text-vision BERT and achieving high retrieval accuracy but is inefficient. To balance effectiveness and efficiency, existing methods (Cao) et al. (2020); Zhang et al. (2020)) adopt the two-tower architecture in lower layers and use the cross- attention architecture in the upper layers. We term this architecture as that case, features from lower two-tower layers can be extracted offline and cached. Then question- answer or query-document attentions can be conducted the upper layers. Since the number of upper cross-attention layers is small, the efficiency is boosted. Similarly, Poly-Encoder (Humeau jet al. (2020) conducts the two-tower architecture for feature extraction, and uses an additional cross-attention layer on the top to obtain the similarities between the query and reference items.
pair a query-
encoder. In
In this paper, we propose a novel architecture, cross-probe (CP) BERT for effective and efficient cross-modal retrieval. Motivated by the great success of the "split-merge" style encoder in query document retrieval, we extend text-vision BERT to adopt it for speeding up the computation. In particular, we devise several vision probes and text probes along with the image's local features and the query's 's word features. In the lower a few layers, we adopt the two-tower architecture. The vision probes and the image's 's local features are concatenated and fed into the vision tower and generate the attended vision probes. In parallel, the text probes and the query's 's word features are concatenated and fed into the text tower, and generates the attended text probes. After that, the attended vision probes and text probes are concatenated and fed into a series of rross-atteniion layers to exploit the cross-modal attentions. Since the number of text probes considerably smaller than the number of words in the query and the number of vision probes is smaller than the number of local features of the image, the cost of our CP BERT in computing cross- moosl attention is significantly less than that of text-vision BERT. Mcanwhile, the cross-modal attention is only exploited in the upper few layers, making our CP BERT more efficient. Systematic experiments conducted on two public benchmarks demonstrate the excellent effectiveness and efficiency of the proposed CP BERT.
2 RELATED WORK
Traditional cross-modal retrieval, e.g, text-image retrieval, relies on joint embedding. It maps texts and images from two modalities into a common feature space through two encoders. Then texts and images can be compared and the distance between their global features in the common feature space measures their similarities. Early joint embedding methods (Gong et al. (2012); Rasiwasia et al. (2100) utilize canonical correlation analysis (CCA) to project hand-crafted text and image features to aoint CCA space. Recently, inspired by great progress achieved by deep neural network, methods based on deep learning emerge. VSE++ (Faghri et al.] (2017)) obtains the image feature through a convolution neural network (CNN) and encodes the text by a gated recurrent unit (GRU). The CNN and the GRU are trained in an end-to-end manner by the designed triplet loss. The triplet loss seeks to minimize the distance between the relevance text-image pairs and maximize the distance between irrelevant pairs. The merit of joint-embedding methods is simplicity and efficiency. In this case, a text query as well the text can be efficiently obtained by computing the cosine similarity of their features. Meanwhile, the global text and image features make it feasible for the approximate nearest neighborhood (ANN) search such as Hashing and inverted indexing, so that the large-scale retrieval can be efficient.
image is represented by a global feature. The relevance between the image and
Nevertheless, the global feature used in joint-embedding methods has limitations. In many cases, the relevance between a text and an image is determined by very few words in the text and some small regions in the image. Therefore. the relevant text words and image regions might be distracted by irrelevant words and regions when using global features. Thus methods based on local features are proposed to overcome the limitations of global features used in joint-embedding methods. In DVS (Karpathy & Fei-Fei (2014)), an image is represented by set of bounding box features extracted from the object detector, R-CNN. Meanwhile, the text is represented by a sequence of word features extracted from an RNN. Then the bounding box features and word features are aligned to obtain the similarity between the image and the text. The aligmment operation can effectively alleviate the distraction from irrelevant word-region pairs. Similarly, SCAN (Lee et al. (2018)) relies on bounding box features from a faster R-CNN and word features from a GRU. It conducts the alignment through soft-attention and optimizes the loss function through hard negative mining. Nevertheless, both DVS
2

---

Under review as a conference paper at ICLR 2021
and SCAN conduct the alignment in the late stage, which limits their effectiveness due to lack of thorough interactions between bounding box features of the image and word features of the query.
To conduct the interactions between bounding box features and word feature thoroughly, text-vision
BERT methods are proposed. They can be categorized into two types; one-stream (Li ct al. 2019)
2020); Chen al. 20200) structure and two-stream (Lu et al. (2019); Tan & Bansal (2199) struc-
ture. A one-stream structure concatenates word features and bounding box features into a long sequence, and conducts a series of self-attention operations for cross-modal interactions, In parallel, a two-stream structure designs IWo streams for word features and bounding box features, respec- tively. In the text stream, word features are attended by bounding box features, and in the vision stream, bounding box features are attended by word features. Benefited from thorough interactions between bounding box features and word features, text- vision BERT achieves state-of-the-ard perfor- mance in many vision-language tasks. Nevertheless, text-vision BERT requires extremely expensive computational cost, limiting its usefulness in large-scale cross- moss- retrieval applications.
3 METHOD
Following existing text-vision BERT models (Lu et al. (2019) Li et al. (2020)), for each reference image set of bounding box features is extracted through faster R-CNN (Ren al. (0155) pre- trained on Visual Genome KKrishna et al. (2017). The set of bounding box features, B b; ) the initial representation of layer to generate word features, W= = w which is the initial representation of The ranking model (RM) takes B and W as input, and generates similarity score between q and I:
parallel, the query sentence 9, goes through a word-embedding
s(q.1) RM W).
In the search phase, the similarity scores s(g, Ii), ... ,s(, IN) are used for ranking images. In the training phase, the similarity scores are used for constructing the triplet loss with the margin m:
L = (s T-) s(g, [+) + m]+
Below, we introduce the architectures of existing ranking models and propose our cross-probe BERT.
3.1 TWO-TOWER BERT
The two-tower BERT (BERITT), visualized in Figure aa), models the image and the query sepa- rately, generates the query's feature W and the reference image's feature B through two BERTs:
W BERT B BERTv (B).
The text query feature q is set as WI, the attended feature of the first token in W. The image feature is obtained by sum-pooling the attended bounding box features in B by I = -Eee b. The similarity score between the query and the image is obtained by the cosine similarity between I and q. In two-tower BERT model, the query feature is independent with reference image features. This independent property makes two-tower BERT model quite suitable for large-scale cross- modal retrieval. Let us denote the computation cost of BERT per query by and denote that of BERT per reference image by cv- Given query and K, reference images, the total complexity cost in feature extraction to obtain their similarity scores is CTT eT + Kιev. Since features of Ki reference images can be pre-computed in the offline phase, the factual complexity cost the online search phase, ÖTT. is only taken on the query-side feature extraction:
ÖTT CT-
Later, we will show that the online computational cost BERT ČrT, significantly lower than that used in text-vision BERT, making it widely used in large-scale cross moosl retrieval.
3.2 TEXT-VISION BERT
Existing text-vision BERT (BERTIv) models can be categorized into two groups: single-stream architecture and two-stream architecture. Since their performance is comparable, we only introduce the single-strean architecture here. BERTTv. visualized in Figure [ (b), concatenates the initial word features of the query and the initial bounding box features of the reference image as the input,
3

---

Under review as a conference paper at ICLR 2021
Score
Score
Wax
b,
Dyy
Mr
Wg
WPr
B
transformer layers
transformer layers
transformer layers
D
Dg
Bx
Wir
(a) BE
(b) BERTTv
Figure Architecture of two-tower BERT (BERT TT) and vision-text BERT (BERT TV).
Score
Attention
Attention
Attention
Score
transformer layers
R
Waarr
bg
Dy
W
B2
B
Dy
L-1 transformer layers
I-f transformer layers
transformer layers
transformer layers
W
M2
War
D
bx
W2
Wwr
by
BERTTMM SM
(b) Poly-Encoder
Figure Architecture of split-merge BERT (BERTsM) and Poly-Encoder.
M M [W, modal attentions obtained through BERT consisting of series of Transformer blocks:
Then the attended word features and attended bounding box features based on cross-
W B] (ER,
Then the global feature of the query and that of the image are obtained the same manner as BERTTT. The relevance between 9 and 7 is the cosine similarity between their global features.
Different from BERTTT which only exploits the intra- modal attentions, BERTTV additionally ex- ploits the cross modal attentions. At this point, the image feature is adaptively tuned based on the query, and the query feature refined by the image as well. Benefited from the query-adapt image features and image-adapt query features brought by cross-modal attentions, BERTTV achieves significantly higher retrieval accuracy than BERTTT- On other hand, the query-adapt and image- adpat mechanism requires to recompute the reference image's 's representation for different queries and meanwhile need recompute the query's 's representation for different images, making BERTTV much more computationally expensive than its counterpart, BERTTT. To be specific, given query and Kj images, to obtain their similarities, we need feed K2 text-image pairs to BERT Let us denote the computation cost per text-image pair BERTTV by CTV, the total computation cost of feature extraction to obtain the similarities between the query q and Ki images is
CTv KICTV
Due to pairwise input, the whole feature extraction should be done in the online search phase, that is, the online cost ČTV CtV. Note that, in large-scale cross-modal retrieval task, K1 is large, the online computation cost Črv will be prohibitively large for real-time retrieval.
3.3 SPLIT-MERGE BERT
To boost the efficiency, Deformer (Cao et al. (220)) and DC-BERT (Zhang et al. (2020) decouple the query and reference items in lower layers and conduct the cross attention only upper layers.

---

Under review as a conference paper at ICLR 2021
We term this structure as split-merge BERT (BERTsM). We define the total number of transformer layers of BERTSM SM as L. Among them, the upper transformer layers adopt the architecture used in BERTTy and the lower L layers adopt the two-tower architecture. Given query sentence and K images, to obtain their similarities, the total computation cost of feature extraction is
L 1-1 L L
-- - +
! KioTv.
CSM
I
Since image features from two-tower layers can be pre-computed and cached, the online cost is:
L- T. cT T. Kierv.
ČSM
When = 0, BERT SM degenerates to BERT which enjoys high efficiency but lacks cross-modal aftentions. On the other hand, when = I,, it degenerates to BERTTV which exploits cross-modal attentions but takes high computation cost. The speedup of BERTSM over BERTTV is
ÊTV
KICTV
OSM ČSM
L- Kierv
CT, asM is around L/l. Thus, achieve high speed-up ratio, / should be small. But a small I might not fully exploit the cross-modal attentions, limiting its effectiveness for high- accuracy cross-modal retrieval. We visualize the architecture of split-merge BERT in Figure (a).
Since KIeTy >>
3.4 POLY-ENCODER
Poly-Encoder (Humeau et al. (0200) is another architecture for fast self-attention operation over pairs. Originally, used for sentence-documen retrieval. We can also use for cross-modal retrieval. To be specific, Poly-Encoder adopts the two-tower architecture to generate word features of the query and bounding boxes features of the reference image. On the image side, the bounding boxes features are pooled into a global reference image feature I. On the text side, uses m context codes to attend over the word features from text BERT to generate attended word features. Then these attended word features further attended by I to generate the global feature of the text query q. Then the similarity between the reference image and the query sentence is determined by the cosine similarity between and q. Since text-vision attentions are only conducted on the last layer, Poly-Encoder might not effectively models the interactions between vision features and text features. We visualize the architecture of Poly-Encoder in Figure (b).
3.5 CROSS-PROBE BERT
We propose a cross-probe BERT architecture for fast cross- modal retrieval, and meanwhile exploits the text-vision attentions. For the easi- ness of illustration, we divide the cross-probe BERT into upper part and lower part. The lower part adopt two-tower architecture. As shown in Figure 3 we devise vision probes p ) and feed them along with N bounding boxes features B (b.) into vision-stream BERT. The probes are learned weights. which are randomly initialized. Uti- lizing the lower L transformer layers of the vision-stream BERT, the image-attend vision probes P and bounding box features are obtained:
6
85
p"
PR
Figure 3: The architecture of cross cross-probe BERT.
p B BERT 11 L ((P", Bl).
In parallel, we devise text probes pt (p) | -- pt and word features W M. fed into the text-stream BERT. Using the lower L - transformer layers of the text-stream BERT, query-attend text probes Dr and self-attend word features W are generated:
pe W BERT 1::4 -1 [ W])
5

---

Under review as a conference paper at ICLR 2021
In the upper part, query-attend text probes Pr and image-attend vision probes Du are concatenated, go through a text-vision BERT with self-attention layers and obtain cross- moos- attended features:
[P", P BERTH 11: ((P', P"]).
Then sum-pooling is conducted on cross cross-modal attended probes p and Pt to obtain global features:
Dd
Σp. p' DEP
-= DEP
In parallel, the self-attended word features W from the lower L, 1 layers of the text-stream BERT are processed by the upper layers of the text-stream BERT. In similar manner, the self-attended bounding box features B are processed by the upper layers of the vision-stream BERT:
W = BERTY (1 (WV), = B (B).
W and B are further sum-pooled into the global text feature and the global image feature:
Σ
W
b
WEW
be
Finally, we obtain the query's 's global feature q and the reference image's global feature I by
q=
1-b
The similarity score between the reference image and the query text is calculated by the cosine similarity between q and I. Given a query sentence 9 and K₁ reference images, to obtain their similarity scores, we need compute once L-layer BERT Ki times L-layer BERTv and K1 times I-layer BERT c. We denote the computational cost per query of BERT by cT, the computational cost per image of BERTy by ev. and the computational cost per query-image pair of BERTc by cc. thus the total computational cost the proposed cross-par cross-probe BERT for feature extraction
Ccr CT Kicv KiCc-
Since features from vision-stream BERT can be pre-computed, the online cost of the retrieval is only
Čcr = cr + Kiec.
Note that, since BERTc consists / layers, and length of input of BERT; is only m + n, the cost per pair of BERT is significantly smaller than that of BERTTV. Let us denote the input length of a transformer layer as T, then the complexity of the self-attention operation is in linear with T whereas the feed-forward operation is in linear with T. The complexity of transformer layer is
c(T) aT + +TT2
where a is unit cost for feed-forwart operation and is unit cost for self-attention operation. In this case, the speed-up ratio of the proposed cross-probe BERT over the text-vision BERT is
Čiv KgL(a(M N) + 3(M N)
acp
Öep
CT Kil(a(m + n) + 3(m + n)
4 EXPERIMENTS
We evaluate the proposed model on two public benchmarks for the cross-modal retrieval task, MS- COCO (Lin al. (2014) and Flickr30K (Young et al. (2014)) datasets. MS-COCO dataset consists of 123, 287 images, and each image is paired with five text descriptions. It was originally split into 82, 783 training images, 5, 000 validation images and 5, 000 testing images. Following (Karpathy & Fei-Fei (2014)). we add 30, 504 images that were originally in the validation set of MS-COCO to the training set. Meanwhile, we adopt IK testing settings on MS-COCO dataset. Flickr30K contains 31, 783 images collected from the Flickr website. Following (Karpathy & Fei-Fei (0144)) we split the dataset into 29, 783 training images, 1, 000 validation images and 1, 000 testing images. We evaluate the modal retrieval performance through image-to-text and text-to-image recall@1 @K, which is the percentage of ground-truth matchings appearing in the top K-ranked results. Though recall @ is the most widely used metric in real-world applications, also report recall @ (5, 10).

---

Under review as a conference paper at ICLR 2021
MS-COCO
Flickr30K
query 2image R@
image2query R@
qumryeimage @
image2query R
10
10
5
10
5
10
TT
58.3
86.5
91.8
68.3
93.5
97.4
42.5
71.4
79.6
54.9
81.0
88.2
PE
59.4
88.0
93.5
71.4
93.9
97.9
45.9
74.8
82.6
56.1
82.1
89.1
SM
60.6
89.6
95.3
74.2
94.4
97.6
45.1
74.8
84.2
57.6
82.2
89.9
TV
66.3
91.7
96.3
80.6
96.4
98.7
55.0
83.3
89.3
71.2
91.3
96.5
CP
66.0
91.0
95.3
80.2
96.1
98.2
55.0
80.3
86.1
70.8
89.6
94.1
Table I: Comparisons among two-tower (TT) BERT, text-vision (TV) BERT, poly-encoder (PE),
split-merge (SM) BERT and our cross cross-probe (CP) BERT
Model
TT
TV
PE
SM
CP (ours)
Time
0.005s
9.226s
0.011s
1.283
0.121s
Table 2: Time cost per query with 1K candidate images.
We adopt BERT-base model (Devlin et al. 2019)) as the backbone. It consists of 12 layers of Transformer blocks (Vaswani et al. (007]). Each block has 768 hidden units and 12 self-attention heads. We load the weights of BERT-bas. pre-trained on text datasets as the initialization. The training is conducted on a Linux server equipped with 256GB memory and 4 V100 GPU cards. We use the ADAM optimizer. Following ct al. (2020), 100 bounding boxes per image is extracted by
faster R-CNN (Ren et al. (2015) pre-trained on visual genome (Krishma et al. (017) provided by Anderson et al. (2018). We set the maximal query length as 44, which is the same Li al. 2020
4.1 WITHOUT PRETRAINING
As shown in Table I text-vision (TV) BERT significantly outperforms two-tower (TT) BERT. In the query2image retrival task, the recall @ I achieved by TT is only 58.3. In contrast, TV achieves 66.3 recall @ In the image2query retrieval task, TT only achieves a 68.3 recall In contrast, TV achieves an 80.6 recall 1. The higher accuracy of TV validates the effectiveness of exploiting multi-modal attention. As shown in Table the proposed cross-probe (CP) BERT also significantly outperforms TT. Meanwhile, it achieves comparable recall@1 in query2image retrieval compared with TV. An interesting observation in Table i is that our CP outperforms SM. Note that, we set I = 2 and L 12 for both CP and SM. We further compare the time cost. The experiments are conducted on a single V100 GPU. As shown in Table 2 TV is significantly slower than TT. Mcanwhile, PolydEncoder (PE) is much faster than TV, and slower than TT. SM is faster than TV, but much slower than TT. Our CP achieves 77 speedup over TVV, and 10x speedup over SM.
The proposed cross-probe BERT relies architecture in the lower I. 1 layers and uses the text-vision architecture in the upper 7 layers. Different from split-merge encoder, in the upper 1 layers, only vision probes and text probes involved in computing cross-modal attentions. Since the number of vision and text probes (m, n) much smaller than the number of words and bounding boxes (M, N), the proposed cross-probe BERT is much more efficient compared with split-merge BERT when using the same and L.
text probes and vision probes. It adopts two-tower
and L / 10. As shown in Table as (m, n) changes from (5, 5) to (15, 15), the performance significantly improves. For instance, when m and n = 5, only achieves 62.2 @ecall 1 in the query-to-image retrieval. contrast, when m 15 and = 15, it achieves 67.0 @ Nevertheless, larger m and will lead to larger computation cost. Considering both effectiveness and efficiency, we choose m and n 15.
Influence of the number of probes. We fix
query2image R@
image2query R@
time per
m
T
5
10
5
10
query
5
62.2
89.3
94.6
75.3
95.7
97.8
0.542s
10
10
63.1
89.8
94.6
76.9
95.4
97.8
0.126s
15
15
67.0
91.2
95.5
80.7
96.3
99.1
0.191s
15
66.0
91.0
95.3
80.2
96.1
98.2
0.121s
Table 3: The influence of the number of text and vision probes m, n. Experiments are on MS-COCO.
7

---

Under review as a conference paper at ICLR 2021
R@@
image2query R@
time per
5
10
5
10
query
8
65.8
91.2
95.5
80.7
96.3
98.4
0.250s
10
66.0
91.0
95.3
80.2
96.1
98.2
0.
11
63.6
89.9
94.7
75.3
96.2
98.4
0.063s
Table 4: The influence of the number of text-vision layers 1. Experiments are on MS-COCO.
MS-COCO
Flickr30K
query2image R@
image2query R@@
query Dimage R@
image2query R
@
5
10
10
10
5
10
TT
62.7
89.3
93.5
76.2
95.9
98.6
57.7
81.2
86.7
73.4
92.6
96.0
PE
61.4
89.1
93.9
75.9
94.8
98.2
58.3
83.1
89.1
72.8
92.2
96.2
SM
62.4
90.1
95.6
75.3
94.8
97.8
58.6
83.3
89.8
74.2
93.1
96.4
TV
69.6
93.1
97.2
83.4
97.2
99.1
69.0
90.4
94.5
81.5
95.9
98.3
CP
70.9
92.5
96.6
83.3
96.9
99.4
69.1
89.8
94.1
83.5
96.0
98.0
Uni-VL UNITER
69.7
93.5
97.2
84.3
97.3
99.3
71.5
90.9
94.9
86.2
96.3
99.0
72.5
92.4
96.1
85.9
97.1
98.8
Table 5: Comparisons among two-tower BERT, text-vision BERT, Poly-Encoder, and cross-probe BERT, Unicoder-VL and UNITER with pre-training.
Influence We fix L 12, m = and n = 15. Since L is fixed, the larger leads to a smaller L- As shown in Table 4 when = 1, that is, using only a single cross-attention layer, our cross- probe (CP) BERT achieves a 63.6 recall@ in query2image retrieval. It is significantly better than two-tower BERT with a 58.3 recall 1. In the meanwhile, when / increases from 1 to 2, the recall @ considerably increases to 66.0, whereas the text-vision (TV) BERT achieves a 66.3 recall 1. That is, using only 2 cross-attention layers in CP BERT has achieved comparable performance with text-vision BERT with 12 cross-attention layers. Mcanwhile, our CP BERT uses only text probes and 15 vision probes, whereas TV BERT uses 44 word features and 100 bounding box features. When increases from 2 to 4, accuracy does not change considerably, thus we set = 2 by default.
4.2 WITH PRE-TRAINING
Existing BERT models rely on pre-training on a large-scale dataset. We further evaluate the influence of pre-training on the proposed cross-probe BERT model. Following (Li al. (2020), ) we two datasets, SBU Captions (Ordonez et al. 2201)) and Conceptual Captions (Sharma et al. (2018). for pre-training. Conceptual Captions contains 3.3M image-caption pairs crawled from the web. Due to broken URLs, the number of image-caption pairs of Conceptual Captions dataset is around 3M. SBU Captions contains 1M image-caption pairs. Due broken URLs, the number of image- caption pairs of SBU Captions dataset is around 0.8M. We pre-train all ranking models by triplet loss and the retrieval performance shown T As shown in the table, our cross-probe BERT achieves comparable retrieval accuracy with text-vision BERT. Meanwhile, our CP considerably outperforms TT, PE, and SM. We also compare two recent text-vision BERT methods, Unicoder- VL (i et al. (2200) and UNITER (Chen et al. (020)). Note that, Unicoder- VL is pre-trained on Conceptual Captions and SBU datasets as ours but UNITER is pre- trained on Visual Genome Krishna et al. (2017) and MSCOCO Lin al. (2014) datasets in addition to Conceptual Captions and SBU datasets. As shown in Table 5) our CP achieves comparable performance with Unicoder- VL on MS-COCO dataset. But performance is worse than Unicoder-V VI (Li et al. (2020) and UNITER (Chen et al. (2020)) on Flickr30K dataset. Note that, both Unicoder-Vi VL and UNITER adopt text-vision BERT architecture, and thus they are significantly slower than ours.
CONCLUSION
Benefited from exploiting cross-modal attentions, text-vision BERT has achieved excellent perfor- mance in vision-language retrieval. Nevertheless, the extremely expensive computational cost of text-vision BERT limits its usefulness in large-scale search. To boost efficiency and meanwhile maintain the high effectiveness, we propose a novel ranking model, cross-probs BERT this work. By utilizing devised probes, the cross-model attentions are conducted which much more efficient than text-vision BERT. Systematic experiments conducted on two public datasets demonstrate the excellent effectiveness and efficiency of our cross-probe BERT.
small number of probes,
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and vqa. In CVPR, 2018.
Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. Deformer: Decomposing pre-trained transformers for faster question answering. 2020.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. In ECCV, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pp. 4171-4186, 2019.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge mhowledge-powered conversational agents. In ICLR, 2019.
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual- semantic embeddings with hard negatives. BMVC, 2017.
Yunchao Gong, Qifa Ke, Michael Isard, and Svetlana Lazebnik, A multi-view embedding space for modeling internet images, tags, and their semanties. IJCV, 2012.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Architec-
tures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR, 2020.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descrip- tions. T-PAMI, 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
Kuang-Huse Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pp. 201-216, 2018.
Gen Li, Nan Duan, Yuejian Fang. Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang, Visualbert: A simple and performant baseline for vision and language. In arXiv, 2019.
Tsung Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context. In arviv, 2014.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-lanm and-language tasks. In NIPS, 2019.
Mazare, Samuel Humeau, Martin Raison, and Antoine Bordes. Training millions of personalized dialogue agents. EMNLP, pp. 2775-2779, 2018.
Vicente Ordonez, Girish Kulkarni, and Tamara I. Berg. 1m2text: Describing images using million captioned photographs. In NIPS, 2011.
Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert R. G. Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In ICME, 2010.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. T-PAMI, 2015.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.

---

Under review as a conference paper at ICLR 2021
Hao Tan and Mohit Bansal. Lxmert: Learning cross cross-modality encoder representations from trans- formers. In EMNLP, 2019.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act a fantasy text adventure game. In ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Wip.c.iilimois.co 2014.
Yuyu Zhang, Ping Nie, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang. De-bert: De- coupling question and document for efficient contextual encoding. In SIGIR, 2020.
A APPENDIX
A.1 ABLATION STUDY ON SPLIT-MERGE BERT.
In the lower - layers, split-merge BERT conducts two-tower architecture, which individually models the text feature and the image feature. In the upper layers, it adopts the text-vision archi- tecture, which exploits the cross-modal attentions. When = 0, split-merge BERT degenerates to two-tower BERT, which fast but not effective. On the other hand, when L. split-merge BERT degenerates to text-vision BERT, which is slow but effective. We evaluate the effectiveness of split- merge BERT when / (0. 2, 8, 9, 10, 12). We fix L 12 and thus case. We conduct experiments on MS- IS-COCO dataset. As shown in Table when 1 = 2, the accu- racy is slightly better than the two-tower BERT
{0, 3, 10, 12) in this
0), but much worse than the text-vision BERT (1 = 12). When / increases to 9, the query2image retrieval has achieved comparable accuracy with text-vision BERT. But the time cost when = 9 also large, limiting its efficiency.
query 2image R @
image2query @
1
L
1
5
10
5
10
12
0
66.3
91.7
96.3
80.6
96.4
98.7
10
2
67.2
92.2
96.5
79.6
97.1
98.8
9
3
66.1
91.9
96.5
78.2
96.2
98.4
8
4
64.4
91.1
96.1
77.8
95.8
98.4
2
10
60.6
89.6
95.3
74.2
94.4
97.6
0
12
58.3
86.5
91.8
68.3
93.5
97.4
Table 6: The influence of the number of text-vision layers l and that of two-tower layers I, I.
A.2 ABLATION STUDY ON POLY-ENCODER.
utilizes m context codes to generate the m query-contert features. Then the query-conter, features attended by the image global feature to generate the image-context query global feature. The similarity score between the query sentence and the reference image is determined by the cosine similarity between the image-context query global feature and the image global feature. Since the cross-modal attentions are only exploited in the last layer, it might not be able to effectively model the interactions between visual features and word features. We evaluate the performance of poly- encoder in cross-modal retrieval and vary m among (5, 10, 20, 40}. The experiments are conducted on MS-COCO dataset. As shown in Table 7 the accuracy of poly-encoder increases as the number of context codes m increases. This is expected since more context codes can encode richer information. Nevertheless, when m increases from 20 to 40, the retrieval accuracy slightly decreases. The worse performance might be due to over-fitting, In the meanwhile, compared with the two-tower encoder, the performance of poly-encoder is better, validating the effectiveness of the cross-modal attention in the last layer. On the other hand, compared with text-vision encoder, the accuracy achieved by poly-encoder is much lower, validating our statement that the cross- cross-modal modal attention in the last layer might not fully exploit the interactions between the query and the image.
10

---

Under review as a conference paper at ICLR 2021
query2image R@@
image2query R@@
10
5
10
m=
57.3
87.
92.9
68.5
93.5
97.3
m=10
58.7
87.7
93.4
70.2
93.7
97.8
m=20
59.4
88.0
93.5
71.4
93.9
97.9
m=40
58.6
87.7
92.9
70.6
93.4
97.7
Table 7: The influence of the number of context codes the retrieval accuracy.
A.3
EXPERIMENTS ON MSCOCOSK
MS-COCO
query 2image
R@
image2query R@
5
10
10
TT
39.3
70.2
80.2
54.8
80.7
88.3
PE
39.7
70.1
80.5
55.6
81.5
89.5
SM
42.9
73.3
82.8
58.8
83.8
90.9
TV
46.5
75.6
85.0
62.6
86.8
92.6
CP
46.8
75.8
85.0
62.9
86.7
92.7
Uni
46.7
76.0
85.3
62.3
87.1
92.8
UNITER
48.4
76.7
85.9
63.3
87.0
93.1
Table 8: Comparisons among two-tower BERT, text-vision BERT, Poly-Encoder, and cross-probe BERT, Unicoder- VL and UNITER with pre-training.
Model
TT
TV
PE
SM
CP (ours)
Time
0.005s
48s
0.03s
0.6s
Table Time cost per query with 5K candidate images.
Table compares the accuracy of the proposed CP methods and the baseline methods as well as existing state-of-the-art methods. As shown in Table X our CP method achieves comparable accuracy with text-vision BERT model (TV), Unicoder- VL and UNITER.
Table 9 compares the time cost query used in all methods. The experiments are conducted on single 100 GPU. Note that, TV method takes more than days to finish the testing on MSCOCO 5K split, which is extremely slow.


Reviews:
[{"review_id": "Y1hq2SxHZvA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improved method, Cross-Probes BERT,  based on the split-merge architecture to further accelerate cross-modal retrieval. Specifically, it devises several vision probes and text probes along with the image inputs and query inputs in the low-level two-tower architecture. After that, the attended vision and text probes are concatenated and fed into cross-modal self-attention layers to interact in the high-level one-tower architecture. The experiments on two benchmarks, MS-COCO and Flickr30K show quite promising performance, especially on computation efficiency.\n\nOverall, the definition of the task and the overall architecture of this paper are both clear and straightforward, making this paper easy to understand. The idea of introducing some probes to collect information and reduce the computation cost in the split-merge architecture makes sense to me. Two kinds of probes, namely vision probes and text probes, are proposed to collect the global features of image inputs and query inputs, which is reasonable and interesting. Besides, the paper provides comprehensive experiments on performance and computation cost to validate the effectiveness and efficiency of their proposed method.\n\n\nHere are some of my questions and concerns for the paper:\n1. Compared with previous works (such as Ploy-Encoder, BERT_{SM}, DCBERT), the proposed method seems a bit incremental.\n2. The architecture proposed in this paper seems to be applicable to text retrieval tasks. Why not test the model in some text-text retrieval tasks explored in Ploy-Encoder and DCBERT.\n3. According to Section 3.5, the \\overline{w} is the sum-pooling of \\hat{w}_1, …, \\hat{w}_N while it is the output of l transformer layers in Figure 3. Is there something wrong in Figure 3? Please make it clear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EZRaBHsqB6h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The efficiency of current SOTA methods on image-text retrieval, especially those visual-language cross-modality pre-training models, has always been a critical problem compared with traditional joint-embedding models. This paper aims to solve this problem by integrating the efficiency of two-tower models in lower layers and the effectiveness of cross-attention learning models in higher layers. Specifically, this paper proposes to generate several vision and text probes with local features to decrease the cost of computation. Experiments on MSCOCO 1K test set and Flickr30K are conducted to show the effectiveness of the proposed method. \n\nThe motivation is insightful, and the method seems reasonable, while the experiment is weak. Following is my detailed concerns.\n1.\tThe paper shows the overall cost of models while lacks the detailed analysis of each part in the model. For example, is the computational cost of each BERT layer on query and image the same? What is the unit cost for feed-forward and self-attention operation? These also have an influence on the total cost of the models.\n2.\tThe operation cost on CPU is not shown in the paper.\n3.\tSince the key of this paper is to discuss the efficiency of retrieval model, the influence of different parameters (e.g., number of probes, l) in terms of efficiency is very important. However, only comparison of performance is shown in the paper. \n4.\tResult of performance and cost time on 5K test set would better demonstrate the efficiency of the proposed model, while this is lacked in this paper.\n5.\tThis approach tries to improve efficiency of image-text retrieval while has some sacrifice on effectiveness. While most SOTA methods on this problem is based on text-vision (TV) BERT. How to effectively adapt this approach to existing models and the generalization of this method is not discussed in the paper.\n\nOverall, I think this paper could be improved, especially in experiment part. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "B5Psz-alhcY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a architecture termed cross-probe BERT to deal with the costly computation from cross-modal attentions used in search task. Motivated by the success of the “split-merge” style encoder in query document retrieval, it extends text-vision BERT to adopt it for speeding up the computation. In particular, it devises several vision probes and text probes along with the image’s local features and the query’s word features. Besides, the cross-modal attention is conducted on text and vision probes. Therefore, it can exploit cross-modal attention with a lightweight computation cost. Systematic experiments demonstrate the state-of-the-art effectiveness and efficiency of the proposed method.\n\nPros: \n- Relative strong results compared to baselines about the effectiveness and efficiency of cross-modal retrieval.\n- Paper is well written and easy to follow.\n\nCons: \n- Limited contribution of the proposed method. This paper extends text-vision BERT to adopt it for speeding up the computation. Besides, ideas are similar to the combination of split-merge BERT + Poly-Encoder based on the architecture in Figs 2 and 3.\n- The hyper-parameters \\alpha and \\beta in the equation of complexity of transformer layer are not clearly stated. What’s the effect and role about them? In addition, the corresponding analysis is not provided.\n- Insufficient experimental analysis, such as the influence of the number of probes.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4vTL0-WhKxI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n1. This paper proposed a new architecture to accelerate the Cross-Modal BERT inference speed. The motivation is current Cross-Modal BERT is too slow during the online inference, especially for the large data. The bottleneck is due to the cross-modal attention. This paper proposed a new architecture that accelerate the inference speed, while keeps the retrieval performance.\n\nStrength:\n1. The motivation is clean. The slow inference speed problem of current image-text BERT is an important problem.\n2. The idea is clean, although there might be some mistakes (listed in the weakness section). The idea basically contains two parts. First, caching as much information as possible during inference. Therefore, the proposed approach applies less layers for Cross-Modal attentions. Those cross-modal attention layers are applied at the end of the architecture. Second, accelerate the cross-modal attention by using less tokens for this module.\n3. The performance is good when comparing with the SoTA w/ and w/o pre-training.\n\nWeakness:\n1. The method seems not novel.  The method seems like a merge of Split-merge model and the Poly-Encoder model.\n2. The equation for $\\alpha_{CP}$ might not be correct. As $\\alpha_{CP}$ is the speed-up ratio of the proposed cross-probe BERT over the text-vision BERT, it should be $\\alpha_{CP} = \\frac{\\hat C_{TV}}{\\hat C_{CP}}$. Furthermore, the numerator seems incorrect. If the numerator is for $\\hat C_{TV}$, it should not include $C_T$. \n3. The Table 3 is a little bit surprising. The performance of $m = n = 15$ is even higher than the performance of cross-modal BERT (TV in Table 1). As the cross-modal BERT has much more cross-attention layer than the proposed approach, I somehow feel the TV model is not trained properly. The lower performance of TV might indicate the TV model is overfitting.\n4. For Table 4, the CP model achieves higher (or similar) performance than Unicoder-VL on COCO, but worse performance on Flickr30k. I wonder what is the reason?\n5. If I remembered correctly, ViLBERT is pretrained using only Conceptual-Caption data (SBU is not used), while the CP is pretrained using both SBU and Conceptual-Caption. Therefore claiming higher performance than ViLBERT might be misleading. \n6. The Table 5 is incomplete. More recent works should be included and compared in the Table, for example UNITER: Learning UNiversal Image-TExt Representations. \n7. The image-text BERT model is very sensitive to the hard negative mining during training. UNITER and UNICODER-VL uses online hard negative mining, which would leads to better performance, while ViLBERT uses offline hard negatives. I wonder what is the implementation detail for Table 1 and Table 5? Specifically, are the model in Table 1 trained using the same hard negative mining strategy?\n8. The paper shows the inference speed at 1K testing case. As COCO has 5K testing split, I wonder what is the speed and image retrieval accuracy for 5K testing case? What is the speed-up ratio against TT, PE, and SM over 5K split? Showing speed-up ratio in large amount of data is important for judging the effectiveness of the proposed approach.\n9. Why PE is much faster than the proposed approach (CP) in Table 2?\n10. As in the method part, the author shows the theoretical speed up ratio. I wonder what is actual number of theoretical speed-up for CP over TT, PE, SM, and TV. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the rebuttal phase, all reviewers give borderline scores (leaning slightly positive, one of these noted in the comment rather than final review). While the reviewers recognize the potential merit of the contribution (efficiency while preserving effectiveness), support for acceptance is not sufficient. The major concerns include novelty (shared by multiple reviewers) and the limited experimental settings.
Decision: Reject