EDIS: Entity-Driven Image Search over Multimodal Web Content
Authors: Siqi Liu
Weixi Feng
Tsu-Jui Fu
Wenhu Chen
William Yang Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Image search; Cross-modal retrieval; Multimodality fusion
TL;DR: We propose a large-scale dataset for image search over multimodal web content under entity-rich queries.

Abstract: Making image retrieval methods practical for real-world search applications requires significant progress in dataset scales, entity comprehension, and multimodal information fusion. In this work, we introduce Entity-Driven Image Search (EDIS), a challenging dataset for cross-modal image search in the news domain. EDIS consists of 1 million web images from actual search engine results and curated datasets, with each image paired with a textual description. Unlike datasets that assume a small set of single-modality candidates, EDIS reflects real-world web image search scenarios by including a million multimodal image-text pairs as candidates. EDIS encourages the development of retrieval models that simultaneously address cross-modal information fusion and matching. To achieve accurate ranking results, a model must: 1) understand named entities and events from text queries, 2) ground entities onto images or text descriptions, and 3) effectively fuse textual and visual representations. Our experimental results show that EDIS challenges state-of-the-art methods with dense entities and the large-scale candidate set. The ablation study also proves that fusing textual features with visual features is critical in improving retrieval results.

EDIS: Entity-Driven Image Search over Multimodal Web Content
Siqi Liu"+ Weixi Feng Tsu-jui Fu² Wenhu Chen
William Yang Wang"
'Cormell University "UC Santa Barbara
University of Waterloo, Vector Institute
s12322@cornell.edu, wentuchen@uwaterloo ca (weixifeng, tsu-juifu, wi william)@css wcsp. edu
Abstract
Text-to-Image Retrieval (MSCOCO, F0Kc
Query: naroow kitchen filled with appliances and cooking utensils.
Retneval Model
1K-3K imsges
Making image retrieval methods practical for real-world search applications requires signif- icant progress in dataset scales, entity com- prehension, and multimodal information fu- sion. In this work, introduce Entity- Driven Image Search (EDIS),s challenging dataset for cross-modal image search the news domain. EDIS consists of 1 million web images from ac- tual search engine results and curated datasets, with each image paired with textual descrip- tion. Unlike datasets that assume a small set of single-modality candidates, EDIS reflects real- world web image search scenarios by including a million multimodal image-text pairs as can- didates. EDIS encourages the development of retrieval models that simultaneously address modal information fusion and matching. To achieve accurate ranking results, a model must: I) understand named entities and events from text queries, 2) ground entities onto im- ages descriptions, and 3) effectively fuse textual and visual representations, Our experi- mental results show that EDIS challenges state- of-the-art methods with dense entities and the large-scale candidate set. The ablation study also proves that fusing textual features with vi- sual features is critical in improving retrieval results.
Rank
Bank 2
Rank 3
Rank
Aetity-Drriven Image Search (EDIS)
1M image text pairs
Query:: GOP gracidential condidate Ron Paul makes à point Mil Romney Republican debere. New Hampshtre
Retrieval Model
THE
ltck Sunurun slaneed Boneay, COR Debito
looa Le-alad La
New Hempahim debele Mil Rummas
pols
ruudd twa:
peiliica Hacke th Solex
Rurmayy's frinsal
feeds sharp citacia tron
Repubbiizz candidates
ce Cai
Misscn Erilly
Mising Eniity 3
Incorvedt Event
Rank
Rank
Rank
Figure I: EDIS contains entity-rich queries and multi- modal candidates. EDIS requires models to recognize subtle differences across different modalities to iden- tify the correct candidates. For instance, the last three sample candidates either miss entities in the image describe a different event.
(Plummer et al., 2015; Lin et al., 2014), which is far less than the number of images that search engines must deal with in real applications. 2) In- sufficient entity-specific content: existing datasets focus on generic objects without specific identities. Specific entities ("Statue of Liberty") in web im- ages and text may be recognized as general objects ("building") "). 3) Modality mismatch: existing im- age retrieval methods usually measure image-text similarity. However, for web image search, the sur- rounding text of an image also plays crucial part in this fast and robust retrieval process.
1 Introduction
Image search, also known as text-to-image retrieval, is to retrieve matching images from candidate set given a text query. Despite the advancements in large-scale vision-and lnd--angu models (Wang et al., 2021; Zhang et al., 2021; Chen et al., 2020; Li et al., 2020b), accurately retrieving images from a large web-scale corpus remains challenging problem. There remain several critical issues: 1) Lack of large-scale datasets: existing image re- trieval datasets typically contain 30K-100K images
Recently, there has been a continuous interest in event-centric tasks and methods in the news domain (Reddy et al., 2021; Varab and Schluter,
Equal contribution
'Code and data: https://g http:///e///////////...

---

2021; Spangher et al., 2022). For instance, NY- Times800K (Tran al., 2020), and Visual News (Liu et al., 2021) are large-scale entity-aware bench- marks for news image captioning. TARA (Fu et al., 2022) is proposed to address time and location rea- soning over news images. NewsStories (Tan et al., 2022) aims at illustrating events from news articles using visual summarization. While many of these tasks require accurate web image search results as a premise, large-scale image retrieval dataset is lacking to address the challenges of understanding entities and events.
image search, which characterizes single- modality queries and multi-modal candidates. EDIS curated to include images and text segments from open sources that depict a sig- nificant amount of entities and events.
We propose feature-level fusion method for multi-modal inputs before measuring align- ment with query features. We show that im- ages and headlines are exclusively crucial sources for accurate retrieval results yet can- not be solved by naive reranking.
Therefore, to tackle the aforementioned three key challenges, we introduce large-scale dataset named Entity-Driven Image Search (EDIS) in the domain. As is shown in 1ig. EDIS has much larger candidate set and more entities in the image and text modalities. In addition to im- ages, text segments surrounding an image are an- other important information source for retrieval. In news articles, headlines efficiently summarize the events and impress readers in the first place (Pan- thaplackel et al., 2022; Gabriel et al., 2022). Hence, simulate web image search with multi-modal information, pair each image with the news headline as a textual summarization of the event. As a result, EDIS requires models to retrieve over image-headline candidates, which is a novel setup over existing datasets.
We evaluate existing upproaches on EDIS and demonstrate that EDIS is more challenging than previous datasets due to its large scale and entity-rich characteristics.
Related Work
Cross-Modal Retrieval Datasets Given a query sample, cross- mross-modal retrieval aims retrieve matching candidates from another modality (Bain et al., 2021; Hu and Lee, 2022; Wang et al., 2022; Sangkloy et al., 2022). Several datasets have been proposed or repurposed for text-to-image retrieval. For instance, MSCOCO (Lin et al., 2014), and Flickr-30K (Plummer et al., 2015) are the two widely used datasets that consist of Flickr images of common objects. CxC (Parekh et al., 2021) extends MSCOCO image-caption pairs with con- tinuous similarity scores for better retrieval evalua- tion. Changpinyo et al.(2021) repurposes ADE20K (Pont- Tust-Tuset&& et al., 2020) for image retrieval with local narratives and mouse trace. WebQA (Chang et al., 2022) has similar scale to EDIS and defines source retrieval prerequisite step for answering questions. The source candidates are either text snippets or images paired with a short description. In contrast, EDIS is a large-scale entity-rich dataset with multi-modal candidates that aligns better with realistic image search scenarios.
Given text query, existing models can only measure query-image or query-text similarity alone. BM25 (Robertson et al., 2009), and DPR (Karpukhin et al., 2020) fail utilize the visual features, while Aision-language models like Visual- Bert (Li et al., 2019) and Oscar (Li et al., 2020b) cannot be adopted directly for image-headline can- didates and are infeasible for large-scale retrieval. Dual-stream encoder designs like CLIP (Radford et al., 2021) are efficient for large-scale retrieval and can compute weighted sum of query-image and query-text similarities to utilize both modali- ties. However, as shown later, such multi- modal fusion sub-optimal for EDIS. In this work, we evaluate image retrieval models on EDIS and re- veal that the information from images and head- lines cannot be effectively utilized with score-level fusion. Therefore, we further proposed feature- level fusion method to utilize information from both images and headlines effectively. Our contri- bution three-fold:
Tmxa-e Retrieval Methods Text-to- image retrieval has become a standard task for vision-language (VL) understanding (Lu et al., 2019; Li et al., 2020a; Kim et al., 2021; Jia et al., 2021; Fu et al., 2021; Dou et al., 2022). Single-stream approaches like VisualBert (Li al., 2019), and UNITER (Chen al., 2020) rely on a unified transformer with concatenated image and text tokens as input. Dual-stream approaches like CLIP (Radford et al., 2021) or ALIGN (Jia
We collect and annotate EDIS for large-scale

---

Multimodal News Datasets ViuaiNwwx TARA
Queries (22K) Gather mage-ccationn
Six rayal gali esits recokweed mmm Sparth
EDIS Dataset
Use captions as querls
stlwwrck noar Florid
Filter data. by complexity, entty count, and image text or text text similariies
(1M)
Positive Samples (84K)
Relevant imags-headline
rieck
hade
Other Negative Samples (792K)
Collect challngingg
candidates with score
Collect mage hagg-heeddine candidates VisualNews
image-l Leadge- pairs from Google Image Search and VisualNews
Hard Negative Samples (172K)
AL AiVorrmm
Irrelevant imaga-headine
Precee Wheid
candidates with scare
pairs with MTuu
Figure EDIS data collection process. The process consists query selection, candidate collection and annotation, and hard negative mining.
et al., 2021) have separate encoders for image and text modality and, thus, are much more efficient for retrieval. However, adopting these models for multi-modal candidates leads to architecture modification or suboptimal modality fusion. In contrast, ViSTA (Cheng al., 2022) aggregate scene text additional input to the candidate encoding branch. In this work, we propose method named mBLIP perform feature-level fusion, achieving more effective information fusion for multi-modal candidates.
illustrates the overall dataset collection pipeline.
4.1 Query Collection
We extract queries and ground truth images from the VisualNews (Liu et al., 2021), and TARA (Fu et al., 2022) datasets. These datasets contain news articles that have headline, image, and im- age caption. We adopt captions as text queries and use image-headline pairs as the retrieval candidates.
3 Task Formation
We design series of four filters select high- quality, entity-rich queries. 1) Query complexity: we first evaluate the complexity of queries and move simple ones with less than ten tokens. 2) Query entity count: we use spaCy to estimate av- erage entity counts in the remaining queries and remove 20% queries with the lowest entity counts. The resulting query set has an average entity count above 4.0. 3) Query-image similarity: to ensure strong correlation between queries and the cor- responding ground truth image, we calculate the similarity score between query-image using CLIP (Radford et al., 2021) and remove 15% samples with the lowest scores. 4) Query-text similarity: we calculate the query-text similarity using Sentence- BERT (Reimers and Gurevych, 2019) and remove the top 10% most similar data to force the retrieval model to rely visual representations.
EDIS contains a set of text queries Q (21,02.-.) and set of candidates Cm. essentially
and image-headline pairs B {c1 (i1. h1), (12, h2). where i, denotes image and h. denotes the associated headline. For query 4m- retrieval model needs to rank top-k most relevant image-headline pairs from B. As shown Fig. I, both images and headlines contain entities that are useful for matching with the query.
We evaluate approaches with both distractor and full candidate sets. The distractor setup is simi- lar conventional text-to-image retrieval using MSCOCO (Lin et al., 2014), or Flickr30K (Plum- mer et al., 2015), where images are retrieved from a limited set i with ~555 (image, headline) pairs. The full setting requires the model to retrieve im- ages from the entire candidate set B.
To avoid repeating queries, we compare each query to all other queries using BM25 (Robertson et al., 2009). We remove queries with high simi- larity scores as they potentially describe the same news event and lead the same retrieved images. As shown in Table 1, we end up with 32,493 queries split into 26K/3.2KK K/3332 for train/validation/ess..
4 Eniity-Driven Image Search (EDIS)
We select queries and candidates from human- written news articles and scraped web pages with different stages of filtering. Then we employ hu- man annotators to label relevance scores. Fig.

---

Retrieval Candidate
Text Query
# Img
Modality
Source
Label
Train
Val
Tost
Entity
Flickr30K (Plummer al., 2015)
1K
Imuge
Flickr
Binury
145K
5K
0.35
MSCOCO (Lin 2014)
5K
Image
Flickr
Binary
566K
25K
25K
0.18
CxC (Purekh et ul., 2021)
5K
Image
Flickr
Continuous
25K
0.18
ADE20K (Changpinyo et al., 2021)
2K
Image
Flickr
Binary
20K
2K
0.16
WebQA (Chang et al., 2022)
390K
Text/Image
Wikimedia
Binary
34K
5K
7.5K
1.96
EDIS (ours)
1M
Image-Text
Open (Google)
3-Likert
26K
3.2K
3.2K
4.03
Table I: Statistics EDIS and existing image retrieval datasets. EDIS has larger set of multi-modal candidates, unrestricted image sources, multi-scale annotations, and entity-rich queries compared previous datasets.
120k
124
Google
16k
X=3
reach a consensus. Controversial candidates that workers cannot agree upon after two rounds of annotations are discarded from the candidate pool. Additionally, one negative candidate is added to each annotation task to verify workers' attention. The final conformity rate among all annotations is over 91.5%.
14k
of candisates
1.0K
News
No of queries
12k
B0k:
10k
8k-
46.0
6k
20k
9.7
Relevance Score
condidates with query
Figure 3: Left: Annotated candidates distribution by relevance score. Right: Query distribution by the scores of annotated candidates.
Hard Negatives Mining We discover that EDIS queries can be challenging to Google Image Search in some cases. Among the 200K images from Google Search, 29K (~ (~15%) are annotated with a score of 1, and 124K (~62%) are annotated with a score of 2. These candidates are hard negatives that require retrieval models to understand and ground visual entities in the images. As for candidates from VisualNews, there are 9.7K (~41%) with score of 1 and 9.5K (~40%) candidates with score of 2. We refer to these samples as in-domain hard negatives as their headlines share some enti- ties with the query but refer to different events with discrepant visual representations.
4.2 Candidate Collection
In web image search experience, multiple relevant images exist for single query. Therefore, we expand the candidate pool so that each query cor- responds to multiple image-headline pairs. Addi- tional candidates are collected from Google Im- age Search and the rest of the VisualNews dataset. For each query from VisualNews, image-headline pairs from Google search. For each query from TARA, select five image-headline pairs from Google search. and two mnagl-headline pairs from VisualNews. Then, we ask annotators to label the relevance score for each candidate on a three-point Likert scale. Score I means "not rele- vant" while 3 means "highly relevant". Formally, denote E(.) as the entity set and E() as the event of query Zm candidate On (In, hn). define the ground truth relevance scores
select seven
Soft Negative Mining of the headlie pairs from VisualNews and TARA to augment the candidate pool. These candi- dates naturally negative candidates with rele- vance score of because of the unique article tents and extensive diversity in topics. Therefore, our dataset consists 1,040,919 image-headline candidates total.
Lastly, we utilize the rest
3 (((m)) S B(cn)
4.3 Dataset Statistics
and E(gm) E(cn),
We demonstrate the major advantage EDIS over existing datasets in Table 1. EDIS has the largest candidate set with a consistent candidate modality. Our images are not restricted to specific source result of collecting images from real search gine. Queries from EDIS are entity-rich compared to datasets with general objects (e.g. MSCOCO). In Fig. 3 (left), we show the score distribution of
2 if (c,, 0 B(gm) + and E(qm) E(cn),
rel(m, n))
1 if E((m) 0 E((n) = 0
or E(gm) # {C,.)
Each candidate is annotated by at least three workers, and is selected only when all workers

---

human annotations. Candidates mined from Visual News are mostly in-domain hard negatives, while the images represent missing entities or different events. These candidates are mostly annotated with scores of or 2. As for Google search candidates, many images depict the same event but with miss- ing entities. Therefore, the annotations concentrate on score 2. In Fig. 3 (right), we show that most of the queries have least one hard negative, usu- ally more score 2 negatives than score negatives. About half of the queries have more than one posi- tive candidate (score 3). We show more examples EDIS candidates in Fig. 8-11.
Waa.
fN
fi
fQ
ft
Albetion
Chiess
fo
fn
En
Headline Query
Headline
Query
hn
4m
An
Qm
Score-level Fusion
Festure-level Fusion
Figure 4: Score-level fusion encodes each modality into a single feature vector, while feature-level fusion outputs single feature vector for multi-modas candidates by adopting cross-attention layers.
5 Multi-Modal Retrieval Method
Given a text query q. model should be able to encode both images in and headlines h, to match the query encoding. Therefore, the model should include multi-modal candidate encoder fc and query encoder (q. Within fc, there is branch for image input fi and branch for headline fn- We formalize the matching process between a query Im and a candidate Cn, = (in hnp) as:
As is shown in Fig. 4, we first extract image embed- dings fr(.) using the image encoder and then feed fr(-) into the cross-attention layers of fH- The out- put from fu is a feature vector U4,4 that fuses the in- formation from both image and text modalities. We separately obtain the query feature U/ fQ((m) where SQ shares the same architecture and weights with fH, except that the cross attention layers are not utilized. We adopt the Image-Text Contrastive (TTC) loss (Li et al., 2021) between Pi and P to align the fused features with query features.
Syn,r fQ(am)² fc(ta,
(2)
where Sm,, n. is the similarity score between on and Cu. Based the design of fc. we categorize meth- ods into score-level fusion and feature-level fusion.
Experiment Setup
Score-level Fusion These methods encode im- age and headline independently and compute a weighted sum of the features, i.e., fo(in, hn) = wi (i, + W2/H (hx). Therefore, 8m,, is equiva- lent to weighted sum of query-image similarity $ and query-headline similarity sk.
6.1 Baselines
For score-level fusion mentioned in Sec. 5, consider CLIP, BLIP, fine-tuned BLIP, and BM25+CLIP reranking utilize both modalities of the candidates. In addition, we benchmark exist- ing text-to-image retrieval methods, and text docu- ment retrieval methods, including VinVL (Zhang et al., 2021), ALBEF (Li et al., 2021), and BM25 (Robertson et al., 2009). Although they are not de- signed for multi-modal candidates, benchmarking these methods facilitates understanding of the importance of single modality in the retrieval pro- cess. We do not consider single-stream approaches like UNITER (Chen al., 2020) as they are not efficient for large-scale retrieval and result in ex- tremely long execution time (see Appendix A).
8mn, = [Q((m) I (Fnn))((wrrii + we whnn)(hm) (3)
= W1 mm + u2sm.
Specifically, CLIP (Radford et al., 2021), BLIP (Li et al., 2022), and combination of models like CLIP and BM25 (Robertson et al., 2009) belong to this category.
Feature-level Fusion In Sec. 6, we show that score-level fusion a compromised choice for en- coding multi-modal candidates. Therefore, we pro- pose a modified version of BLIP (mBLIP) to fuse features throughout the encoding process. The over- all fusion process can be abstracted as follows:
6.2 Evaluation Metries
We evaluate retrieval models with the standard met- ric Recall@k (R@k) k) that computes the recall rate of the top-k retrieved candidates. l is to 1,5, 10. We report mean Average Precision (mAP) to re- flect the retrieval precision considering the ranking
So(in, kn.) = [ur(n,, Ji((n))
(5)
8m, = fm )T (hn: fi()))

---

position of all relevant documents. Formally,
rel EDIS Test
Methods
Fine-tuned tuned
R
R
R 10
mAP
NDCG
1
Full Distrnetor
BLIP
18.4
46.6
59.0
37.8
58.1
Recall @
BLIP
32.6
62.0
72.1
53.8
67.3
Q
rel(m,
mBLIP
27.8
66.0
81.4
56.2
75.3
Q
BLIP BLIP
5.8
17.3
24.0
13.4
32.4
1
En P(m, m)rel(m,
13.1
29.6
37.6
23.4
38.9
mAP-
QQ]
En rel(m, m)
(8)
mBLIP
12.3
33.3
44.1
27.4
47.9
Table 2: Retrieval performance BLIP and mBLIP un- der the distractor and full settings. We use grid search the validation split to find the best score fusion weights (see Eq. 4) for zero-shot and fine-tuned BLIP.
if rel(m, =
rel(m, m)
otherwise
where P(m, n2) is the Precision @n of a query 9m- For R@ and mAP, candidates with relevant score 3 are positive candidates, while candidates with scores 2 or are (hard) negative samples. These two metrics reflect the model's ability to retrieve the most relevant candidates, which aligns with the definition in Fig.
proaches on BLIP (Li et al.. 2022). Table 2 com- pares the result under two different setups where "BLIP" denotes the score-level fusion using the original BLIP architecture, and "mBLIP" denotes our proposed feature-level fusion. For score-level fusion, we obtain the weights from a grid search on the validation set under the distractor setup.
To give merits to candidates with score of we also report Normalized Discounted Cumulative Gain (NDCG). NDCG assigns importance weights proportional to the relevance score so that ranking score 2 candidates before score candidate will lead to higher metric value.
Distractor Set Pre-trained BLIP achieves 18.4 R@ and 46.6 R@5, which means that almost one- third of the queries have positive candidate in top-1 results, and around half of the positive can- didates are retrieved in top-5 results. After fine- (10) tuning, BLIP doubles R@) to 32.6 and achieves significant gain in other metrics. The improvement shows that entities EDIS are out-of-domain con- (11) cepts for zero-shot BLIP, and EDIS training split is useful for models to adapt to the news domain. mBLIP outperforms BLIP in all metrics except R @ 1. The overall improvement entails that feature- level fusion is superior to score-level fusion by utilizing headlines more effectively. The degrada- tion in R can be attributed the fact that the image-query alignment accurate enough for a small number of queries. Therefore, utilizing head- lines slightly harms the results as they only provide high-level summarization.
DCG(m) = Σ rel(m, n) 1
log2(1 + +n)
[QI DCG(m)
NDCG = 1
TDCG(m)
where IDCG(m) is the DCG value of 4m with the ideal candidate ranking.
6.3 Implementation Details
For BLIP fine-tuning, we adopt the same loss and hyperparameters as reported in the original imple- mentation. We increase the learning rate to le-5 for optimal validation results. We directly rank the can- didates by computing the cosine similarity of query features and candidate features and do not use any linear regression heads for reranking. Therefore, we abandon the image-text matching (ITM) loss in mBLIP fine-tuning and increase the learning rate to 5e-5 for optimal performance. More details can be found in Appendix A.
Full Set Retrieving from the full candidate set significantly degrades the performance by over 50% all metrics. Though the distractor setup was widely adopted in previous work, we show that a larger candidate set imposes remarkable challenges to the SOTA models. We can observe similar trends by comparing the three variants of BLIP. mBLIP achieves over 17% relative improvement across all metrics except R 1, even more significant than 4- 12% relative improvement under the distractor set. The degradation R is also much less severe. Therefore, feature-level fusion is a more effective
7 Experimental Results
7.1 BLIP-based fusion methods
We first investigate the performance difference be- tween score-level and feature-level fusion as men- tioned in Sec. 5. We implement these two ap-

---

EDIS Val
EDIS Test
Methods
Fine-tuned
fn
fi
R@1
R@5
R@
mAP
NDCG
R@1
R@5
R 10
mAP
NDCG
Upper Bound
60.1
97.1
100
100
61.0
100
100
BM25
X
6.7
22.2
29.5
20.0
54.1
6.9
21.7
28.8
19.4
52.9
CLIP
X
8.1
28.6
40.5
25.9
58.7
8.2
27.7
39.1
24.8
57.5
BM25+CLIP
13.4
46.8
37.7
32.5
CLIP
30.7
68.0
80.5
58.2
73.8
30.1
68.1
80.2
57.2
73.3
BLIP
V
32.0
62.1
72.2
53.9
67.7
32.6
62.0
72.1
53.8
67.3
mBLIP
V
27.4
65.9
81.5
57.0
75.8
27.8
66.0
81.4
56.2
75.3
BM25
X
4.7
13.3
17.3
11.9
39.4
4.7
12.9
16.6
11.4
38.0
CLIP
X
4.6
13.9
18.6
12.6
38.9
13.4
18.2
12.4
28.2
BM25+CLIP
V
6.8
18.8
23.6
16.6
45.9
6.3
18.0
22.7
15.9
45.0
CLIP
/
/
14.1
35.4
45.7
28.3
45,9
13.7
36.0
47.2
28.0
46.0
BLIPP
12.3
29.0
37.8
22.9
38.8
13.1
29.6
37.6
23.4
38.9
mBLIP
13.3
34.1
44.3
28.9
12.3
33.3
44.1
27.4
47.9
Table 3: Evaluation results on additional baselines.
way to encode multi-modal candidates, considering that users usually receive more than one searched image in reality.
EDIS Test
Methods
fs
fr
R@
R@5
R@ 10
mAP
NDCG
BL.IP
/
6.6
21.6
30.5
19.7
$1.9
Distractor
BI.IP
x
33.9
61.2
71.3
54.0
66.5
BLIP
32.6
62.0
72.1
53.8
67.3
7.2 Additional Baselines
mBLIIP
8.8
29.1
40.1
26.0
$7.9
Table 3, the defective recall rates of BM25 and
mBLIP
x
22.2
49.9
60,2
40.9
58.2
mBLIP
27.8
66.0
81.4
56.2
75.3
CLIP lext encoder imply that headlines solely are insufficient for accurate retrieval. However, text- based retrieval achieves promising NDCG values, indicating that headlines are useful for ranking score 2 candidates higher positions.
Table 4: Ablation study on the effectiveness feature fusion in BLIP and mBL.I.
score 2 candidates higher. We conjecture that many score 2 images have insufficient entities, resulting in lower query-image similarity scores. Hence, models must rely on headlines to simultaneously recognize entities from multiple modalities.
Score-level Fusion "BM25+CLIP" first ranks candidates using BM25 and then reranks the top 50 200 candidates with CLIP utilize the images. Despite the improvement compared to text-base based methods, underperforms zero-shot CL.P or BL.IP. This implies that ranking with query-headline sim- ilarity imposes bottleneck on the reranking pro- cess. CLIP achieves the best performance in terms of R@ 1/5/10 and mAP compared to other meth- ods. We hypothesize that the "CLIP filtering" step in Sec. 4.1 eliminates hard negative query-image pairs for CLIP and thus introduces performance bias towards CLIP. Fine-tumed CLIP does not show apparent improvement and thus is not shown in Table 3. Therefore, EDIS is still challenging for SOTA retrieval models.
7.3 Ablation Study
Component Analysis Table shows the perfor- mance two fusion approaches without either im- age or headline branch. BLIP achieves much lower performance when relying solely on query-headline headline alignment (6.6 R @ 1, 29.7 mAP) compared to uti- lizing images only (33.9 9 R @ 1, 54.0 mAP). BLIP only achieves comparable and slightly degraded performance when using images and headlines for score fusion. Therefore, score-level fusion cannot easily tackle multi-modal candidates in EDIS.
Feature-level Fusion mBLIP consistently out- performs other approaches in NDCG regardless of the candidate set scale, achieving 75.3/47.9 NDCG under distractor/full set. mBLIP fuses headline features with visual features more effectively and thus proves that headlines are critical for ranking
In contrast, mBLIP shows improved perfor- mance with the headline encoder while decreased performance with the image encoder only. This is intuitive as the BLIP fine-tuning process only uti- lizes images without headlines, yet mBLIP utilizes both images and headlines. More interestingly,

---

Query: The Westerdam cruse ship docked in Sihanoukville, Cambodia, Monday, Rank 1
Rank 2
Rank
Rank
BLIP Scone-level Fusion
Oer ontection of ooe of the best mas thattyrphhree
A appeent of francce stordips hes pacond tirrugh the ting chamed "modtre"
Aledd secrd pooter wertt Tanddy Cantaal bar crriee op
NETI Namoaggtnn Ohnin thi seatingtt th Nerth Ameetts hand ongot Hhampo
Traandog anottd abs:
Eaias denas Cio du
dhinig
Nery
on
mBLIP Feahure-leva Fusion
Vina leie ris Comroddees
daj ovar
Cast Cut
Canbata's Creaoaaaeeaaaaaeeeeereeeee
adianá dadiin
Doda
Query: John
Hillary campuign looks before presidential debata Farnville, Oct 2016 Rank
Rank
Rank 5
Score-Jevel Fusion
1ith Frisetre Awarde meggeda maager
Domarntt cuciiim Clenre Siee
Loks Hal buicaly aniill at produatid ohiat
Pronk Yorkewursg pellrca and rends husder encat
oppeetle dendtre en snggl party crimel
CNN
Peatures level Fusion Claass poisay. Sanal ia Breskiou Thansky dagiica carl Desacceds Ldxas
Hilley Clerre only rry sbeggrai
Danaorell Edbaoo
Figure 5: A success (top) and a failure case (bottom) of mBLIP compared to BLIP.
when using both image and headline encoders, mBLIP demonstrates over 20% relative increase in all metrics. The results imply that feature-level fusion is a more effective method to combine can- didate features from multiple modalities.
results. The third candidate mBLIP shows the image with the correct person but from a different event. It further proves that the EDIS is a chal- lenging dataset that requires specific knowledge of entities, cross cross-modal entity matching, and event understanding.
7.4 Case Study
Success Case We show one success case and one failure case of mBLIP in Fig. 5. the success case (top), mBLIP manages retrieve all four relevant images while BLIP retrieves five false positives. Since all ten images contain a "cruise", we conjec- ture that entities in headlines (c.g., "Cambodia", "Sihanoukville") play critical role for mBLIP to outperform BLIP in this case. The case shows feature-level fusion is much more effective in uti- lizing headline features than score-level fusion.
Conclusion
Training and evaluating large-scale image retrieval datasets an inevitable step toward real image search applications. To mitigate the gap between existing datasets and real-world image search chal- lenges, we propose large-scale dataset EDIS with novel retrieval setting and million candidates. EDIS queries and candidates are collected from the news domain describing abundant entities and events. EDIS candidates are image-headline pairs since realistic image search utilizes the surround- ing text of an image to facilitate accurate searching results. As primary step towards handling multi- modal candidates in EDIS, we review two primary fusion approaches and propose feature-level fu- sion method utilize the information from both images and headlines effectively, Our experimen- tal results show ample space for improvement on EDIS. Future work should consider more princi- pled solutions involving knowledge graphs, entity
Failure Case As for the failure case in Fig. (bottom), BLIP and mBLIP fail to retrieve the pos- itive candidates in the top-5 results. Both meth- ods fail to recognize "John Podesta" and align the text with the visual representation. For example, the top-2 candidates retrieved by mBLIP depict different person from different event. "Hillary Clinton" becomes distracting entity the query, and the model must understand the event instcad of just matching entities to achieve accurate retrieval

---

linking, and training algorithm design.
the datasets that have newly annotated. As introduced in Sec. 4.2, we annotated the data with crowd-workers through Amazon Mechanical Turk. The data annotation part of the project is classified as exempt by our Human Subject Com- mittee via IRB protocols. We required the work- ers to be in English-speaking regions (Australia, Canada, New Zealand, the United Kingdom, and the United States). We keep the identity of workers anonymized throughout the collection and post- processing stages. We also require the workers to have a HIT approval rating 2 96% or higher. We pay each completed HIT $0.2, and each HIT takes around 30-40 seconds to complete on average. Therefore, this resulted in an hourly wage of $ $18- $24, as determined by the estimation of completing time for each annotation task. Example screenshots of our annotation interface can be found in Fig. 6-7 under Appendix A.
9 Acknowledgement
The work was funded by the National Science Foundation award #2048122. The writers' opin- ions and conclusions in this publication are their own and should not be construed representing the sponsors' official policy, expressed inferred.
10 Limitations
In this study, we only cover image retrieval datasets with English instructions. Queries and headlines in other languages may characterize different types of ambiguity underspecificattio. Thus, expand- ing the datasets multi-lingual image retrieval based on our dataset is important. Secondly, we only consider the news domain to collect entity- rich queries and images. We plan to expand our dataset to open-domain where other entities like iconic spots will be included. In addition, we only consider the headlines as the text information to utilize in the retrieval process. However, in real im- age search scenarios, search engines usually utilize multiple paragraphs of the surrounding text deter- mine the relevance of the image. In the future, we will expand the text of the multimodal candidates with news articles or segments of the articles. Our dataset and models trained on it could be biased if the model is not accurate enough. The model may return completely incorrect candidates and cause users confuse persons objects with in- correct identities. We will provide all ground truth annotations with visualization code help users learn about the ground truth candidates. Last but not least, we do not consider the phenomenon of underspecification in the image search experience. Users search with phrases or incomplete sentences to save typing efforts. Therefore, more realistic queries can be underspecified and grammatically incorrect. However, this is problem universal to all existing image retrieval datasets, as collecting real human search results could be challenging. We plan to make our dataset more realistic in the future by utilizing powerful tools such as large language models to generate underspecificd, near-realistic queries.
References
Max Bain, Arsha Nagrani, Gäl Varol, and Andrew Zis- serman. 2021. Frozen time: A joint video and image encoder for end-to-end retrieval. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1735
Yingshan Chang, Mridu Narang, Hisami Suzuki, Gui- hong Cao. Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. In Proceed- ings of the IEFT/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 16495-16504.
Soravit Changpinyo, Jordi Pont-Tused Vittorio Ferrari, and Radu Soricut. 2021. Telling the what while point- ing to the where: Multimodal queries for image re- trieval. In Proceedings of the IEEF/CVF Interna- tional Conference Computer Vision, pages 12136- 12146.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104-120. Springer.
Mengjun Cheng. Yipeng Sun, Longchao Wang, Xiong- wei Zhu, Kun Yao, Jie Chen, Guoli Song, Junyu Han, Jingtuo Liu, Errui Ding, al. 2022. Vista: Vision and scene text aggregation for moos- re- trieval. In Proceedings of the IEFF/CVE Conference Computer Vision and Pattern Recognition, pages 5184 1184-
11 Ethics Consideration
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. 2022. An empirical study of training end-to-end
We will release our dataset EDIS for academic purposes only and should not be used outside of research. We strictly follow any licenses stated in

---

vision-and -anguage transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18 66-18177
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare. Shafiq Joty. Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems. 34:9694-9705.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin,
William Yang Wang. Lijuan Wang, and Zicheng Liu. 2021. Violet: End-to-end video-language trans- formers with masked visual- token modeling. arXiv preprint (tt::2111.1260
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint
Xingyu Fu, Ben Zhou, Ishaan Chandratreya, Carl Von- drick, and Dan Roth. 2022. There's sa time and place for reasoning beyond the image. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers).
Xiujun Li. Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object- semantics aligned pre-training for vision-language tasks. In Eumppan Conference on Computer Vision, pages 121-137. Springer.
pages 1138- 1149.
Saadia Gabriel, Skyler Hallinam, Maarten Sap, Pemi Nguyen, Franziska Roesner, Eunsol Choi, and Yejin Choi. 2022. Misinfo reaction frames: Reasoning about readers' reactions to news headlines. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers). pages 3108-3127.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects context. In European confer- ence on computer vision, pages 740-755. Springer.
Conghui Hu and Gim Hee Lee. 2022. Feature represen- tation learning for unsupervised cross-domain image retrieval. In European Conference on Computer Vi- sion, pages 529-544, Springer.
Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2021. Visual news: Benchmark and chal- lenges in news image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6761-6771
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun Hun-a Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMLR.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-lange lnn-language tasks. Ad- vances in neural information processing systems, 32.
Sheena Panthaplackel, Adrian Benton, and Mark Dredze. 2022. Updated headline generation: Cre- ating updated summaries for evolving news stories. In Proceedings of the 60th Annuul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36661
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Wa- ters, and Yinfei Yang. 2021. Crisscrossed captions: Extended intramodal and intermodal semantic sim- ilarity judgments for MS-COCO. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 2855-2870, Online, Association for Computational Linguistics.
Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-an -nd---nd--ndaa languae transformer without convolu- tion region supervision. In International Con- ference on Machine Learning, pages 5583-5594. PMLR.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal en- coder for vision and language by cross-modal pre- training. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336- 11344.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to to to-haasse correspondences for richer image- to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language- -mmgg pre- training for unified vision- language understanding and generation. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900. PMLR
Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. 2020. Connecting vision and language with localized narratives. ECCV.

---

Alee Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748 8863. PMIR.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision. In International Conference on Learning Representations.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. VinvI: Making visual representa- tions matter in vision-language models. CVPR 2021.
Revanth Gangi Reddy, Sai Chetan, Zhenhailong Wang. R Fung, Kathryn Conger, Ahmed Elsayed, Martha Palmer, Preslav Nakov, Eduard Hovy, Kevin Small, et al. 2021. Newsclaims: A new benchmark for claim detection from news with attribute knowledge. arXiv preprint arXiv: 212.::1.....5....
A Appendix
A.1 Additional Implementation Details
Hyperparameters We fine-tuned BLIP and mBLIP on 4 40GB NVIDIA A100. It takes 5 hours for BLIP fine-tuning and 3 hours for mBLIP. For both BLIP and mBLIP, we train the model for 6 epochs with batch size 16 per GPU. The model checkpoint with the best recall rate over the valida- tion set selected for final evaluation. We apply grid search for score-level fusion using BLIP or CLIP to find the optimal W1. We first search over ten intermediate numbers between and and then narrow the range to search for 100 intermediate numbers. Finally, we found training and valida- tion results stable without much randomness for all implemented methods. Therefore, we evaluate every model once and report the metric values of one-time evaluation.
Nils Reimers and Tryna Gurevych. 2019. Sentence-bert. Sentence embeddings using siamese nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 3982-399 1992.3992.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ in Information Re- trieval, 334:3333.38.
Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, and James Hays. 2022. sketch worth thousand words: Image retrieval with text and sketch. In Eu- ropean Conference on Computer Vision, pages 251- 267. Springer.
Alexander Spangher, Xiang Ren, Jonathan May, and Nanyun Peng. 2022. Newsedits: A news article re- vision dataset and novel document- level reasoning challenge. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 127-157.
BLIP Training For BLIP fine-tuning, we follow the original implementation and adopt the original image-text contrastive (ITC) loss and image-text matching (ITM) loss. We only utilize images with scores of 3 and text queries for training. As for mBLIP, the headline encoder and the query encoder share the same weight. We utilize images with scores of 3, associated headlines, and text queries for training. The output from the image encoder is fed into the transformer layers of the headline encoder through cross-attenticn layers. Then the outpul of the headline encoders can be treated as the fused feature of image-headline pairs. We compute ITC loss based on the headline encoder outputs and the query encoder outputs.
Reuben Tan, Bryan A Plummer, Kate Saenko, JP Lewis, Avneesh Sud, and Thomas Leung. 2022. Newssto- ries: Illustrating articles with visual summaries. In European Conference on Computer Vision, pages 641. Springer.
Alasdair Tran, Alexander Mathews, and Lexing Xie. 2020. Transform and tell: Entity-aware news image captioning. In Proceedings of the IEEF/CVF Confer- ence on Computer Vision and Pattern Recognition, pages
Daniel Varab and Natalie Schluter. 2021. Massivesumm: a very large-scale, very multilinuual, news summari- sation dataset. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 10150-10161.
Single Stream Models We do not evaluate any single stream models or modules due to time com- plexity. Consider queries and candidates. The complexity for a dual encoder model to obtain all features Omm+n). The computation cost of com- puting cosine similarity is trivial compared the forward process of model and can be neglected. However, for a single stream model, it takes 0(m))
Yuxuan Wang, Difei Gao, Licheng Yu. Weixian Lei, Matt Feiszli, and Mike Zheng Shou. 2022. Geb+: A benchmark for generic event boundary captioning, grounding and retrieval. European Conference on Computer Vision, pages 709-725. Springer.

---

obtain similarity scores for all query-candidate pairs. Since it takes around 3.5 minutes for BLIP to evaluate over 3.2k queries with 25K candidates, it is taking more than days for a single encoder model to complete retrieval under the distractor set- ting. It takes more than a year to complete retrieval under the full setting.

---

You are given Query eight image-text samples. task determine each image-text matches the Query.
Please read the Query carefulty.
For each image-txx text cample, give score on how much matches the Query. understand the background information.
Facus images
How to rate ecore:
CORRECT, subject, 2: Image is relevant to the query, but some entities are missing incorreat.
EVERYTHING
Image incorrect or missing the semantic meaning the query.
Notice: There may be additional laize semplee with an expected score of inserted to verily your atlention.
Take look the examples below.
Example
An examplo of relevance scare 3:
Query: American troops landed at the National Palace in Port-au- Prince, Halti, Tuesday. They bagan ralling through the capital and assisting with the relief operation.
Patrol
mary
Troops (corract location, event, action)
Sayy
(wrong event)
(corrct action wrong location)
Desperate Boarch for Victirs Explosion Connestioul
tuyy Humariar un Assistance Responss (HADR)
8010 Halt eerthquske
(wiong location and unrolated event)
(correc location, event, and action)
2 (porrect location but wrong time)
Example
exammpe
Query: Protesters in Katmandu tried break police line on Friday to press legislators to work for consensus a new constitution.
Clock Tickss Nepal to Settle Future
Hong Kong protests signal alam specal freecims tacing
30,000 Protect Govemment's Plan Consiitutinn in Nepal
Photesters againet US grant dishh with polie Nepal 2 (correct location and awant, but no action foreak police line")
(wrong location)
(wrong location)
Nesu
Nopal election
souffe ant cnfi-contilttion protester g
Benux
2 (correct location and but action foreak police line")
(wrong location and unrelsted event)
(correct location, event, and actio)
Figure 6: Amazon Mechanical Turk interface with annotation examples for crowd- to read

---

You are given Query and eight image-text samples. Your task is to determine each image-text matches the Query.
Query caraluily
For each image-tent text sample, give score on how much matches the Querry.
images
background intormation.
How rate a score:
image
Image is relevant to the query, but some entities are missling or Incorrect.
image incorret or missinn semantic meaning query.
Notic:: Thare may be additional taisa samples with an expected scora Insarted to varty your attention.
Take look the two examples below.
Example
Example
Complete the tollowing:
An exampple relevance score 3:
Query: Audiences react standup comedians performing called AisiTaisi Democracy at the Canvas Laugh Club at The Palladium Mall in Mumbai
Arta & Paclici-dia da boo ming stand comedy conno tocts bounddrree with outling edge jiies
Auclences mact shandup comedd partorming Pheto
Desplta lowor venues, Mumbal witnesses encouitging nigingg tumout for lvve cortedy shows
Indla's booming ngg standkup comedy ecens tnats boundariees outlnngelgge jokes
200 Pa ladium Maxl Pholoe and Phemium High Pictumes
AUGH CLUB
COUPLES, GURGAON AUR DAARU NOUP CONCDY
Alghan 1
Distanding Changed Band-up Gomedy
comocians
Couples Gurgaon our Deers Standup Comedy
Laugh
0
2
O
O
Figure 7: Amazon Mechanical Turk interface and annotation task

---

Query: Six royal gold coins recovered from Spanish shipwreck near Florida
H
INCENTSAMA
Text: Florida treasure hunters found a trove of S4m Score:
Text: Florida treasure hunters find $4.5m rare Spanish coins Florida Score:
'Txt: Fla. family finds $1M of sunken Spanish treasure Score: 2
Police vehicles front of Deutsche Bank headquarters in Frankfurt on Thursday. Prosecutors raided the companys office in case related to the Panama Papers
Query:
Deutsche Bank
Deutsche Bank Cuts Workforce, Exits Countries
Text: Deutsche Bank Oflices Are Searched in Money Laundering Investigation
Text: Police raid Deutsche Bank headquarters as part of Panama Papers money laundering investigation
Text: Deutsche Bank said Thursday it would shed 35,000 jobs and close operations in 10 countries
Score:
Score:
Score:
Query: Barack Obama accompanied by first lady Michelle was making his first visit to Fort Bragg
UNITE STAT
Text: President Barack Obama marked the end of the Iraq war with tribute the troops who fought and died in conflict he opposed from the start Score:
Text: President Obama and First Lady Michelle Obama Return from FL. Bragg
Text: First Lady visits Fort Bragg, support for military families Score:
Score:
Figure 8: Additional EDIS dataset examples set

---

Query: A rally in front of the New Jersey State House on Thursday, when the authorization for the state's ss transportation trust fund expired
Text: With No Deal New Jersey Gas Tax, Christie Orders Shutdown of Road Projects Score:
Text: Aftermath of the 2021 United States Capitol attack Score:
Text: TEAMSTERS 320 Score:
Query: Unloading goods Lianyungang, China. The United States trade deficit with China
soared in the first three months of this year
Trade with
Text: A Temporary U.S.-China
Text: The US trade deficit with China has hit a record high, fuelling tensions between the countries over currency imbalances
Text: Will China Reverse Its Trade Surplus with the United States Score:
Trade Truce Starts Look Durahle
Score:
Score:
Query:
Femke Van Den Driessche during the world cyclocross championships in HcusdenZolder Saturday A concealed motor later found on her bicycle
Text: Chris Froome has said
Text: UCI confirms motorised bike at Cyclo-cross World
Text: Controversy as motor found inside bike at the world cyclo-cross championships Score: 2
raised his concerns
motorised doping to the
Championships
authorities year ago
Score:
Score:
Query: Diane Keaton wins her Oscar for 1977 S Annie Hall
Figure 9: Additional EDIS dataset examples set

---

annie hall
Text: The actress Greta Gerwig surely deserves to receive the keys New York in recognition of her on screen celebrations of that city Score:
Text: Diane Keaton Says It's All Because of 'Annie Hall' Score:
Text: Annie Hall DVD 1977 Woody Allen Diane Keaton Oscar Winner Best Picture LIKE NEW Score:
Query: A dog rescued last month from farm in Wonju, South Korea. Humane Society International offers to pay farmers to release dogs so they can be sent abroad to be adopted
IIII
Text:
From Dog Farms in South
Text: From Dog Farms in South
Text: Flagler Humane Society relates story of South Korean dogs arriving US Score: 2
Korca to New Lives as Pets Abroad
Korca to New Lives as Pets Abroad
Score:
Score:
Query: Hubble Telescope view reveals Uranus surrounded by its four major rings and 10 of its
17 known satellites
Text: Speaking of ScienceHubble captures a shimmering butterfly in space
Text: Hubble Finds Rings
Text: Scientists Revisit Old Data, Discover New Moons Around Uranus
Uranus Orbit
Score:
Score: 3
Score:
Figure 10: Additional EDIS dataset examples set

---

Query: Matthew Centrowitz won the first US gold men 1500 since 1908
USA
Text: Mau Centrowitz wins gold men's 1,500; first USA 108 years Score:
Text: Mau Centrowitz Takes
Text: Robby Andrews upsets Olympic gold medalist Matl Centrowitz in 1,500 U.S. championships
I, 1,500; First U.S. Winner Since
1908: The Torch NPR
Score:
Score: 2
Federal Reserve Chair Janet L Yellen testifics Wednesday before the House Financial Services Committee in Washington
Query:
FINANCIAL SeRvIces
Oversight Treasury Department's Pandemic Response
Tuesday Marsh 23, 2001
Text: Wonkblogl awmakers grill Fed chief in testy hearing on Capitol Hill
Text: Andrew Ross Sorkin: Janet Yellen, Fed 'go it wrong' on inflation, were 'a little political' in their response
Text: Virtual Hearing Oversight Treasury Department's and Federal Reserve's Pandemic Response Score:
Score:
Score: 2
Query: Ben Fogle holds an Olympic torch at the Eden Project near Bodelva Cornwall
Text: It was meant to be a chance for ordinary people across Britain
Text: TV adventurer Ben Fogle
Text: Olympic flame goes out in relay
set to swim the Atlantic
to inspire others with the Olympic Score: 2 spirit
Score:
Score:
Figure Additional EDIS dataset examples set


Reviews:
[{"review_id": "aCH7I7OL1j", "reviewer": "Reviewer_eaas", "summary": "", "questions": "- will you release the new collected EDIS dataset?\n- Please refer to the previous section.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and feedback. \n\n> mBLIP is a minor modification. The proposed mBLIP does not always work. \n\n\nWe would like to highlight that our contribution goes beyond mBLIP, and a significant emphasis is placed on the EDIS dataset which is a large-scale entity-rich retrieval dataset in a practical setting. Compared to BLIP, mBLIP improves all metrics by relatively 4-12% on the distractor set and 15-26% on the full set except R@1. We conjecture that the degradation attributed to the fact that the image-query alignment is accurate enough for matching queries with the most similar candidates. Therefore, introducing high-level headlines slightly harms the results. Note that the degradation in R@1 is much less severe in the full set and mBLIP even improves R@1 for the validation set, which proves that mBLIP is more effective than BLIP in a more practical setting. \n\nAs for CLIP, we believe that the gap originates from the differences in pre-training data for BLIP and CLIP. Note that BLIP is pre-trained on COCO+VG+Conceptual3M+SBU+Conceptual12M where named entities are deliberately filtered out deliberately filtered out from the text. Though the pre-training data of CLIP is unknown, we find out that CLIP is inherently better than BLIP in understanding named entities. As a proof, we run image retrieval on the full TARA [1] development set. CLIP outperformed BLIP by a large margin. As EDIS is an entity-rich dataset, the gap is caused by the pre-training stage instead of mBLIP design or fine-tuning strategy.\n| Model       | R@1         | R@5         | R@10        |\n| ----------- | ----------- | ----------- | ----------- |\n| CLIP        | 66.42       | 87.83       | 92.27       |\n| BLIP        | 41.73       | 66.36       | 75.55       |\n\n\n> Table 4, we can find that BLIP performs better than mBLIP in two situations.\n\nFor the gap in R@1, please refer to our response to the first reason to reject. The gap at R@1 in Table 4 is the same as in Table 3 while the purpose of Table 4 is to demonstrate the effectiveness of our feature-fusion design in mBLIP.\n\n> Evaluation on MSCOCO or Flickr30K\n\nThank you for your feedback. While MSCOCO or Flickr30K are standard datasets for image-text retrieval, please note that 1) these datasets do not have the “headlines” for mBLIP to perform multi-modal fusion. The candidates are single modalities, either image (if performing text-to-image retrieval) or text (if performing image-to-text retrieval). EDIS addresses a different setting where each candidate contains multimodal information (image+headline). 2) These datasets only contain objects of general categories instead of named entities.\n\n> Questions: will you release the EDIS datasets\n\nPlease be assured that we will release the full datasets, including raw images, annotation files, and the mBLIP scripts after the anonymity period.\n\n[1] Fu, Xingyu, et al. \"There’s a time and place for reasoning beyond the image.\" ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper collected a new large-scale dataset EDIS, and evaluated several existing methods.  ", "reasons_to_reject": "- The author only proposed a modified BLIP method which is a minor modification. Based on the results of Table 3, we found that the best results usually come from CLIP and BLIP, not their proposed new method. It looks like their proposed method does not always work. \n- on Table 4, we can find that BLIP perform better than mBLIP in two situations. It also demonstrates my previous conclusion.\n- I think the authors can also evaluate their proposed method on MS-COCO and Flikr30K with other SOTAs.", "questions_to_authors": "- will you release the new collected EDIS dataset?\n- Please refer to the previous section.", "paper_topic_and_main_contributions": "This paper is about Entity-driven image search topics, and the main contributions of this paper are:\n\n1. They collect and annotate EDIS for large-scale image search, which characterizes single-modality queries and multi-modal candidates.\n2. They propose a feature-level fusion method for multi-modal inputs before measuring alignment with query features. \n3. They evaluate several existing methods on the new EDIS dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "0FFVAYnjPS", "reviewer": "Reviewer_q3bm", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and acknowledgment of our work. Please let us know if you have any questions or concerns. We are always happy to address them in the discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and it contributes a useful artifact, i.e., the EDIS dataset.\n", "reasons_to_reject": "I don't see any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors propose a dataset called EDIS with 1 million images for cross-modal image search in the news domain.\nEDIS dataset includes images and text with entities and events. Moreover, the authors propose a feature-level fusion method for multi-modal inputs. Existing approaches are evaluated on the new dataset.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "I didn't identify any missing references.", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 405: Distrator Set -> Distractor Set"}, "extra_text": {}}, {"review_id": "B5uIvKaoFw", "reviewer": "Reviewer_QYbK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and comments. \n\n> How to evaluate the understanding of the entity names?\n\nPlease note that understanding the entity names in the query requires both grounding to the images, matching entities and understanding events in the headlines. For example, in the Fig. 1 Rank 2 image, EDIS requires the model to ground Rick Santorum to the visual figures and understand that Rick Santorum is missing from the image. In the Rank 4 headline, EDIS requires the retrieval model to understand that “the New Hampshire Republican debate” in the query is different from the “Final GOP Debate on CNN” in the headline. \n\nFor now, the understanding of entity names is implicitly reflected in the reported Recall, mAP, and NDCG metrics. A higher metric score usually represents a better understanding of the named entities and grounding abilities. A possible direct evaluation would be masking out each of the named entities in the text query, and computing the change in metrics scores to see the influence of each entity on the retrieval results. \n\n\n> News corpus is of high quality while real applications includes noisy web data.\n\nWhile the real web environments could consist of noisy web data, we are not targeting at the problem of multi-modal retrieval under noisy data or how to learn from noisy data to perform accurate retrieval. Our work mainly addresses the challenges in understanding named entities and events and grounding them to multi-modal candidates in a large-scale. The introduced high-level headlines are challenging for existing models to effectively fuse the information with the image modality. Thank you for your suggestions and we will consider more noisy web data as a future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposed a new task to enhance the entity recognition \n- A new dataset is collected which can facilitate future research and evaluation\n- Experiments are conducted on the dataset to show the ", "reasons_to_reject": "This paper claims three expectations in the abstract. However, the task formulation and the collected dataset cannot evaluate these aspects properly. For example, how to evaluate the understanding of the entity name? Is it grounded in an image or simply in the headline? In addition, the news corpus is usually of high quality in terms of both format and expression. However, the real application includes much more noisy web data.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new retrieval task from textual query to the pair consisting of an image and corresponding headline. To implement this, the authors modified the architecture of BLIP to allow the fusion of image and headline. Furthermore, they propose a large-scale dataset to evaluate the task.  ", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper brings the EDIS dataset that can evaluate the image retrieval task in news domain. This news domain challenges the model with more common sense knowledge and also fine-grained entity understanding. The reviewer raises the concern of the lack of novelty in proposing methods, which I partly agree with but the main novelty of the dataset paper will be mainly evaluated based on the data. Methodology would be considered more as a bonus.

Overall I feel that this dataset can be helpful for the community given that the authors promise on releasing dataset.
Decision: Accept-Main