An Empirical Investigation of Implicit and Explicit Knowledge-Enhanced Methods for Ad Hoc Dataset Retrieval
Authors: Weiqing Luo
Qiaosheng Chen
Zhiyang Zhang
Zixian Huang
Gong Cheng
Venue: EMNLP 2023 Findings (2023)
Keywords: ad hoc dataset retrieval
dataset search
dense retrieval
semantic search
TL;DR: We investigate whether existing dense reranking and KB-based reranking methods are effective in ad hoc dataset retrieval.

Abstract: Ad hoc dataset retrieval has become an important way of finding data on the Web, where the underlying problem is how to measure the relevance of a dataset to a query. State-of-the-art solutions for this task are still lexical methods, which cannot capture semantic similarity. Semantics-aware knowledge-enhanced retrieval methods, which achieved promising results on other tasks, have yet to be systematically studied on this specialized task. To fill the gap, in this paper, we present an empirical investigation of the task where we implement and evaluate, on two test collections, a set of implicit and explicit knowledge-enhancement retrieval methods in various settings. Our results reveal the unique features of the task and suggest an interpolation of different kinds of methods as the current best practice.

An Empirical Investigation of Implicit and Explicit Knowledge-Enhanced Methods for Ad Hoc Dataset Retrieval
Weiqing Luo, Qiaosheng Chen, Zhiyang Zhang, Zixian Huang, Gong Cheng"
State Key Laboratory for Novel Software Technology, Nanjing University, China (wqluo, qschen, zhinlngzhnhyyggggan ctapeeaaanniiiiiiiiiiiiii jixiaanaanngeesaaillll nju. edu.edu.ueeeeeee cn, gchengênju. edu. cn
Abstract
Table 1: A query and the metadata of a relevant dataset.
Ad hoc dataset retrieval has become impor- tant way of finding data on the Web, where the underlying problem how to measure the relevance of dataset to query. State-of- the-art solutions for this task are still lexical methods, which cannot capture semantic simi- larity. Semantics-aware knowledge-enhancee retrieval methods, which achieved promising results on other tasks, have yet be systemat- ically studied on this specialized task. To fill the gap, in this paper, we present an empirical investigation of the task where implement and evaluate, on two test collections, a set of implicit and explicit nnowledge-enhancemee retrieval methods in various settings. Our re- sults reveal the unique features of the task and suggest an interpolation of different kinds of methods the current best practice.
Query Title
american protein intake duily
USDA National Nutrient Database for Standard Reference Dataset for What We Eat In America, NHANES (Survey-SR)
Deseription
The dataset. Survey-SR, provides the nu- trient data for assessing dictary intakes from the national survey What We Eat In America...
Tags
food compositoo,,,, cood-consumption....
et al., 2020; Lin et al., 2022). It is well- that such lexical retrieval methods cannot identify mantic matches and hence fail find datasets that are lexically disjointed with but semantically rele- vant to the query. The problem can be alleviated by incorporating knowledge into serrivaal-etthee implicit knowledge encoded in pre-trained lan- guage model (PLM) or explicit knowledge stored in an encyclopedic knowledge base. For exam- ple, to identify the semantic connection between "protein" in the query and "nutrient" in the dataset in Table implicit knowledge-enharee enhanced retrieval methods may achieve by measuring the similar- ity between their word embeddings, while explicit methods may employ the similarity between their linked entities in a knowledge base. However, to the best of our knowledge, the effectiveness of in- corporating knowledge into the emerging task of ad hoc dataset retrieval has not been systemati- cally investigated. In particular, the following three research questions remain open.
1 Introduction
Tens of millions of datasets have been published on the Web (Benjelloun et al., 2020), providing government data, scientific data, etc. Accordingly, ad hoc dataset retrieval is becoming an important specialized information retrieval task (Kato al.. 2021; Lin et al., 2022), aiming at finding datasets that are relevant to a user's query, which ad hoc since the number of possible queries is huge. Due to the magnitude and heterogeneity of dataset con- tents, existing solutions such as Google Dataset Search (Brickley et al., 2019) rely on the retrieval of dataset metadata provided by data publishers for describing their datasets, as illustrated in Table 1 which is a real example taken from the NTCIR-E test collection (Kato et al., 2021).
RQ1. Despite a few preliminary PLM-based implementations (Kato et al., 2021), they were fine- tuned on small training set. Researches on ad hoc datasct retrieval are still in their infancy, and exist- ing test collections provide less than 300 queries for training (Kato et al., 2021; Lin et al., 2022). The performance of PL-d- methods fine-tuned in such an in-domain setting may not be generaliz- able to practical settings. Will implicit knowledge-
Motivation. The metadata of a dataset resem- bles a structured multi-field document, and cur- rent implementations of ad hoc dataset retrieval are mainly adapted from conventional document retrieval methods like BM25 FSDM (Chapman
"Corresponing author

---

enhanced retrieval methods remain effective in ad hoc dataset retrieval in an out-o--domain setting? ? RQ2. The high-quality encyclopedic knowledge bases available today, like Wikipedia and Wikidata, can be used to annotate query and dataset metadata so that their semantic similarity can be measured to enhance lexical retrieval. More importantly, this is unsupervised and not limited by the availability of training data, However, such methods are so far under-studied and their effectiveness remains un- known. Will explicit srowledge-enhaneee retrieval methods be effective in ad hoc dataset retrieval?
dataset provided by its publisher typically con- sists of set of fields such as title and description. While existing retrieval methods commonly rely on metadata (Chapman et al., 2020) which resembles a structured document, the task of dataset retrieval differs from document retrieval and has its unique properties, e.g. queries often mention geospatial and temporal entities (Kacprzak et al., 2019; Chen et al., 2019), and metadata is relatively short and often incomplete (Neumaier et al., 2016).
Knowledge-enhanced retrieval methods have the potential to exploit these features, but they have not been sufficiently studied for this task. Indeed, in a recent benchmarking effort (Lin et al., 2022). only number of lexical retrieval methods were implemented and evaluated. In Kato et al. (2021) and Chen et al. (2023), a few PLM-based imple- mentations were evaluated but were fine-tuned on small training set risking overfitting and their reported performance might not be generalizable.
RQ3. Lexical matching, implicit knowledge, and explicit knowledee-they have the potential to capture different signals in retrieval. While each of them alone may not exhibit superb performance, their appropriate combination may produce better results, e.g.. by an interpolation of their retrieval scores. Will interpolated methods be more effective in ad hoc dataset retrieval?
Our Work and Contribution. To answer the above questions, we conducted systematic investi- gation of implicit and explicit knowledge-cnhanced methods for ad hoc dataset retrieval on two test col- lections (Kato al., 2021; Lin al., 2022). For implicit knowledge, we evaluated five PL.M-based methods in both in-domain and oft-of-domain set- tings. For explicit knowledge, we designed and evaluated methods based on entity similarity com- puted over two knowledge bases. We also explored different interpolation strategies.
Our empirical investigation significantly extends the above evaluation efforts. We systematically evaluate a range of state-of-the-art PLM-based methods for ad hoc dataset retrieval in both in- domain and out-of-domain settings. Moreover, we design and evaluate explicit knowledge-enhancee methods, which are under-studied the literature.
2.2 Implicit Ghowledge-Enhancee Retrieval
PLMs encode knowledge into learnable dense vec- tors (Talmor et al., 2020). PLM-based retrieval, aka dense retrieval, has developed rapidly in re- cent years and exhibited powerful text understand- ing capabilities which helped improve the accu- racy of document retrieval (Zhao et al., 2022). Among others, monoBERT (Nogueira and Cho, 2019) directly leverages the text classification ca- pability of BERT (Devlin et al., 2019) to rank doc- uments. DPR (Karpukhin et al., 2020) adopts a dual-encoder architecture that employs the implicit knowledge in PLM and performs metric learn- ing. Other dense retrieval models such as Col- BERT (Khattab and Zaharia, 2020) and COIL (Gao et al., 2021) further exploit the implicit knowl- edge in PLMs by computing token-level match- ing through multiple vectors. Xiong et al. (2021) proposes ANCE which features dynamic negative sampling to improve the informativeness of train- ing data. Condenser (Gao and Callan, 2021) adopts a novel pre-training architecture to compress infor- mation in the text, Further, coCondenser (Gao and
As the first empirical investigation of this kind, our work fills the gap and our results will provide practical guidelines for researchers and developers working with ad hoc dataset retrieval, or even in- formation retrieval in general. helps establish an empirical basis that will facilitate future studies on this trending information retrieval task.
httpt/hhww.oooo/wwwweeeeeee KnowledgeFnhanceed
Code:
Paper Structure. We will discuss related work in Section 2, describe the evaluated methods in Sec- tion 3, present our experimental setup and results in Section 4 and Section 5, respectively, and finally conclude the paper in Section 6.
2 Related Work
2.1 Ad Hoc Dataset Retrieval
Ad hoc dataset retrieval specialized informa- tion retrieval task that aims to find the most rele- vant datasets to a user's query. The metadata of

---

Callan, 2022) extends Condenser by pre-training with query-ugnostee contrastive loss.
where represents concatenation.
Any dense retriever reviewed in Section 2.2 can be used to compute the relevance of Ta to the query q as the relevance score d:
These state-of-the-art dense retrieval methods have not been applied to ad hoc dataset retrieval. We adapt them to this new task and thoroughly analyze their effectiveness in various settings.
rel q) DenseRetrieva () (2)
which exploits knowledge implicitly encoded by PLMs into learnable dense vectors. Dense retriev- ers are commonly supervised. Fine-tuning can be performed on task-specific training data-referred to as an in-domain setting, or on data for other lisks-referred to as an out-of-domain setting.
2.3 Explicit Knowledge-Fnhanee Retrieval
Since the introduction of the Semantic Web, it has benefited information retrieval systems by incorpo- rating explicit knowledge. In McCool and Miller (2003), an early semantic search prototype named TAP was presented, showing that knowledge bases can enhance retrieval systems, For Web search, Lu et al. (2009) demonstrated that ranking methods can be improved by semantic features. For entity search, researchers have employed entity linking techniques to annotate queries and measured the se- mantic relevance of target entity to query based on their learned embeddings (Gerritse et al., 2020). retrieval
3.2 Explicit Khowledge-Enhancee Retrieval
We assume that explicit knowledge is given as knowledge base describing a set of entities E. To exploit such knowledge, we firstly link the query q and the document representation Ta for each dataset d to two sets of entities in E, denoted by Fig F. and F. S F,, respectively. Then we ag- gregate the pairwise entity similarities between Eg and Ed as the relevance of d to q. Entity similarity is measured based on their textual and structural descriptions in the knowledge base.
Such explicit knowledge-enhance methods have not received much attention in the research of ad hoc dataset retrieval. We design a method for this new task and evaluate its various configurations using different knowledge bases, en- tity linking tools, and entity embeddings.
3.2.1 Entity Linking
We link to a set of entities Eg S E that are men- tioned in q to represent q. Analogously, we link Ta to set of entities Eg S E' that are mentioned in Ta to represent d. Entity linking is established re- search problem (Shen et al., 2015, 2023) and we use off-the-shelf tools in the experiments.
3 Methods
We divide knowledge-enhancee methods for ad hoc dataset retrieval into two types: using implicit knowledge and using explicit knowledge. Employ- ing implicit knowledge embodied in PLMs to en- hance retrieval has been widely used, which we will briefly review in Section 3.1. In Section 3.2 will present a method for employing explicit knowledge a knowledge base to enhance retrieval.
3.2.2 Entity Set Similarity
Let esim eÿ) be the similarity between two tities ei, E E, which will be elaborated in Section 3.2.3. Let be an m x n dimensional similarity matrix containing m = Exy] rows and n = Eal columns. For < S m and 1 S j S n, each element 8i,j represents the en- tity similarity between e E Egg and ej E,, i.e., 8i.j = esim(c;, Cz). We aggregate the similarity values in S as follows, also depicted in Figure 1. For each entity e; C Egg. we find its most similar entity in Eg and take their similarity value. We calculate the arithmetic mean (arithmean) of such similarity values over all the m entities in Eg:
Problem Statement. Given a query q, the main task in ad hoc dataset retrieval is to compute the relevance of each dataset d to y denoted by (el q). that ranked list of datasets can be returned.
3.1 Implicit Knowledge-Enhamen Retrieval
While the contents of different datasets can be in different formats (e.g., TXT, CSV, JSON), the metadata of dataset typically consists of set of fields such as title and description which are com- monly used retrieval. For each dataset d, we concatenate the textual values of its metadata fields (Ta.1, .a...... into document Ty:
row arithmean { max sij | i m), (3) 15j5.
Analogously, we compute to what extent the entities in Eg can best "cover" the entities in Eg:
col arithmean max sij 1 S n). (4)
Ta = Tai ① Tap ① ....,
15i5m

---

max
We choose harmonic mean because it empirically outperforms several other combination functions such as arithmetic mean, maximum, and minimum. Table 7 in the appendix presents the results of using different combination functions.
max
+L.3
F12
Fius
Axx
2
4 Experimental Setup
re))d,a))
Barmm
4.1
Test Collections
We conducted experiments on two collections for ad hoc dataset retrieval.
Figure 1: Aggregation of similarity matrix.
4.1.1 NTCIR-E
Finally, considering that two similar sets of en- tities should both largely cover each other, we cal- culate the harmonic mean (harmomean) of srow and gool, which aggregates the similarity values in as the relevance score of d:
NTCIR-E' is the English version of the test col- lection used in the NTCIR-15 Dataset Search task (Kato et al., 2021), including 46,615 datasets and 192 queries. The datasets were collected from Data.gov. The queries were crowdsourced, and were originally split into 96 as the training set and 96 as the test set. We further split the former into 76 as our training set and 20 as our validation set. The relevance of dataset query has been annotated irrelevant (0), partially-relevant (1), relevant (2) as the gold standard.
rel(d, q) = harmomean(s s00))
(5)
By the property of harmonic mean, rel(d, (d 9) will be high only both sow and gool are high.
3.2.3 Entity Similarity
Now elaborate esim(et, e.), the similarity be- tween Iwo entities e; ej E. We measure and integrate their entity- and word-level similarities. First, we measure the entity-level similarity be- tween e; and e based on their embedding vectors. Representation learning is an established research problem (Wang et al., 2017; Yang et al., 2022), which encodes the textual and structural descrip- tion of each entity in knowledge base into dense vector. Let ej, ej be the embeddings of ex, ej. re- spectively. We calculate their cosine similarity:
4.1.2 ACORDAR
ACORDAR2 is test collection specifically over RDF datasets (Lin et al., 2022), including 31,589 datasets and 493 queries. The datasets were collected from 543 open data portals. The queries were partially crowdsourced and partially collected from TREC, and were split into five subsets for five- fold cross-validation, each fold using three subsets as the training set, one subset as the validation set, and one subset as the test set. Similar to NTCIR-E. the relevance of dataset to query has been an- notated as irrelevant (0), partially-relevant (1), relevant (2) as the gold standard.
cat i.j cos(e;, ej).
(6)
Second, we measure the werd-level similarity between & and ej based on the embedding vec- tors of their mentions. Specifically, let W,, W be the sets of words that appear in the mentions of Ei, Ej, respectively. We construct W,| x [Wjl dimensional similarity matrix where each element represents the cosine similarity between two word embedding vectors. Then we aggregate the similar- ity values in the matrix in a way that resembles the aggregation process described Section 3.2.2 and Figure 1. The result is denoted by s Finally, we integrate entity- and levdl simi- larities by calculating their harmonic mean:
4.2 Evaluation Metrics
Retrieval on NTCIR-E relied on two metadata fields: title and description. ACORDAR further included two other fields: author and tags. Both NTCIR-E and ACORDAR have provided top-10 retrieval results returned by (field-weighted) BM25 over these fields. Based on these first-stage retrieval results, we investigated the reranking performance of anwwledge-enhanced retrieval methods.
Two evaluation metrics were used: normalized discounted cumulative gain (NDCG) and mean
'nttss//nccirddaascccccpipaa search 1/ hatgs://sgiaa..oommmuuueess////p
esim(e;, e2) = harmomean ent word
[ij $/.
(7)

---

average precision (MAP). When calculating MAP scores, both partially-relevan and relevant in the gold standard were considered as relevant.
the results using REL (Van Hulst al., 2020)² for comparison. We collected entity embeddings from Wikipedia2vee (Yamada et al., 2018). 10
For Wikidata, we used Falcon 2.0 (Sakor et al., 2020) for entity linking. We collected enttyy embeddings from KGTK (Ilievski et al., 2021 (the text version); we also reported the results using RDF2vec (Ristoski et al., 2019) (the sg sg_200 11000__5511_____11 1_44_000 version) for comparison. For word embeddings, we consistently collected from Wikipedia2vee.
4.3
Implementation Details
4.3.1 Implicit Knowledge-Enhancee Retrieval
We used five popular PLM-based methods for our experiments: monoBERTT-arge (Nogueira and Cho, 2019),² monoTS (Nogueira et al.. 2020),* 4 coCondenser (Gao and Callan, 2022),* CoIBERT-v2 (Khattab and Zaharia, 2020), and ANCE (Xiong et al., 2021)?. For each model, we used its checkpoint pre-trained on MS MARCO (Nguyen et al., 2016), and reported its performance the test set of each test col- lection to measure its performance in an out-of- domain (OOD) setting. We also fine-tuned it on the training and validation sets cach test collection and then measured its performance on the test set in an in-domain (ID) setting, where partially-relevnn and relevant datasets in the training set were used as positive samples, and irrelevant datasets were used negative samples. It is worth noting that we fine- tuned and used all these methods black boxes, e.g., ANCE's special negative sampling strategy as well as all the other special strategies incorporated into these methods were executed.
5 Experimental Results
Each of the following three subsections reported experimental results to answer one of the three research questions raised in Section 1.
5.1 Implicit Seaeledge-Enhancee Retrieval
As shown in Table 2, in an in-domain setting, reranking by coCondenser. ColBERT, and ANCE achieved significant improvements on both test collections terms of NDCG@5. However, for monoBERT the improvement was marginal on NTCIR-E, and for mono'T5 we even observed a performance drop on ACORDAR. The results in- dicated that the fine-tuned PLM-based methods might have overfitted the small training sets of existing test collections for ad hoc dataset trieval. monoBERT and monoT5 performed exten- sive query-document interaction, probably leading to more severe overfitting and worse test results.
For hyperparameter tuning in the in-domain set- ting, monoBERT and monoT5 were tuned with learning rate from (3e-5, 5e-5, le-4) and batch size from [16, 24). coCondenser, ColBERT, and ANCE were tuned with learning rate from (1e-6, 3c-6, 5c-6) and batch size from (32, 64), {8, 16), and [4, 8), respectively. Each model was trained for 10 epochs. The best-performing configuration on the validation set was selected to run on the test set. We used NVIDIA GeForee RTX 3090 GPUs.
The out-of-domain setting analyzed the general- izability of PLM- based methods. While reranking by monoTS, coCondenser, and ANCE achieved significant improvements NTCIR-E in terms of NDCG@5, only mono'T5 also achieved signif- icant improvement on ACORDAR, and for Col- BERT we even observed performance drop on both test collections. The results suggested that only a few domain-adapted PLM-based meth- ods could directly generalize to the task of ad hoc dataset retrieval having its unique features. ColBERT which relies word-level matching achieved relatively poor results possibly due some ambiguous query words such as 'MCI' which could be mistakenly matched with the same word
4.3.2 Explicit Knowledge-Enhanee Retrieval We used two well-known encyclopedic knowledge bases, Wikipedia and Wikidata, and used their corresponding entity linking tools and embeddings. For Wikipedia, we used TAGME (Ferragiina and Scaiella, 2010) for entity linking; we also reported
hhtps://ngginnfaacccoooaoooonnnn large- msmarco
wow.aatoooiiiiiimmoooi... large- sange-msmarco
phttp::/github.com/// htt.://u...ooo/ooooma//////i
'ntps://huggingffcecco http://nnwwg..oooooa/o///oonnnnn
httpps//wwwiippddiieeeeiiiu http:e..gitttttoiiiiiiieee io/wikipediaavee/
"htts://dowwloads.cc hut.t/////oooooooaaaaaaaaaaaaaan
http::://gituub.oo/////. WTB.SDMM-MM/////oo......
colbeett22/coleent220 2.0.tar.zz
"hupps://gituuub.ommo http://tww.oooooooosss/illliliia
hhtps::/hugginggaac
httppp///hugggggaae..ccsssssttte covesstoiii/nnc-mmmmarroooo
L3 hhupps//daaa.de wtw.ossiiiiaati.uum..
"htpss//taame bwciincc..og//tagmaa
mannheim,

---

Table 2: Performance of implicit knowledgg-enhanceeenn retrieval methods, with indicating significant
improvement after reranking (according to paired I-test under p < 0.05).
Test collection
Reranking method
NDCG@5
NDCG@ 10
MAP@5
MAP@ 10
before reranking
0.2252
0,2385
0.1232
0.1556
monoBERT (ID)
0.2280
0.2364
0.1188
0.1494
monoBERT (OOD)
0,2554
0,2513
0.1361
0.1645
monoT's (ID)
0,2532
0,2497
0,1413
0.1694
monoTT (OOD)
0
0.2702
0.1593
0.1852
NTCIR-E
coCondenser (ID)
0
0.2760*
0.1756
0,1981
coCondenser (OOD)
0.2933*
0,2775
0.1785
0,2001
ColBERT (ID)
0.2664*
0.2604
0,1522
0.1797
ColBERT (OOD)
0.2121
0.2217
0.1047
0.1351
ANCE (ID)
0.3024*
0.2794
0,1803
0.2023
ANCE (OOD)
0,2764
0,2650
0.1601
0.1851
before reranking
0.5045
0.5249
0,2859
0.3837
monoBERT (ID)
0.5521
0.5511
0.3239
0.4130
moRoBERT (OOD)
0,5133
0,5305
0,2855
0.3835
mono'I5 (ID)
0.3570
0.4418
0.1904
0.3056
monoTS (OOD)
0.5262*
0.5338
0.3049
0.3981
ACORDAR
coCondenser (ID)
0,5300
0.5360
0.3076
0.3998
coCondenser (OOD)
0.5144
0.5259
0,2963
0.3901
ColBERT (ID)
0.5273
0,5372
0.3080
0.4010
ColBERT (OOD)
0,4451
0.4889
0.2430
0.3473
ANCE
0.5327*
0.5416
0.3123
0.4048
ANCE (OOD)
0.5125
0.5289
0,2892
0.3850
Table 3: Performance explicit knowledge-enhanceennne enhancement retrieval methods, with indicating a significant improvement after reranking (according to paired t-test under p < 0.05).
Test collection
Reranking method
NDCG@5
NDCG@
MAP@5
MAP@
hefore reranking
0,2252
0.2385
0.1232
0.1556
Wikipedia w/ TAGME
0.2661
0.2576
0.1521
0.1782
NTCIR-E
Wikipedia w/ REL
0.2241
0.2378
0.1227
0.1551
Wikidata w/ KGTK
0.2314
0.2392
0.1211
0.1521
Wikidata w/ RDF2vee
0,2135
0,2337
0.1191
0.1518
before reranking
0.5045
0.5249
0.2859
0.3837
Wikipedia w/ TAGME
0.4963
0.3150
0.2746
0.3725
ACORDAR
Wikipedia w/ REL
0.4636
0.5002
0.2519
0.3563
Wikidata w/ KGTK
0.4682
0.4968
0,2552
0.3558
Wikidata w/ RDF2vec 0.4607
0.4964
0.2459
0.3516

---

having a different meaning in an irrelevant docu- ment. Other models relying on the global [CLS] token representation were less influenced by such individual word mismatches.
the improvements were significant in terms of NDCG@5 (represented by t). On NTCIR-E, inter- polation improved the performance of monoBERT and ColBERT, although it worsened the perfor- mance of the other methods. With interpolation, reranking by almost all the methods (except for ColBERT) achieved significant improvements on both test collections in terms of NDCG@S (repre- sented by *), whereas without interpolation only monoT5 achieved that. The results demonstrated that domain adapted PLM-based methods interpo- lated with BM25 scores could effectively generalize to ad hoc dataset retrieval.
5.2 Explicit Knowledge-Enhanck Retrieval
As shown in Table 3, only reranking by using Wikipedia knowledge with TAGME achieved significant improvement on NTCIR-E in terms of NDCG@5. We observed performance drops in all the other configurations, and observed noticeable differences between the performance of using dif- ferent entity linking tools (i.c., TAGME or REL) and different entity embeddings (i.e., KGTK or RDF2vec). The results suggested that reranking with explicit knowledge could have the potential but also require very careful implementation ob- tain effectiveness in ad hoc dataset retrieval.
For explicit knowledge-enhaneeee retrieval meth- ods, as shown in Table 5, interpolation helped improve the performance of all the methods on ACORDAR, and all the improvements were signifi- cant (represented by t). On NTCIR-E, interpolation also noticeably improved the performance of us- ing Wikidata knowledge, while its influence on the methods using Wikipedia knowledge was minor. With interpolation, reranking by using Wikipedia knowledge with TAGME achieved significant im- provements on both test collections in terms of NDCG@5 (represented by "), whereas without in- terpolation it achieved that only on NTCIR-E. The results demonstrated that reranking with explicit knowledge interpolated with BM25 scores could effectively benefi ad hoc dataset retrieval.
More concretely, we attributed the relatively good performance of TAGME to its higher recall than REL. Indeed, the sparse Wikipedia links found by could not sufficiently capture the semantics of the original text. RDF2vec which mainly em- ployed the graph structure of Wikidata was inferior KGTK whose text version used in our experi- ments ignored graph structure and only exploited the textual description of entitics, which seemed to be more helpful than the graph structure.
5.3 Score Interpolation
5.3.2
Interpolation of Implicit and Explicit Knowledge-Enhand Enhanced Retrieval
5.3.1 Interpolation with BM25 Scores
Section 5.1 and Section 5.2, we directly used the relevance scores computed by knowledge- enhanced retrieval methods to rerank first-stage retrieval results. Their unsatisfying results might be partially related to their weak sensitivity to ex- act lexical matches. Therefore, a straightforward extension would be to interpolate their score with BM25 score to enhance their capability of lexical matching, i.e., by calculating the sum of the two scores (both after min-max min-max normalization). We chose sum because it was empirically among the best perf-perfonning fusion algorithms for interpola- tion. Table 8 in the appendix presents the results of using different fusion algorithms provided by ranx (Bassani and Romelli, 2022).14 14
In Scction 5.3.1, reranking with implicit and ex- plicit knowledge interpolated with BM25 scores both showed effectiveness, so we continued to ex- plore whether the two kinds of knowledge could complement cach other by further interpolating implicit knowledge-enhanced relevance score, ex- plicit knowledge-enhaneee relevance score, and BM25 score, i.e., by calculating the sum of the three scores (all after min-max normalization). For a focused discussion, we reported the re- sults of monoT5 and Wikipedia with TAGME, the best pest-performin implicit and explicit knowledge- enhanced retrieval methods interpolated with BM25 scores according to Table 4 and Table 5, respectively. Similar results were observed on the other methods and hence omitted.
For implicit khowledge-enhancee retrieval meth- ods, shown in Table 4, interpolation helped improve the performance of all the methods on ACORDAR an out-of-domain setting, and all
As shown in Table 6, such further interpolation achieved the highest scores in all the settings. particular, compared with using explicit knowledge interpolated with BM25 scores, incorporating im-

---

Table 4: Performance of implicit knowledge-enhanced retrieval methods interpolated with BM25 scores, with indicating significant improvement after reranking, and f indicating a significant improvement after interpolation with BM25 scores (paired I-test with p < 0.05).
Test collection
Reranking method hefore reranking
N
NDCG@10
MAP@5
MAP@
0.2252
0.2385
0.1232
0.1556
monoBERT (OOD)
0.2554
0.2313
0.1361
0.1645
interpolated with BM25
0,2582
0.2569
0,1422
0,1711
monoTS (OOD)
0.2833
0.2702"
0.1593*
0.1852
interpolated with BM25
0.2697
0.2616"
0.1498*
0.1778
-
coCondenser (OOD)
0.2933
0.2775
0.1785
0.2001
interpolated with BM25
0.2693
0.2638
0.1600*
0.1855*
ColBERT (OOD)
0.2121
0.2217
0.1047
0.1351
interpolated with BM25
0.2361
0,24674
0.1334
0.1657f
ANCE (OOD)
0.2764
0,2650
0,1601
0,1851
interpolated with BM25
0.2648
0.2555
0.1486*
0,1748*
before reranking
0.5045
0,5249
0.2859
0.3837
mon (OOD)
0.5133
0.5305
0,2855
0,3835
interpolated with BM25
0.5324*
0.5398**
0.3011
0.3952
monoT5 (OOD) interpolated with BM25
0.5262
0.5338
0,3049
0,398
0.5496*
0,5499**
0.3212*
0.4126*
ACORDAR
coCondenser (OOD)
0.5144
0.5259
0.2963
0,3901
interpolated with BM25
0.5378
0.5386*
0.3101
0.4000*
ColBERT (OOD) interpolated with BM25
0.4451
0,4889
0,2430
0.3473
0,5003#
0,5210
0,28331
0,3800f
ANCF (OOD)
0.5125
0.5289
0,2892
0,3850
interpolated with BM25
0.5368
0.5427*
0.3149*
0.4056*1
Table 5: Performance of explicit knowledge-enhance retrieval methods interpolated with BM25 scores, with * indicating significant improvement after reranking, and f iindicating a significant improvement after interpolation with BM25 scores (paired I-test with p < 0.05).
Test collection
Reranking method before reranking Wikipedia TAGME
NDCG@5
NDCG@10
MAP@5
MAP@ 10
0.2252
0.2385
0.1232
0.1556
0.2661
0.2576
0.1521
0.1782
interpolated with BM25
0.2634
0.2565
0.1499
0. 0.1755
Wikipedia w/ REL interpolated with BM25
0.2241
0.2378
0,1227
0,1551
NTCIR-E
0.2252
0.2385
0.1232
0. 0.1556
Wikidata w/ KGTK interpolated with BM25
0,2314
0.2392
0.1211
0,1321
0.2590
0.2557
0,1468
0,1737
Wikidata w/ RDF2vee
0,2135
0.2337
0.1191
0,1318
interpolated with BM25
0.2310
0.2435
0.1328
0,1634
before reranking
0.5045
0.5249
0.2859
0.3837
Wikipedia w/ TAGME
0.4963
0.5150
0.2746
0.3725
interpolated with BM25
0.5243*
0.5347
0.3003*
0.3952
Wikipedia w/ REL interpolated with BM25
0.4636
0,5002
0.2519
0.3563
ACORDAR
0.49874
0,5218
0,2801
0.37941
Wikidata w/ KGTK interpolated with BM25
0.4682
0.4968
0.2552
0.3558
0.5103
0.5274**
0.2912*
0,3885
Wikidata w/ RDF2vec
0,4607
0,4964
0,2459
0,3516
interpolated with BM25
0.50924
0.52774
0.28851
0,38731
Table 6: Performance of interpolation of implicit and explicit knowledge-enhanced retrieval methods, with indi-
cating a significant improvement after reranking, f and indicating significant improvement after incorporating explicit and implicit knowledge, respectively (paired t-test with p < 0.05).
Test collection
Reranking method
NDCG@S
NDCG@ 10
MAP@5
MAP@ 10
before reranking
0.2385
0.1232
0.1556
NTCIR-E
moniT5 (OOD), interpolated
0.2697
0.2616"
0.1498*
0.1778*
Wikipedia TAGME, interpolated with BM25
0.2634
0.2565*
0.1490"
0.1755*
interpolation of monoT5 (OOD), Wikipedia w/ TAGME, and BM25
0.2827*
0.2676
0,1631
0.1870"
before feranking
0,5043
0.5249
0.2859
0.3837
ACORDAR
monoT5 (OOD), interpolated with BM25
0,5496
0,5499
0.3212
0.4126
Wikipedia w/ TAGME, interpolated with BM25
0.5243*
0.5347*
0.3003
0.3952"
interpolation monoTS (OOD), Wikipedia TAGME, and BM25
0.5570" $
0.5523* g
0.3244
0.4139" :

---

plicit knowledge brought significant improvements on ACORDAR (represented by $), while their com- plementarity the other settings was not strong. The results suggested that using both implicit and explicit knowledge interpolated with first-stage lexi- cal retrieval scores could represent the current best practice in reranking for ad hoc datasel retrieval.
test collection support in-domain supervision of more generalizable PLM-based retrieval meth- ods. We also plan to apply PLM-based methods to not only dataset metadata but also dataset contents which are large and heterogeneous, thus posing great challenges. A related trending research direc- tion is to explore the effectiveness of large language models in ad hoc dataset retrieval. Regarding ex- plicit enowledge-enhaneed retrieval, since the accu- racy of entity linking observed in our experiments was not satisfactory and might distort the results, an idea is to extend existing test collections with man- ually annotated entities, which would also provide a useful resource for entity linking research.
6 Conclusion and Future Work
We summarize our empirical findings to answer the three research questions raised in Section 1.
RQI: Will implicit knowledge-enhancee trieval methods remain effective in ad hoc dataset retrieval in an out-of-domain setting? Accord- ing to the results presented in Section 5.1, only monoT5 remained effective on its own, whereas other PLM-based methods could not consistently bring improvements. However, interpolated with BM25 scores, most of these methods effectively generalized to this new task as shown by the results Section 5.3.1. It demonstrates the necessity of combining dense and sparse retrieval for this task. enhanced retrieval methods be effective in ad hoc dataset retrieval? According to the results presented in Section 5.2 and Section 5.3.1, reranking with explicit knowl- edge could be beneficial to this task only when per- forming interpolation with BM25 scores, and sig- nificant improvements were consistently observed only for the configuration using Wikipedia knowl- edge with TAGME for entity linking. It suggests that the incorporation of explicit knowledge into this task should be carefully designed, and calls for more effective implementations in the future.
re-
Acknowledgements
This work was supported by the NSFC (62072224).
Limitations
First, although we have already used two test col- lections for ad hoc dataset retrieval, there was the possibility that our results were still biased due to the small number of queries (less than a few hun- dred) in these test collections. The generalizability of our conclusions could be improved in the future when new and larger test collections available.
RQ2: Will explicit knowledge doowledgeenhaaneed
Second, some observations in our experiments have yet to be justified. For example, while rerank- ing by most PLM-based methods interpolated with BM25 scores showed effectiveness, it remains un- clear why ColBERT was an exception. Exploring such reasons could help deepen our understanding of this task as well as the strengths and weaknesses of existing retrieval methods.
RQ3: Will interpolated methods be more effec- live in ad hoc datasel retrieval? According to the results presented in Section 5.3.1 and Section 5.3.2, not only interpolation with BM25 scores was help- ful, but also implicit and explicit knowledge ex- hibited complementarity. A combination of lexical matching, implicit knowledge, and explicit knowl- edge consistently achieved the best performance in our experiments, representing the current best practice for solving this task.
Third, our score interpolation performed in the experiments was helpful but simple. There could be more effective interpolation strategies. For ex- ample, instead of score interpolation, implicit and explicit knowledge could be integrated into a single model. We have witnessed the emergence of such efforts, which will be evaluated in our future work.
Fourth, following common practice in the lit- erature, we only considered dataset metadata in retrieval without using dataset contents due to their magnitude and heterogeneity, which could not be effectively handled by current PLMs. It remains an open question as to whether and how dataset content should be exploited in retrieval.
Hopefully our empirical findings and conclu- sions should provide useful guidelines for the com- munity to research and practice ad hoc dataset re- trieval. In future work, we will continue explore and address the unique challenges of this impor- tant task. Regarding implicit knowledge-enhand retrieval, we are interested in constructing a large

---

References
Luyu Gao. Zhuyun Dai, and Jamie Callan. 2021. COIL: revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Compulational Linguistics (NAACL), pages 3030-3042.
Elias Bassani and Luca Romelli. 2022. fuse: A python library for metasearch. CIKM, pages 4808- 4812. ACM.
Omar Benjelloun, Shiyu Chen, and Natasha F. Noy. 2020. Google dataset search by the numbers. In The Semantic Web ISWC 2020- 19th International Se- muntic Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Part II, volume 12507 of Lecture Notes in Computer Science, pages 682. Springer.
Emma J Gerritse, Faegheh Hasibi, and Arjen P de Vries. 2020. Graph-emhedding empowered entity retrieval. In Advances in Information Retrieval: 42nd Euro- pean Conference on IR Research, ECIR 2020, Lishon, Portugal, April 14-17, 2020, Proceedings, Part 42, pages 97-110. Springer.
Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google dataset search: Building a search engine for datasets in an open web ecosystem. In The World Wide Web Conference, pages 1365- 1375.
Filip llievski, Pedro Szekely, Gleb Satyukov, and Aman- deep Singh. 2021. User-friende comparison of similarity algorithms on wikidata. arXiv preprint arXiv:2108.05410
Adriane Chapman, Elena Simperl, Laura Koesten, George Konstantinidis, Luis-Daniel Ibáñez, Emilia Kacprzak, and Paul Groth. 2020. Dataset search: a survey. The VLDB Journal, 29(1):251-277
Emilia Kacprzak, Laura Koesten, Luis-Danicl Ibáñez, Tom Blount, Jeni Tennison, and Elena Simperl. 2019. Characterising dataset search an analysis of search logs and data requests. J. Web Semant., 55:37-55.
Jinchi Chen, Xiaxia Wang, Gong Cheng, Evgeny Khar- lamov, and Yuzhong Qu. 2019. Towards more usable dataset search: From query characterization to snip- pet generation. In Proceedings of the 28th ACM Inter- national Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019, pages 2445-2448. ACM.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis. Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6781.
pagess 6769-
Qiaosheng Chen, Zixian Huang, Zhiyang Zhang, Weiqing Luo, Tengteng Lin, Qing Shi, and Gong Cheng. 2023, Dense re-ranking with weak supervi- sion for rdf dataset search. In Proceedings the 22nd International Semantic Web Conference, ISWC 2023.
Makoto Kato, Hiroaki Ohshima, Ying-Hsang Liu, and Hsin-Liang Chen. 2021. A test collection for ad-hoc dataset retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2450 2456.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 4171-4186.
Omar Khattab and Matei Zaharia. 2020. Colbert: Ef- ficient and effective passage search via contextual- ized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pages 39-48.
Paolo Ferragina and Ugo Scaiella. 2010. Tagme: on on-the- fly annotation of short text fragments (by wikipedia entities). In Proceedings of the 19th ACM inter- national conference on Information and knowledge management, pages 1628.
Tengteng Lin, Qiaosheng Chen, Gong Cheng, Ahmet Soylu, Basil Ell, Ruoqi Zhao, Qing Shi, Xiaxia Wang. Yu Gu, and Evgeny Kharlamov. 2022. Acordar: A test collection for ad hoc content-based (rdf) dataset retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 2981-2991.
Luyu Gao and Jamie Callan. 2021. Condenser: pre- training architecture for dense retrieval. In Proceed- ings of the 2021 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 981-993.
Yumao Lu, Fuchun Peng, Gilad Mishne, Xing Wei, and Benoit Dumoulin. 2009. Improving web search relevance with semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 648-657.
Luyu Gao and Jamie Callan. 2022. Unsupervised cor- pus aware language model pre-training for dense pas- sage retrieval. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (ACL), pages 2843-2853.
Rob McCool and Eric Miller, 2003. Semantic search. In Proceedings of the 12th international conference on World Wide Web, pages 700-709.

---

Sebastian Neumaier, Jürgen Umbrich, and Axel Polleres. 2016. Automated quality assessment of metadata actoss open data portals. ACM J. Data Inf. Qual.,
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2018. Wikipedia2vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from wikipedia. arXiv preprint arXiv: N.Xx:88111 181. 882.06280.
11):2:1-2:29.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
Carl Yang. Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2022. Heterogeneous network represen- tation learning: A unified framework with survey and benehmark. IEEE Trans. Knowl. Data Eng. 34(10):4854-4873.
Rodrigo Nogueira, Zhiying Jiang. Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pre- model. In Findings
trained Sqquence-to-sequeee of the Association for Computational Linguistics: EMNLP 2020, pages (08-118, Online. Association for Computational Linguistics.
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2022. Dense text retrieval based on pretrained language models: A survey. arXiv, abs/2211. 14876.
Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv, abs/1901 1440855.
A Appendix: Additional Experiments
Petar Ristoski, Jessica Rosati, Tommaso Di Noia, Re- nato De Leone, and Heiko Paulheim. 2019. Rd/2vec: Rdf graph embeddings and their applications. Se- mantic Web, 10(4):721-752.
Ahmad Sakor, Kuldeep Singh, Anery Patel, and Maria- Esther Vidal. 2020. Falcon 2.0: An entity and rela- tion linking tool over wikidata. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 3141-3148.
Wei Shen, Yuhan Li, Yinan Liu, Jiawei Han, Jianyong Wang, and Xiaojie Yuan. 2023, Entity linking meets deep learning: Techniques and solutions. IEEE Trans. Knowl. Data Eng., 35(3):2556-2578.
Wei Shen, Jianyong Wang, and Jiawei Han. 2015. En- tity linking with a knowledge base: Issues, tech- niques, and solutions. IEEE Trans. Knowl. Data Eng. 27(2))443-466)
Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Gold- berg, and Jonathan Berant. 2020. trap-of-thought Teaching pre-trained models to systematically rea- over implicit knowledge. In Advances in Neural Information Processing Systems (NeurIPS).
Johannes M Van Hulst, Faegheh Hasibi, Koen Dercksen. Krisztian Balog, and Arjen P de Vries. 2020. Rel: An entity linker standing on the shoulders of giants. Proceedings of the 43rd International ACM SI- GIR Conference on Research and Development in Information Retrieval, pages 2197-2200.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: survey of approaches and applications. IEEE Trans. Knowl. Data Eng., 29122):724--2743
Lee Xiong, Chenyan Xiong, Ye Li. Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In 9th International Conference on Learning Representations (ICLR).

---

Table Performance explicit knowledge-enhareg retrieval (Wikipedia w/ TAGME) using different functions for combining entity- and word-level similarities.
Test collection
Combination
NDCG@5
NDCG@
MAP@5
MAP@
arithmean
0.2641
0,2578
0.1511
0.1779
harmomean
0.2661
0.2576
0.1521
0,1782
NTCIR-E
geomemcan
0.2661
0.2576
0.1521
0.1781
max
0.2472
0.2450
0.1317
0.1619
min 0,2538
0.2524
0.1434
0.1719
arithmean
0,4936
0.5149
0.2752
0,3735
harmomean
0.4963
0,5150
0.2746
0.3725
ACORDAR
geomemean
0.4948
0.5144
0.2739
0.3719
max
0,4767
0,5019
0.2586
0.3585
min
0,4900
0,5117
0.2707
0.3697
Table 8: Performance implicit knowledge enhanncedeee retrieval methods interpolated with BM25 scores using different fusion algorithms for interpolation.
Test collection
Reranking method
Fusion
NDCG@5
NDCG@
MAP@5
MAP@
sum 0,2494
0.2561
0,1408
0.1707
mf
0.2501
0.2499
0,1389
0.1666
monoBERT (ID) interpolated with BM25
wmnz
0.2494
0.2561
0.1408
0.1707
wsum
0.2609
0.2618
0.1498
0.1781
sum
0,2621
0.2553
0,1502
0.1760
mf
0.2457
0.2498
0.1361
0.1658
monoT5 (ID) interpolated with BM25
wmnz 0.2621
0.2553
0.1502
0.1760
wsum 0.2583
0.2568
0,1484
0.1771
sum
0.2794
0,2709
0,1705
0.1943
mf
0,2695
0.2609
0,1583
0.1824
N
coCondenser (ID) interpolated with RM25
wmnz 0.2794
0,2709
0.1705
0.1943
wsum 0),2252
0,2385
0.1232
0.1556
sum 0.2741
0,2685
0,1608
0.1883
mf
0.2657
0.2598
0,1503
0.1784
ColBERT (ID) interpolated with BM25
wmnz
0.2741
0.2685
0.1608
0.1883
wsum
0.2673
0.2628
0,1520
0.1804
sum
0.2864
0,2693
0,1649
0.1891
rf 0.2687
0.2656
0,1517
0.1815
ANCE (ID) interpolated with BM25
wmnz 0,2864
0.2693
0.1649
0.1891
wsum 0.3024
0,2766
0,1803
0,2010
0,5625
0,5563
0.3307
0.4191
mf 0.5534
0.5504
0.3241
0.4124
monoBERT (ID) interpolated with BM25
wmnz 0.5616
0.5557
0.3303
0.4187
0.5305
0.5388
0.3066
0.3993
0,4015
0.4657
0,2242
0.3323
nf
0,4236
0.4701
0,2288
0.3338
monoT55 (ID) interpolated with BM25
wmnz 0,4003
0.4649
0,2238
0.3319
wsum 0.4966
0,5195
0,2846
0.3820
sum
0,5502
0,5465
0,3206
0.4092
nf
0.5403
0.5423
0,3141
0.4042
ACORDAR
coCondenser (ID) interpolated with BM25
wmnz 0.5493
0.5459
0,3202
wsum 0.5156
0,2984
0.3895
sum 0,5472
0,3223
0.4132
0.5390
0.5403
0,3107
0.4014
CaIBERT (ID) interpolated with BM25
Wmnz 0.5463
0.5481
0,3219
0.4129
wsum 0.5087
0.5247
0,2898
0.3851
sum 0.5583
0.5552
0,3320
0.4196
nf 0.5419
0.5425
0,3160
0.4050
ANCE (ID) interpolated with BM25
WmnZ 0.5574
0.5546
0,3317
0.4192
wsum
0.5076
0.5248
0.2902
0.3859


Reviews:
[{"review_id": "Eq1W82osbs", "reviewer": "Reviewer_Pb56", "summary": "", "questions": "1. In subsection 5.1, why coCondenser, ColBERT, and ANCE achieve more significant improvements than monoBERT and monoT5 in in-domain setting? Also, why monoT5, coCondenser, and ANCE achieve more significant improvements than Col-BERT in out-domain setting?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the time and comments from the reviewer.\n\nFirstly, we answer the reviewer's **question**, which also **addresses the reviewer's second reason to reject**.\n\nQuestion: Why coCondenser, ColBERT, and ANCE achieve more significant improvements than monoBERT and monoT5 in in-domain setting? Also, why monoT5, coCondenser, and ANCE achieve more significant improvements than Col-BERT in out-domain setting?\n\nOur answer:\n\nIn an in-domain setting, as mentioned in Section 5.1, since the training sets are small, models might have overfitted. For monoBERT and monoT5, these models perform extensive query-document interaction, probably leading to **more severe overfitting** and worse results on the test set.\n\nIn an out-of-domain setting, ColBERT which relies on word-level matching achieved relatively bad results possibly due to some ambiguous query words such as 'MCI' which could be mistakenly matched with the same word having a different meaning in an irrelevant document. Other models relying on the global [CLS] token representation were **less influenced by such individual word mismatches**.\n\nWe will incorporate the above **insights** into the camera-ready version.\n\nNext, we would like to **clarify about the reviewer's first reason to reject** about novelty. Please note that our work addresses ad hoc dataset retrieval which is a relatively new and under-explored IR task. The methods we applied are \"well-established\" in conventional IR tasks but **their effectiveness in this emerging task remains unknown**, as characterized by our three research questions posed in the Introduction section. Our work **fills this gap** by conducting an extensive empirical investigation to answer these research questions and to help establish the current best practice for this task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The article is easy to follow.\n2. The article provides a good literature survey of the field.\n3. The authors show the effectiveness of knowledge aware retrieval technique on dataset retrieval task and reveal the strengths and weaknesses of employed methods, which give some successful cases to this underexplored task.\n", "reasons_to_reject": "1. Although this work explore more effective methods on dataset retrieval task, the employed dense retrieval techniques are well-established in information retrieval-related tasks which does not bring much inspiration. \n2. The analysis on the experimental results is not strong, more detailed insights explaining the observations are needed.\n", "questions_to_authors": "1. In subsection 5.1, why coCondenser, ColBERT, and ANCE achieve more significant improvements than monoBERT and monoT5 in in-domain setting? Also, why monoT5, coCondenser, and ANCE achieve more significant improvements than Col-BERT in out-domain setting?", "paper_topic_and_main_contributions": "This work explores the effectiveness of incorporating knowledge into the emerging task of ad hoc dataset retrieval, where the authors conducted a systematic investigation of implicit and explicit knowledge-enhanced methods for ad hoc dataset retrieval on two test collections.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQHz56sx1x", "reviewer": "Reviewer_SVC3", "summary": "", "questions": "* why didn't you experiment with other functions (geometric mean, maximum, top-k) besides arithmetic mean for aggregating row and column maximum values? \n\n* why do you interpolate the proposed methods with BM25 by just calculating the arithmetic mean of the two scores and not as a weighted interpolation, which would give you an opportunity to find the optimal contribution from both methods?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the time and comments from the reviewer.\n\nFirstly, we answer the reviewer's **questions** in the review, which also helps **address the reviewer's first reason to reject**.\n\nQ1. why didn't you experiment with other functions (geometric mean, maximum, top-k) besides arithmetic mean for aggregating row and column maximum values?\n\n**Actually we have experimented with other functions.** For example, below Equation (7) we have explained why we empirically chose harmonic mean. Similarly, for Equations (3) and (4), we chose arithmetic mean because it outperformed other functions in our experiments. **We did not include these experimental results in the paper because they may distractingly overload readers who might overlook our main results.** Since the reviewer referred to these results, we will add them to the appendix.\n\nQ2. why do you interpolate the proposed methods with BM25 by just calculating the arithmetic mean of the two scores and not as a weighted interpolation, which would give you an opportunity to find the optimal contribution from both methods?\n\n**We have tried a weighted interpolation.** However, due to the small training sets, we could not learn an effective weight, and the result on the test set was even worse than that by simply taking the arithmetic mean. We will add this explanation to the paper, and add the detailed results to the appendix.\n\nThe reviewer's **second reason to reject** is related to **presentation issues** and can be easily addressed in the camera-ready version.\n\nNext, we believe that **the reviewer's third reason to reject is debatable**. First, the reviewer said \"no statistically significant improvement after interpolation with BM25 for any LLM\". **This comment is inaccurate.** As shown in Table 4, we observed statistically significant improvements with ColBERT. Second, please note that we observed **statistically significant improvements in most of (though not all) the settings**, and **in the remaining settings the improvements are often considerable** (e.g., +0.02 or +0.03) although not significant under p<0.05. We believe that such results are sufficient for supporting the observations we summarized, but we respect the reviewer's comment and are willing to further revise our expressions if required.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper appears to be the first study to examine the utility of LLMs and knowledge graphs for dataset retrieval", "reasons_to_reject": "* Lack of experimentation and analysis: the proposed method for scoring the similarity of document and query, which is the only novelty of this paper from the technical perspective, is non-parametric. However, there are many options and alternatives within the proposed method that should have been explored to better understand its functioning and capabilities:\n\nIt would be interesting to find out what are the relative contributions of semantic vs. lexical similarity functions in computing entity similarity\n\nIt would be interesting to see if other functions (geometric mean, maximum, top-k) besides arithmetic mean for aggregating row and column maximum values would be more effective\n\nIt would be interesting to experiment with a parametric similarity function (e_i W e_j, where W is a parameter) besides cosine similarity\n\nIt seems sub-optimal to interpolate the proposed methods with BM25 by just calculating the arithmetic mean of the two scores. In prior IR research, such interpolation is normally done as a linear combination. \n \n* Lack of illustrative examples: it is not at all clear why measuring similarity between a query and dataset metadata in terms of entities would benefit retrieval. A clear and convincing illustrative example would make the proposed approach stronger\n\n* Unconvincing experimental results: the results presented in Tables 3, 4, 5 and 6 only partially support the proposed ideas. Specifically:\n\n- Table 3: the proposed explicit knowledge-enhanced methods result in statistically significant improvement of *only one metric* in *only one setting* across both datasets  \n- Table 4: the claim that the proposed implicit knowledge-enhanced methods interpolated with BM25 are effective is supported by statistically significant improvement of *2 out of 4 metrics only in the case of 1 out of 5 LLMs* on the NTCIR-E dataset\n- Table 5: the claim that the proposed explicit knowledge-enhanced methods interpolated with BM25 are effective is not supported by statistically significant improvement *in any setting* on the NTCIR-E dataset \n- Table 6: the claim that the proposed implicit and explicit knowledge-enhanced methods complement each other when interpolated with BM25 is not supported by any statistically significant improvement on the NTCIR-E dataset.\n\nOverall, the paper leaves a feeling that the authors were trying to jam-pack two ideas (utilizing LLMs and knowledge graphs in dataset retrieval) at the expense of properly exploring and determining the true potential of the most promising one - utilizing knowledge graphs in dataset retrieval.  I would suggest the authors focus on just knowledge graphs and properly explore the potential of this idea through more extensive experiments.", "questions_to_authors": "* why didn't you experiment with other functions (geometric mean, maximum, top-k) besides arithmetic mean for aggregating row and column maximum values? \n\n* why do you interpolate the proposed methods with BM25 by just calculating the arithmetic mean of the two scores and not as a weighted interpolation, which would give you an opportunity to find the optimal contribution from both methods?", "paper_topic_and_main_contributions": "This paper investigates the utility of Large Language Models (which the authors call \"implicit knowledge\") and knowledge graphs (which the authors call \"explicit knowledge\") for the task of dataset retrieval. The main contribution is that the authors find out that their proposed methods based on LLMs and knowledge graphs are ineffective when used on their own, but can be effective when interpolated with BM25 scores. The latter claim is, however, supported only by some experimental results.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iUNfUih5Uz", "reviewer": "Reviewer_RyqX", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the reviewer's time. However, **we could not agree with the reviewer's two very general reasons to reject**.\n\nR1. The reviewer said \"the paper does not present any novel findings\" and \"its main contributions are on surveying already existing approaches\".\n\n**Both comments are inaccurate.**\n\nOur main contributions are **not a survey but an extensive empirical investigation**, and we are among the first to conduct such an investigation for the task of ad hoc dataset retrieval which is a relatively new and under-explored IR task. Our experiments provided **a lot of new and insightful findings** (see Section 5 for details and Section 6 for a summary), which successfully answered three Research Questions posed in the Introduction section.\n\nR2. The reviewer said \"this problem is marginally relevant for the EMNLP conference\".\n\n**We could not agree because the relevance is explicit.** Ad hoc dataset retrieval is an emerging IR task, which is clearly within the scope of the Information Retrieval and Text Mining track.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study on ad-hoc dataset retrieval is thorough and captures a wide variety of approaches on a diverse set of corpora.", "reasons_to_reject": "- The paper does not present any novel findings, either at problem setting, methodology, etc. Its main contributions are on surveying already existing approaches, which in my opinion it is not enough for this conference.\n\n- The scope of this problem is marginally relevant for the EMNLP conference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "An analysis of the different approaches for ad-hoc dataset retrieval. The authors carry out an extensive study on what are the methods that work best.\n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "x213MqpEzr", "reviewer": "Reviewer_Afe6", "summary": "", "questions": "A - In Section 3.3.2, when first discussing the use of the harmonic mean to combine two results, the authors should briefly describe what are some of the properties of this approach that make it preferable to the arithmetic mean (e.g., the fact that having a high score with the harmonic mean requires both input scores to also have high values, and not just one of them).\n\nB - In Section 3.2.1, the authors state that \"entity linking is an established research problem and we use off-the-shelf tools in the experiments.\" However, the performance of entity linking systems often degrades on short texts (such as queries), given that there is not enough context to disambiguate entities. Ideally, the paper should discuss the accuracy/performance of the entity linking methods on the data in which they are being applied.\n\nC - Section 3.2.3 should ideally describe in more detail the embeddings for the entities that are used in the computation of entity set similarity and the entity-level similarity. Later in Section 4.3.2 the authors state that they use either Wikipedia2vec or KGTK for the entity set similarity, but they also say that Wikipedia2vec was also used as the source of word embeddings for entity-level similarity (which does not seem to make much sense, since I believe that only the words that correspond to a Wikipedia concept will be included, rather that the words that are part of entities). Why haven't the authors considered the use of more standard word embedding models?\n\nD - The description of the neural retrieval models used in the experiments should provide more details. It should be noted that some of these models correspond to cross-encoders (which only make sense to use in a re-ranking setting), whereas some of the models correspond to dual encoders (which would make more sense to use as first-stage retrievers). The different models are also associated to different training strategies and loss functions. For instance, the main innovation in ANCE is in the way the negative samples are obtained for training, but the authors seem to be using always the same training procedure when fine-tuning the models to the in-domain data (and aspects of this training procedure, such as the loss function, are not explained in the paper). On were in fact these models trained with the in-domain data?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the time and detailed comments from the reviewer.\n\nFirstly, we would like to clarify that the reviewer's **reasons to reject are mainly sourced from the limited availability of test collections rather than from our experimental design**.\n\nSpecifically, the first reason to reject is \"the re-ranking of only the top-10 results is too limiting\". We clarify that **the available test collections such as ACORDAR only provide annotated relevance labels for top-10 retrieval results**. Therefore, although we agree that it would be interesting to re-rank \"a larger set of datasets returned by first-stage retrieval\" as suggested by the reviewer, it is not feasible to conduct such an experiment based on existing test collections.\n\nThe second reason to reject (related to **Question B**) is \"it does not address the performance of the pre-existing entity linking systems on the in-domain data\". Again, it is difficult to conduct such an experiment because **the available test collections for ad hoc dataset retrieval do not provide gold-standard entity annotations**. However, we agree that such an analysis would be helpful, and we plan to extend existing test collections with our own annotations, but it has to be in our future work.\n\nThis reason to reject is also related to **Question D**: details about the training of the neural retrieval models with in-domain data. The reviewer said \"the authors seem to be using always the same training procedure\". **This is a misunderstanding.** We clarify that we directly used their original source code and we followed **their own training strategies and loss functions**. For example, for a fair comparison, we fed the same training data into all the models which were treated as black boxes, but if a model such as ANCE had a special negative sampling strategy inside the black box, it would certainly be executed. We will add this clarification to the paper.\n\nWe appreciate the reviewer's **suggestion in Missing References** about adding experiments using other rank fusion methods. **We have completed this experiment, which also helps address the reviewer's last reason to reject.** Specifically, by using the suggested ranx library, we tried different fusion methods: \"min\", \"max\", \"med\", \"sum\", \"anz\", \"mnz\", \"isr\", \"log_isr, \"bordafuse\", and \"condorcet\". We observed that \"sum\" (which is equivalent to our arithmetic mean) and \"mnz\" are the two best-performing methods, which empirically supported our choice of arithmetic mean in the paper. We will add this experiment to the appendix.\n\nNext, we answer the remaining questions in the review.\n\n**Question A is a minor suggestion on writing**, which we will easily incorporate into the camera-ready version.\n\nRegarding **Question C**, the reviewer said \"I believe that only the words that correspond to a Wikipedia concept will be included\". **This comment is inaccurate.** Wikipedia2vec provides embeddings for all the words occurring in Wikipedia articles, not limited to those in Wikipedia concepts/titles; that is, Wikipedia2vec is exactly one of the \"standard word embedding models\" suggested by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is clear and well-written, addressing an interesting topic that has not received much attention.\n\n* The extensive experimental results that are reported on the paper will likely constitute a useful benchmark for people working on dataset retrieval. ", "reasons_to_reject": "* The re-ranking setup is not the best evaluation setup that could have been considered. From the description provided in Section 4.2, the authors seem to be evaluating the different methods on a re-ranking setup, using the different models to re-rank the top-10 results provided by field-weighted BM25. However, the re-ranking of only the top-10 results is too limiting, and there are few opportunities to see increased effectiveness when focusing only on this small list. Ideally, the authors should report on experiments focused on re-ranking a larger set of datasets returned by first-stage retrieval.\n\n* The paper does not give much details about some of the models/methods that are being used (e.g., it does not describe the training of the neural retrieval models with in-domain data in sufficient detail, and it does not address the performance of the pre-existing entity linking systems on the in-domain data).\n\n* A different approach should have been considered for combining the different retrieval methods, more in alignment with previous rank fusion methods in the information retrieval literature.", "questions_to_authors": "A - In Section 3.3.2, when first discussing the use of the harmonic mean to combine two results, the authors should briefly describe what are some of the properties of this approach that make it preferable to the arithmetic mean (e.g., the fact that having a high score with the harmonic mean requires both input scores to also have high values, and not just one of them).\n\nB - In Section 3.2.1, the authors state that \"entity linking is an established research problem and we use off-the-shelf tools in the experiments.\" However, the performance of entity linking systems often degrades on short texts (such as queries), given that there is not enough context to disambiguate entities. Ideally, the paper should discuss the accuracy/performance of the entity linking methods on the data in which they are being applied.\n\nC - Section 3.2.3 should ideally describe in more detail the embeddings for the entities that are used in the computation of entity set similarity and the entity-level similarity. Later in Section 4.3.2 the authors state that they use either Wikipedia2vec or KGTK for the entity set similarity, but they also say that Wikipedia2vec was also used as the source of word embeddings for entity-level similarity (which does not seem to make much sense, since I believe that only the words that correspond to a Wikipedia concept will be included, rather that the words that are part of entities). Why haven't the authors considered the use of more standard word embedding models?\n\nD - The description of the neural retrieval models used in the experiments should provide more details. It should be noted that some of these models correspond to cross-encoders (which only make sense to use in a re-ranking setting), whereas some of the models correspond to dual encoders (which would make more sense to use as first-stage retrievers). The different models are also associated to different training strategies and loss functions. For instance, the main innovation in ANCE is in the way the negative samples are obtained for training, but the authors seem to be using always the same training procedure when fine-tuning the models to the in-domain data (and aspects of this training procedure, such as the loss function, are not explained in the paper). On were in fact these models trained with the in-domain data?", "paper_topic_and_main_contributions": "The authors present a comparative study on different methods for the task of ad-hoc dataset retrieval, which concerns retrieving datasets matching a query with a textual description, from a large collection of datasets where the instances are described through a set of English metadata elements (e.g., title, description, author, or tags). Specifically, using two datasets from previous work (ACORDAR and NTCIR-E), the authors evaluated a field-weighted BM25 baseline against dense retrieval approaches (using models trained on MS-MARCO, or further fine-tuned with in-domain data), against approaches based on entities extracted through off-the-shelf entity linking methods, or against approaches that combine dense retrieval and entities. Experimental results show that combined methods can achieve a better performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "I argue that the paper should feature more in-depth discussions about (a) other methods for representing text with dense embeddings, and (b) methods for combining results of different retrieval methods. The experimental results should ideally reflect these other studies as well.\n\nA - The experiments with neural/dense retrieval models focused on approaches pre-trained on MS-MARCO. However, there are other methods that could have been used and that achieve a better performance on datasets that are perhaps more similar to the setting of dataset retrieval (e.g., methods with better performance on benchmarks such as BEIR). The out-of-domain experiments should ideally also consider models such as those on the top results for the Massive Text Embedding Benchmark ( see https://huggingface.co/spaces/mteb/leaderboard ).\n\nB - On what regards score interpolation, instead of using the arithmetic mean of normalized scores, the authors should instead leverage methods from the information retrieval literature on rank fusion (e.g., methods such as reciprocal rank fusion -- see for instance https://amenra.github.io/ranx/fusion/ for an implementation of several of these methods )."}}]

Author rebuttal:

Meta-review: The paper investigates the utility of incorporating knowledge into Large Language Models for the task of ad hoc dataset retrieval. The authors conducted a systematic investigation of implicit and explicit knowledge-enhanced methods for ad hoc dataset retrieval on two test collections. The main contribution is that the authors evaluated a field-weighted BM25 baseline against dense retrieval approaches against approaches based on entities extracted using established entity linking methods and against approaches that combine dense retrieval and entities. Experimental results show that combined methods can achieve a better performance.

The approach presented is sound and the paper is clear and well written. The arguments are well supported by results and analysis.
Decision: Accept-Findings