DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions
Year: 2023

Abstract:
Modern machine learning relies on datasets to develop and validate research ideas. Given the growth of publicly available data, finding the right dataset to use is increasingly difficult. Any research question imposes explicit and implicit constraints on how well a given dataset will enable researchers to answer this question, such as dataset size, modality, and domain. We operationalize the task of recommending datasets given a short natural language description of a research idea, to help people find relevant datasets for their needs. Dataset recommendation poses unique challenges as an information retrieval problem; datasets are hard to directly index for search and there are no corpora readily available for this task. To facilitate this task, we build the DataFinder Dataset which consists of a larger automatically-constructed training set (17.5K queries) and a smaller expertannotated evaluation set (392 queries). Using this data, we compare various information retrieval algorithms on our test set and present a superior bi-encoder retriever for text-based dataset recommendation. This system, trained on the DataFinder Dataset, finds more relevant search results than existing third-party dataset search engines. To encourage progress on dataset recommendation, we release our dataset and models to the public.1

heading: 1 Introduction
text: Innovation in modern machine learning (ML) depends on datasets. The revolution of neural network models in computer vision (Krizhevsky et al., 2012) was enabled by the ImageNet Large Scale Visual Recognition Challenge (Deng et al., 2009). Similarly, data-driven models for syntactic parsing saw rapid development after adopting the Penn Treebank (Marcus et al., 1993; Palmer and Xue, 2010).
1Code and data: https://github.com/viswavi/ datafinder
fef7e0 e4f7fb e6f4eaF8F9FA FEEFE3 fad2cf
With the growth of research in ML and artificial intelligence (AI), there are hundreds of datasets published every year (shown in Figure 2). Knowing which to use for a given research idea can be difficult (Paullada et al., 2021). To illustrate, consider a real query from a graduate student who says, “I want to use adversarial learning to perform domain adaptation for semantic segmentation of images.” They have implicitly issued two requirements: they need a dataset for semantic segmentation of images, and they want datasets that include diverse visual domains. A researcher may intuitively select popular, generic semantic segmentation datasets like COCO (Lin et al., 2014) or ADE20K (Zhou et al., 2019), but these are insufficient to cover the query’s requirement of supporting domain adaptation. How can we infer the intent of the researcher and make appropriate recommendations?
To study this problem, we operationalize the task of “dataset recommendation”: given a fullsentence description or keywords describing a research topic, recommend datasets to support research on this topic (§2). A concrete example is
10288
shown in Figure 1. This task was introduced by Färber and Leisinger (2021), who framed it as text classification. In contrast, we naturally treat this task as retrieval (Manning et al., 2005), where the search collection is a set of datasets represented textually with dataset descriptions3, structured metadata, and published “citances” — references from published papers that use each dataset (Nakov et al., 2004). This framework allows us to measure performance with rigorous ranking metrics such as mean reciprocal rank (Radev et al., 2002).
To strengthen evaluation, we build a dataset, the DataFinder Dataset, to measure how well we can recommend datasets for a given description (§3). As a proxy for real-world queries for our dataset recommendation engine, we construct queries from paper abstracts to simulate researchers’ historical information needs. We then identify the datasets used in a given paper, either through manual annotations (for our small test set) or using heuristic matching (for our large training set). To our knowledge, this is the first expert-annotated corpus for dataset recommendation, and we believe this can serve as a challenging testbed for researchers interested in representing and searching complex data.
We evaluate three existing ranking algorithms on our dataset and task formation, as a step towards solving this task: BM25 (Robertson and Zaragoza, 2009), nearest neighbor retrieval, and dense retrieval with neural bi-encoders (Karpukhin et al., 2020). BM-25 is a standard baseline for text search, nearest neighbor retrieval lets us measure the degree to which this task requires generalization to new queries, and bi-encoders are among the most effective search models used today (Zhong et al., 2022). Compared with third-party keywordcentric dataset search engines, a bi-encoder model trained on DataFinder is far more effective at finding relevant datasets. We show that finetuning the
3From www.paperswithcode.com
bi-encoder on our training set is crucial for good performance. However, we observe that this model is as effective when trained and tested on keyphrase queries as on full-sentence queries, suggesting that there is room for improvement in automatically understanding full-sentence queries.

heading: 2 Dataset Recommendation Task
text: We establish a new task for automatically recommending relevant datasets given a description of a data-driven system. Given a query q and a set of datasets D, retrieve the most relevant subset R ⊂ D one could use to test the idea described in q. Figure 1 illustrates this with a real query written by a graduate student.
The query q can take two forms: either a keyword query (the predominant interface for dataset search today (Chapman et al., 2019)) or a fullsentence description. Textual descriptions offer a more flexible input to the recommendation system, with the ability to implicitly specify constraints based on what a researcher wants to study, without needing to carefully construct keywords a priori.
Evaluation Metrics Our task framing naturally leads to evaluation by information retrieval metrics that estimate search relevance. In our experiments, we use four common metrics included in the trec_eval package,4 a standard evaluation tool used in the IR community:
• Precision@k: The proportion of relevant items in top k retrieved datasets. If P@k is 1, then every retrieved document is valuable.
• Recall@k: The fraction of relevant items that are retrieved. If R@k is 1, then the search results are comprehensive.
• Mean Average Precision (MAP): Assuming we have m relevant datasets in total, and ki is the rank of the ith relevant dataset, MAP is calculated as ∑m i P@ki/m (Manning et al., 2005).
High MAP indicates strong average search quality over all relevant datasets.
• Mean Reciprocal Rank (MRR): The average of the inverse of the ranks at which the first relevant item was retrieved. Assuming Ri is the rank of the i-th relevant item in the retrieved result, MRR is calculated as ∑m i Ri/m. High MRR
means a user sees at least some relevant datasets early in the search results. 4https://github.com/usnistgov/trec_eval. We use
the -c flag for the trec_eval command.

heading: 3 The DataFinder Dataset
text: To support this task, we construct a dataset called The DataFinder Dataset consisting of (q,R) pairs extracted from published English-language scientific proceedings, where each q is either a fullsentence description or a keyword query. We collect a large training set through an automated method (for scalability), and we collect a smaller test set using real users’ annotations (for reliable and realistic model evaluation). In both cases, our data collection contains two primary steps: (1) collecting search queries q that a user would use to describe their dataset needs, and (2) identifying relevant datasets R that match the query. Our final training and test sets contain 17495 and 392 queries, respectively. Figure 3 summarizes our data collection approach. We explain the details below and provide further discussion of the limitations of our dataset in the Limitations section. We will release our data under a permissive CC-BY License.

heading: 3.1 Collection of Datasets
text: In our task definition, we search over the collection of datasets listed on Papers With Code, a large public index of papers which includes metadata for over 7000 datasets and benchmarks. For most datasets, Papers With Code Datasets stores a short human-written dataset description, a list of different names used to refer to the dataset (known as
“variants”), and structured metadata such as the year released, the number of papers reported as using the dataset, the tasks contained, and the the modality of data. Many datasets also include the paper that introduced the dataset. We used the dataset description, structured metadata, and the introducing paper’s title to textually represent each dataset, and we analyze this design decision in §5.4.

heading: 3.2 Training Set Construction
text: To ensure scalability for the training set, we rely on a large corpus of scientific papers, S2ORC (Lo et al., 2020). We extract nearly 20,000 abstracts from AI papers that use datasets. To overcome the high cost of manually-annotating queries or relevant datasets, we instead simulate annotations with few-shot-learning and rule-based methods.
Query Collection We extract queries from paper abstracts because, intuitively, an abstract will contain the most salient characteristics behind a research idea or contribution. As a result, it is an ideal source for comprehensively collecting potential implicit constraints as shown in Figure 1.
We simulate query collection with the 6.7B parameter version of Galactica (Taylor et al., 2022), a large scientific language model that supports fewshot learning. In our prompt, we give the model an abstract and ask it to first extract five keyphrases: the tasks mentioned in paper, the task domain of the paper (e.g., biomedical or aerial), the modality of data required, the language of data or labels required, and the length of text required (sentencelevel, paragraph-level, or none mentioned). We then ask Galactica to generate a full query containing any salient keyphrases. We perform few-shot learning using 3 examples in the prompt to guide the model. Our prompt is shown in Appendix A.
Relevant Datasets For our training set, relevant datasets are automatically labeled using the body text of a paper.5 We apply a rule-based procedure to identify the dataset used in a given paper (corresponding to an abstract whose query has been auto-labeled). For each paper, we tag all datasets that satisfy two conditions: the paper must cite the paper that introduces the dataset, and the paper must mention the dataset by name twice.6
5Note that our queries are obtained from the abstract alone while the relevance judgements are obtained from the text body, to encourage more general queries.
6We apply the additional requirement that the counted dataset mentions must occur in a section with section title con-
This tagging procedure is restrictive and emphasizes precision (i.e., an identified dataset is indeed used in the paper) over recall (i.e., all the used datasets are identified). Nonetheless, using this procedure, we tag 17,495 papers from S2ORC with at least one dataset from our collection of datasets.
To estimate the quality of these tagged labels, we manually examined 200 tagged paper-dataset pairs. Each pair was labeled as correct if the paper authors would have realistically had to download the dataset in order to write the paper. 92.5% (185/200) of dataset tags were deemed correct.

heading: 3.3 Test Set Construction
text: To accurately approximate how humans might search for datasets, we employed AI researchers and practitioners to annotate our test set. As mentioned above, the dataset collection requires both query collection and relevant dataset collection. We use SciREX (Jain et al., 2020), a humanannotated set of 438 full-text papers from major AI venues originally developed for research into full-text information extraction, as the basis of our test set. We choose this dataset because it naturally supports our dataset collection described below.
Query Collection We collect search queries by asking annotators to digest, extract, and rephrase key information in research paper abstracts.
Annotators. To ensure domain expertise, we recruited 27 students, faculty, and recent alumni of graduate programs in machine learning, computer vision, robotics, NLP, and statistics from major US universities. We recruited 23 annotators on a voluntary basis through word of mouth; for the rest, we offered 10 USD in compensation. We sent each annotator a Google Form that contained between 10 and 20 abstracts to annotate. The instructions provided for that form are shown in Appendix B.
Annotation structure. For each abstract, we asked annotators to extract metadata regarding the abstract’s task, domain, modality, language of data required, and length of data required. These metadata serve as keyphrase queries. Then, based on these keyphrases, we also ask the annotator to write a sentence that best reflects the dataset need of the given paper/abstract, which becomes the fullsentence query. Qualitatively, we found that the keyphrases helped annotators better ground and
taining “results”, “experiment”, “evaluation”, “result”, “training”, or “testing”, to avoid non-salient dataset mentions, such as those commonly occurring in “related work".
concretize their queries, and the queries often contain (a subset of) these keyphrases.
Model assistance. To encourage more efficient labeling (Wang et al., 2021), we provided autosuggestions for each field from GPT-3 (Brown et al., 2020) and Galactica 6.7B (Taylor et al., 2022) to help annotators. We note that annotators rarely applied these suggestions directly — annotators accepted the final full-sentence query generated by either large language model only 7% of the time.
Relevant Datasets For each paper, SciREX contains annotations for mentions of all “salient" datasets, defined as datasets that “take part in the results of the article” (Jain et al., 2020). We used these annotations as initial suggestions for the datasets used in each paper. The authors of this paper then skimmed all 438 papers in SciREX and noted the datasets used in each paper. 46 papers were omitted because they either used datasets not listed on Papers With Code or were purely theorybased papers with no relevant datasets, leaving a final set of 392 test examples.
We double-annotated 10 papers with the datasets used. The annotators labeled the exact same set of datasets for 8 out of 10 papers, with a Fleiss-Davies kappa of 0.667, suggesting that inter-annotator agreement for our “relevant dataset” annotations is substantial (Davies and Fleiss, 1982; Loper and Bird, 2002).

heading: 3.4 Dataset Analysis
text: Using this set of paper-dataset tags, what can we learn about how researchers use datasets?
Our final collected dataset contains 17,495 training queries and 392 test queries. The training examples usually associate queries with a single dataset much more frequently than our test set does. This is due to our rule-based tagging scheme, which emphasizes precise labels over recall. Meanwhile, the median query from our expert-annotated test set
had 3 relevant datasets associated with it. We also observed interesting dataset usage patterns:
• Researchers tend to converge towards popular datasets. Analyzing dataset usage by community,7 we find that in all fields, among all papers that use some publicly available dataset, more than 50% papers in our training set use at least one of the top-5 most popular datasets. Most surprisingly, nearly half of the papers tagged in the robotics community use the KITTI dataset (Geiger et al., 2013).
• Researchers tend to rely on recent datasets.
In Figure 4, we see the distribution of relative ages of datasets used (i.e., the year between when a dataset is published, and when a corresponding paper uses it for experiments). In Figure 4, We observe that the average dataset used by a paper was released 5 years before the paper’s publication (with a median of 5.6 years), but we also see a significant long tail of older datasets. This means that while some papers use traditional datasets, most papers exclusively use recently published datasets.
These patterns hint that researchers might overlook less cited datasets that match their needs in favor of standard status-quo datasets. This motivates the need for nuanced dataset recommendation.
4 Experimental Setup on DataFinder
How do popular methods perform on our new task and new dataset? How does our new paradigm differ from existing commercial search engines? In this section, we describe a set of standard methods which we benchmark, and we consider which thirdparty search engines to use for comparison.
7We define “communities” by publication venues: ACL, EMNLP, NAACL, TACL, COLING for NLP, CVPR, ICCV, WACV for Vision, IROS, ICRA, IJRR for Robotics, and NeurIPS, ICML ICLR for Machine Learning. We include proceedings from associated workshops in each community.

heading: 4.1 Task Framing
text: We formulate dataset recommendation as a ranking task. Given a query q and a search corpus of datasets D, rank the datasets d ∈ D based on a query-dataset similarity function sim(q, d) and return the top k datasets. We compare three ways of defining sim(q, d): term-based retrieval, nearestneighbor retrieval, and neural retrieval.

heading: 4.2 Models to Benchmark
text: To retrieve datasets for a query, we find the nearest datasets to that query in a vector space. We represent each query and dataset in a vector space using three different approaches:
Term-Based Retrieval We evaluated a BM25 retriever for this task, since this is a standard baseline algorithm for information retrieval. We implement BM25 (Robertson and Walker, 1999) using Pyserini (Lin et al., 2021).8
Nearest-Neighbor Retrieval To understand the extent to which this task requires generalization to new queries unseen at training time, we experiment with direct k-nearest-neighbor retrieval against the training set. For a new query, we identify the most similar queries in the training set and return the relevant datasets from these training set examples. In other words, each dataset is represented by vectors corresponding to all training set queries attached to that dataset. In practice we investigate two types of feature extractors: TF-IDF (Jones, 2004) and SciBERT (Beltagy et al., 2019).
Neural Retrieval We implement a bi-encoder retriever using the Tevatron package.9 In this framework, we encode each query and document into a shared vector space and estimate similarity via the inner product between query and document vectors. We represent each document with the BERT embedding (Devlin et al., 2019) of its [CLS] token:
sim(q, d) = cls(BERT(q))T cls(BERT(d))
where cls(·) denotes the operation of accessing the [CLS] token representation from the contextual encoding (Gao et al., 2021). For retrieval, we separately encode all queries and documents and retrieve using efficient similarity search. Following recent work (Karpukhin et al., 2020), we minimize a contrastive loss and select hard negatives using
8We run BM25 with k1 = 0.8 and b = 0.4. 9https://github.com/texttron/tevatron
BM25 for training. We initialize the bi-encoder with SciBERT (Beltagy et al., 2019) and finetune it on our training set. This model takes 20 minutes to finetune on one 11GB Nvidia GPU.

heading: 4.3 Comparison with Search Engines
text: Besides benchmarking existing methods, we also compare the methods enabled by our new data recommendation task against the standard paradigm for dataset search — to use a conventional search engine with short queries (Kacprzak et al., 2019). We measured the performance of third-party dataset search engines taking as input either keyword queries or full-sentence method descriptions.
We compare on our test set with two third-party systems– Google Dataset Search10 (Brickley et al., 2019) and Papers with Code11 search. Google Dataset Search supports a large dataset collection, so we limit results to those from Papers with Code to allow comparison with the ground truth.
Our test set annotators frequently entered multiple keyphrases for each keyphrase type (e.g. “question answering, recognizing textual entailment” for the Task field). We constructed multiple queries by taking the Cartesian product of each set of keyphrases from each field, deduplicating tokens that occurred multiple times in each query. After running each query against a commercial search engine, results were combined using balanced interleaving (Joachims, 2002).
10https://datasetsearch.research.google.com 11https://paperswithcode.com/datasets

heading: 5 Evaluation

heading: 5.1 Time Filtering
text: The queries in our test set were made from papers published between 2012 and 202012, with median year 2017. In contrast, half the datasets in our search corpus were introduced in 2018 or later. To account for this discrepancy, for each query q, we only rank the subset of datasets D′ = {d ∈ D | year(d) ≤ year(q)} that were introduced in the same year or earlier than the query.

heading: 5.2 Benchmarking and Comparisons
text: Benchmarking shows that DataFinder benefits from deep semantic matching. In Table 1, we report retrieval metrics on the methods described
12We could not include more recent papers in our query construction process, because SciREX was released in 2020.
in §4. To determine the standard deviation of each metric, we use bootstrap resampling (Koehn, 2004) over all test set queries. Term-based retrieval (BM25) performs poorly in this setting, while the neural bi-encoder model excels. This suggests our task requires capturing semantic similarity beyond what term matching can provide. Term-based kNN search is not effective, implying that generalization to new queries is necessary for this task.
Commercial Search Engines are not effective on DataFinder. In Table 2, we compare our proposed retrieval system against third-party dataset search engines. For each search engine, we choose the top 5 results before computing metrics.
We find these third-party search engines do not effectively support full-sentence queries. We speculate these search engines are adapted from termbased web search engines. In contrast, our neural retriever gives much better search results using both keyword search and full-sentence query search.

heading: 5.3 Qualitative Analysis
text: Examples in Figure 7 highlight the tradeoffs between third-party search engines and models trained on DataFinder. In the first two examples, we see keyword-based search engines struggle when dealing with terms that could apply to many datasets, such as “semantic segmentation” or “link prediction”. These keywords offer a limited specification on the relevant dataset, but a system trained on simulated search queries from real papers can learn implicit filters expressed in a query.
On the final example, our system incorrectly focuses on the deep architecture described (“deep neural network architecture [...] using depthwise separable convolutions”) rather than the task described by the user (“machine translation”). Improving query understanding for long queries is a key opportunity for improvement on this dataset.

heading: 5.4 More In-depth Exploration
text: We perform in-depth qualitative analyses to understand the trade-offs of different query formats and dataset representations.
Comparing full-sentence vs keyword queries As mentioned above, we compare two versions of the DataFinder-based system: one trained and tested with description queries and the other with keyword queries. We observe that using keyword queries offers similar performance to using fullsentence descriptions for dataset search. This suggests more work should be done on making better use of implicit requirements in full-sentence descriptions for natural language dataset search.
Key factors for successful queries What information in queries is most important for effective dataset retrieval? Using human-annotated keyphrase queries in our test set, we experiment with concealing particular information from the keyphrase query.
In Figure 8, we see task information is critical for dataset search; removing task keywords from queries reduces MAP from 23.5 to 7.5 (statistically significant with p < 0.001 by a paired bootstrap t-test). Removing constraints on the language of text data also causes a significant drop in MAP (p < 0.0001). Removing keywords for text length causes an insignificant reduction in MAP
(p = 0.15), though it causes a statistically significant reduction on other metrics not shown in Figure 8: P@5 and R@5. Based on inspection of our test set, we speculate that domain keywords are unnecessary because the domain is typically implied by task keywords.
Comparing textual representations of datasets We represent datasets textually with a communitygenerated dataset description from PapersWithCode, along with the title of the paper that introduced the dataset. We experiment with enriching this dataset representation in two ways. We first add structured metadata about each dataset (e.g. tasks, modality, number of papers that use each dataset on PapersWithCode). We cumulatively experiment with adding citances — sentences from other papers around a citation — to capture how others use the dataset. In Table 3, our neural biencoder achieves similar retrieval performance on all 3 representations for full-sentence search.
Keyword search is more sensitive to dataset rep-
resentation. adding structured information to the dataset representation provides significant benefits for keyword search. This suggests keyword search requires more specific dataset metadata than fullsentence search does to be effective.
The value of finetuning Our bi-encoder retriever is finetuned on our training set. Given the effort required to construct a training set for tasks like dataset recommendation, is this step necessary?
In Table 4, we see that an off-the-shelf SciBERT encoder is ineffective. We observe that our queries, which are abstract descriptions of the user’s information need (Ravfogel et al., 2023), are very far from any documents in the embedding space, making comparison difficult. Using a state-of-the-art encoder, COCO-DR Base — which is trained for general-purpose passage retrieval on MS MARCO (Campos et al., 2016), helps with this issue but still cannot make up for task-specific finetuning.

heading: 6 Related Work
text: Most work on scientific dataset recommendation uses traditional search methods, including termbased keyword search and tag search (Lu et al., 2012; Kunze and Auer, 2013; Sansone et al., 2017; Chapman et al., 2019; Brickley et al., 2019; Lhoest et al., 2021). In 2019, Google Research launched Dataset Search (Brickley et al., 2019), offering access to over 2 million public datasets. Our work considers the subset of datasets from their search corpus that have been posted on Papers with Code.
Some work has explored other forms of dataset recommendation. Ben Ellefi et al. (2016) study using “source datasets” as a search query, while Altaf et al. (2019) use a set of related research papers as the user’s query. Färber and Leisinger (2021) are the only prior work we are aware of
that explores natural language queries for dataset recommendation. They model this task as classification, while we operationalize it as open-domain retrieval. Their dataset uses abstracts and citation contexts to simulate queries, while we use realistic short queries (with an expert-annotated test set).

heading: 7 Conclusion
text: We study the task of dataset recommendation from natural language queries. Our dataset supports search by either full-sentence or keyword queries, but we find that neural search algorithms trained for traditional keyword search are competitive with the same architectures trained for our proposed fullsentence search. An exciting future direction will be to make better use of natural language queries. We release our datasets along with our ranking systems to the public. We hope to spur the community to work on this task or on other tasks that can leverage the summaries, keyphrases, and relevance judgment annotations in our dataset.
Limitations
The primary limitations concern the dataset we created, which serves as the foundation of our findings. Our dataset suffers from four key limitations:
Reliance on Papers With Code Our system is trained and evaluated to retrieve datasets from Papers With Code Datasets (PwC). Unfortunately, PwC is not exhaustive. Several queries in our test set corresponded to datasets that are not in PwC, such as IWSLT 2014 (Birch et al., 2014), PASCAL VOC 2010 (Everingham et al., 2010), and CHiME4 (Vincent et al., 2017). Papers With Code Datasets also skews the publication year of papers used in the DataFinder Dataset towards the present (the median years of papers in our train and test set are 2018 and 2017, respectively). For the most part, PwC only includes datasets used by another paper listed in Papers With Code, leading to the systematic omission of datasets seldom used today.
Popular dataset bias in the test set Our test set is derived from the SciREX corpus (Jain et al., 2020). This corpus is biased towards popular or influential works: the median number of citations of a paper in SciREX is 129, compared to 19 for any computer science paper in S2ORC. The queries in our test set are therefore more likely to describe mainstream ideas in popular subields of AI.
Automatic tagging Our training data is generated automatically using a list of canonical dataset
names from Papers With Code. This tagger mislabels papers where a dataset is used but never referred to by one of these canonical names (e.g. nonstandard abbreviations or capitalizations). Therefore, our training data is noisy and imperfect.
Queries in English only All queries in our training and test datasets were in English. Therefore, these datasets only support the development of dataset recommendation systems for Englishlanguage users. This is a serious limitation, as AI research is increasingly done in languages other English, such as Chinese (Chou, 2022).
Ethics Statement
Our work has the promise of improving the scientific method in artificial intelligence research, with the particular potential of being useful for younger researchers or students. We built our dataset and search systems with the intention that others could deploy and iterate on our dataset recommendation framework. However, we note that our initial dataset recommendation systems have the potential to increase inequities in two ways.
First, as mentioned in Limitations, our dataset does not support queries in languages other than English, which may exacerbate inequities in dataset access. We hope future researchers will consider the construction of multilingual dataset search queries as an area for future work.
Second, further study is required to understand how dataset recommendation systems affect the tasks, domains, and datasets that researchers choose to work on. Machine learning models are liable to amplify biases in training data (Hall et al., 2022), and inequities in which domains or tasks receive research attention could have societal consequences. We ask researchers to consider these implications when conducting work on our dataset.

heading: Acknowledgements
text: This work was supported in part by funding from NEC Research Laboratories, DSTA Singapore, the National Science Foundation (NSF) grant IIS1815528, and a gift from Google. We thank Sireesh Gururaja, Soham Tiwari, Amanda Bertsch, Liangze Li, Jeremiah Millbauer, Jared Fernandez, Nikhil Angad Bakshi, Bharadwaj Ramachandran, G. Austin Russell, and Siddhant Arora for helping with data collection. We give particular thanks to Carolyn Rosé, Saujas Vaduguru, and Ji Min Mun for their helpful discussions and feedback.

heading: A Few-Shot Prompt for Generating Keyphrases and Queries
text: When constructing our training set, we use incontext few-shot learning with the 6.7B parameter version of Galactica (Taylor et al., 2022). We perform in-context few-shot learning with the following prompt:
Given an abstract from an artificial intelligence paper: 1) Extract keyphrases regarding the task (e.g. image classification), data modality (e.g. images or speech), domain (e.g. biomedical or aerial), training style (unsupervised, semi-supervised, fully supervised, or reinforcement learning), text length (sentence-level or paragraph-level), language required (e.g. English) 2) Write a brief, single-sentence summary containing these relevant keyphrases. This summary must describe the task studied in the paper.
Abstract: We study automatic question generation for sentences from text passages in reading comprehension. We introduce an attention-based sequence learning model for the task and investigate the effect of encoding sentence- vs. paragraph-level information. In contrast to all previous work, our model does not rely on hand-crafted rules or a sophisticated NLP pipeline; it is instead trainable end-to-end via sequence-to-sequence learning. Automatic evaluation results show that our system significantly outperforms the state-of-the-art rulebased system. In human evaluations, questions generated by our system are also rated as being more natural (i.e., grammaticality, fluency) and as more difficult to answer (in terms of syntactic and lexical divergence from the original text and reasoning needed to answer).
Output: (Task | Modality | Domain | Training Style | Text Length | Language Required | Single-Sentence Summary) Task: question generation Modality: text Domain: N/A
Training Style: fully supervised Text Length: paragraph-level Language Required: N/A Single-Sentence Summary: We propose an improved end-to-end system for automatic question generation. –
Abstract: We present a self-supervised approach to estimate flow in camera image and top-view grid map sequences using fully convolutional neural networks in the domain of automated driving. We extend existing approaches for self-supervised optical flow estimation by adding a regularizer expressing motion consistency assuming a static environment. However, as this assumption is violated for other moving traffic participants we also estimate a mask to scale this regularization. Adding a regularization towards motion consistency improves convergence and flow estimation accuracy. Furthermore, we scale the errors due to spatial flow inconsistency by a mask that we derive from the motion mask. This improves accuracy in regions where the flow drastically changes due to a better separation between static and dynamic environment. We apply our approach to optical flow estimation from camera image sequences, validate on odometry estimation and suggest a method to iteratively increase optical flow estimation accuracy using the generated motion masks. Finally, we provide quantitative and qualitative results based on the KITTI odometry and tracking benchmark for scene flow estimation based on grid map sequences. We show that we can improve accuracy and convergence when applying motion and spatial consistency regularization.
Output: (Task | Modality | Domain | Training Style | Text Length | Language Required | Single-Sentence Summary) Task: optical flow estimation Modality: images and top-view grid map sequences Domain: autonomous driving Training Style: unsupervised Text Length: N/A Language Required: N/A Single-Sentence Summary: A system for selfsupervised optical flow estimation from images and top-down maps. –
Abstract: In this paper, we study the actor-action semantic segmentation problem, which requires joint labeling of both actor and action categories in video frames. One major challenge for this task is that when an actor performs an action, different body parts of the actor provide different types of cues for the action category and may receive inconsistent action labeling when they are labeled independently. To address this issue, we propose an end-to-end region-based actor-action segmentation approach which relies on region masks from an instance segmentation algorithm. Our main novelty is to avoid labeling pixels in a region mask independently - instead we assign a single action label to these pixels to achieve consistent action labeling. When a pixel belongs to multiple region masks, max pooling is applied to resolve labeling conflicts. Our approach uses a two-stream network as the front-end (which learns features capturing both appearance and motion information), and uses two region-based segmentation networks as the back-end (which takes the fused features from the two-stream network as the input and predicts actor-action labeling). Experiments on the A2D dataset demonstrate that both the region-based segmentation strategy and the fused features from the two-stream network contribute to the performance improvements. The proposed approach outperforms the state-of-the-art results by more than 8 Output: (Task | Modality | Domain | Training Style | Text Length | Language Required | SingleSentence Summary) Task: actor-action semantic segmentation Modality: video Domain: N/A Training Style: fully supervised Text Length: N/A Language Required: N/A Single-Sentence Summary: I want to train a supervised model for actor-action semantic segmentation from video. –
For a given abstract that we want to process, we then add this abstract’s text to this prompt and ask the language model to generate at most 250 new tokens.
B Information on Expert Annotations
As mentioned in §3, we recruited 27 graduate students, faculty, and recent graduate program alumni for our annotation collection process. For each annotator, we received their verbal or written interest in participating in our data collection.
We then sent them a Google Form containing between 10 and 20 abstracts to annotate. An example of the form instructions is included in Figure 9.
We originally had annotators label the “Training Style” (unsupervised, semi-supervised, supervised, or reinforcement learning), in addition to Task, Modality, Domain, Text Length, and Language Required. However, this field saw excessively noisy labels so we ignore this field for our experiments.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We discuss the limitations in Lines 552 - 599.

heading: 3 A2. Did you discuss any potential risks of your work?
text: Yes, we discuss some potential ethical risks related to the use of our work in the "Ethics Statement" (Lines 601 - 626)
7 A3. Do the abstract and introduction summarize the paper’s main claims? See the abstract and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We mention our use of the SciREX dataset in Section 3.3: "Test Set Construction".

heading: 3 B1. Did you cite the creators of artifacts you used?
text: We cite the SciREX authors in Section 3.3 (line 242).

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Yes, under the main header of Section 3 we discuss that we will release our data under a CC-BY license.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section 3.3 (line 244) we mention the use of an existing artifact. In the Conclusion (Section 7), we discuss the liberal intended uses of our dataset.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, but our dada contains no anonymous information about annotators.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Yes, in Section 3.4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, in Section 3.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Yes, in Section 4.2.
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We mentioned the number of parameters in some cases (in Section 3.2 we mention the size of an LLM we use), an we mention the computing infrastructure in the bottom of Section 5. We do not mention total computational budget because our paper was very compute-light, so we did not feel that total computational budget was salient enough to mention.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? No. We did not perform hyperparameter search.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We include standard deviations in Tables 1 and 3, and we also discuss signiicance tests in Section 5.2 and 5.4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We discuss the evaluation package and parameters we use under Section 2 (footnote 4), and we discuss the BM25 retrieval parameters we use in Section 4.2 (footnote 8).
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? We include the initial instructions provided to participants in Appendix B.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.3

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix B
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No. We procured annotations *from* annotators rather than *about* annotators, and therefore we did not feel that IRB approval was necessary.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No. We mentioned that the annotators were students, faculty, and recent alumni of graduate programs in AI, robotics, computer vision, NLP, and statistics. For the purposes of our dataset, more detailed demographic and geographic information would not be relevant.

heading: Future Work
text: An exciting future direction will be to make better use of natural language queries. We release our datasets along with our ranking systems to the public. We hope to spur the community to work on this task or on other tasks that can leverage the summaries, keyphrases, and relevance judgment annotations in our dataset. Limitations
The primary limitations concern the dataset we created, which serves as the foundation of our findings. Our dataset suffers from four key limitations:
Reliance on Papers With Code Our system is trained and evaluated to retrieve datasets from Papers With Code Datasets (PwC). Unfortunately, PwC is not exhaustive. Several queries in our test set corresponded to datasets that are not in PwC, such as IWSLT 2014 (Birch et al. , 2014), PASCAL VOC 2010 (Everingham et al. , 2010), and CHiME4 (Vincent et al. , 2017). Papers With Code Datasets also skews the publication year of papers used in the DataFinder Dataset towards the present (the median years of papers in our train and test set are 2018 and 2017, respectively). For the most part, PwC only includes datasets used by another paper listed in Papers With Code, leading to the systematic omission of datasets seldom used today. Popular dataset bias in the test set Our test set is derived from the SciREX corpus (Jain et al. , 2020). This corpus is biased towards popular or influential works: the median number of citations of a paper in SciREX is 129, compared to 19 for any computer science paper in S2ORC. The queries in our test set are therefore more likely to describe mainstream ideas in popular subields of AI. Automatic tagging Our training data is generated automatically using a list of canonical dataset
names from Papers With Code. This tagger mislabels papers where a dataset is used but never referred to by one of these canonical names (e. g. nonstandard abbreviations or capitalizations). Therefore, our training data is noisy and imperfect. Queries in English only All queries in our training and test datasets were in English. Therefore, these datasets only support the development of dataset recommendation systems for Englishlanguage users. This is a serious limitation, as AI research is increasingly done in languages other English, such as Chinese (Chou, 2022)