Dense Retrieval as Indirect Supervision for Large-space Decision Making
Authors: Nan Xu
Fei Wang
Mingtao Dong
Muhao Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: large decision spaces
dense retrieval
extreme multi-label classification
ultra-fine entity typing
few-shot intent classification

Abstract: Many discriminative natural language understanding (NLU) tasks have large label spaces. Learning such a process of large-space decision making is particularly challenging due to the lack of training instances per label and the difficulty of selection among many fine-grained labels. Inspired by dense retrieval methods for passage finding in open-domain QA, we propose a reformulation of large-space discriminative NLU tasks as a learning-to-retrieve task, leading to a novel solution named Dense Decision Retrieval (DDR). Instead of predicting fine-grained decisions as logits, DDR adopts a dual-encoder architecture that learns to predict by retrieving from a decision thesaurus. This approach not only leverages rich indirect supervision signals from easy-to-consume learning resources for dense retrieval, it also leads to enhanced prediction generalizability with a semantically meaningful representation of the large decision space. When evaluated on tasks with decision spaces ranging from hundreds to hundred-thousand scales, DDR outperforms strong baselines greatly by 27.54% in P @1 on two extreme multi-label classification tasks, 1.17% in F1 score ultra-fine entity typing, and 1.26% in accuracy on three few-shot intent classification tasks on average.

Dense Retrieval as Indirect Supervision for Large-space Decision Making
Nan Xu
Fei Wang
Mingtao Dong
Muhao Chen
University of Southern California
University of California, Davis (nanx, fwang598, mingtaod edu muhchen ucdavis.edu
Abstract
Previous studies for such NLU tasks still train classifiers as the solution (Gupta et al., 2021; Yu et al., 2022; Lin et al., 2023a). However, we argue that this straightforward approach is less practical for large-space decision making for several reasons. First, more decision labels naturally lead to data scarcity, since collecting sufficient training data for all labels will need significantly more cost. This issue is also accompanied by the issue of rare labels in the long tail of highly skewed distributions suf- fering severely from the lack of sufficient training instances (Zhang et al., 2023), leading to overgener- alization bias where frequent labels are more likely to be predicted compared with rare ones (Xu et al., 2022). Second, the non-semantic logit-based rep- resentation for decision labels in classifier also makes the model hard to generalize to rarely seen labels, and adaptable to unseen ones in train- ing. This issue also impairs the applicability of the decision makion model to real-worid scenarios, such as recommendation (Xiong et al., 2022) and task-oriented parsing (Zhao et al., 2022a), where the decision space may expand rapidly and crucial labels may be absent in training.
Many discriminative natural language under- standing (NLU) tasks have large label spaces. Learning such process of large-space decision making particularly challenging due to lack of training instances per label and the dif- ficulty of selection among many fine-grained labels. Inspired by dense retrieval methods for passage finding in open-domain QA, we propose a reformulation of large-space dis- criminative NI.U lasks learning-to-reerieee task, leading to novel solution named Dense Decision Retrieval (DDR {o). Instead of pre- dicting fine-grained decisions as logits, DDR adopts dual-encodee architecture that learns to predict by retrieving from a decision thesaurus. This approach not only leverages rich indi- rect supervision signals from tus-to--c learning resources for dense retrieval, also leads to enhanced prediction generalizability with a semantically meaningful representation the large decision space. When evaluated on tasks with decision spaces ranging from hun- dreds to hundred- -housaand scales, DDR out- performs strong baselines greatly by 27.54% in P&1 on two extreme multi-label classification tasks, 1.17% in Fl score ultra-fine entity typ- ing, and 1. 1.26% in accuracy on three few-shot intent classification tasks on average.'
On the contrary, motivated by semantic similar- ity of examples annotated with identical labels, re- cent studies propose contrastive learning schemes that leverage Siamese encoding architectures maximize similarity scores of representation for positive example pairs (Henderson et al., 2020; Zhang et al., 2020a; Dahiya et al., 2021a; Mehri and Eric, 2021; Zhang et al., 2021; Xiong et al., 2022). Meanwhile, inductive bias from NLU tasks such as masked language modeling (Mehri et al., 2020; Dai et al., 2021) and natural language infer- ence (NLI; Li et al. 2022: Du al. 2022) has shown beneficial for learning on rare and unseen labels via indirect supervision (Yin et al., 2023). However, when dealing with very large decision spaces, ex- isting methods still face critical trade-offs between
1 Introduction
Many discriminative natural language understand- ing (NLU) tasks require making fine-grained deci- sions from a large candidate decision space. For example, a task-oriented dialogue system, when re- sponding to users' requests, needs to frequently de- tect their intents from hundreds of options (Zhang et al., 2020b; Ham et al.. 2020). Description-based recommendation in e-commerce needs search from millions of products in response to users' de- scriptions (Gupta et al., 2021; Xiong al., 2022).
"Code and resources are available at https://github com/luka com/luka-group//ie /DOR

---

generalizability and efficiency of prediction.
2 Related Work
Inspired by the recent success of dense retrieval methods that learn to select answer-descriptive pas- sages from millions of candidate documents for open-domain QA (Karpukhin et al., 2020; Lee et al., 2021; Zhan et al., 2021), we propose an indi- rectly supervised solution named Dense Decision Retrieval (DDR 83) DDR provides a general reformulation of large-space decision making as learning to retrieve from a semantically meaning- ful decision thesaurus constructed based on task- relevant resources. The model adopts the dual- encoder architecture from dense retrieval models to embed input texts and label descriptions, and learns predict by retrieving from the informative deci- sion thesaurus instead of predicting fine-grained decisions as logits. In this way, DDR not only leverages rich indirect supervision signals from other casy-to-consume learning resources for dense retrieval. It also leads to enhanced prediction per- formance and generalizability with semantically meaningful representation of the large decision space. We evaluate DDR on large decision spaces ranging from hundreds for few-shot intent detec- tion, ien-thousand for ultra-fine entity typing, to nundred-thousan scales for extreme classification. DDR obtains state-of-the-ar perfor- mance on 6 benchmark datasets in multiple few- shot settings, improving the most competitive base- lines by 27.5 .54% in P@1 for extreme classification, 1.17% in F1 for entity typing and 1.26% in ac- curacy for intent detection on average. Ablation studies show that both the constructed informative label thesaurus and indirect supervision from dense retrieval contribute to performance gain.
Indirect Supervision Indirectly supervised meth- ods (Roth, 2017; He et al., 2021; Yin et al., 2023) seek to transfer supervision signals from more resource- rich task to enhance specific more resource-limited task. A method of this kind of- ten involves reformulation of the target task to the source task. Previous studies have investigated us- ing source tasks such as NLI (Li et al., 2022; Yin et al., 2019; Lyu et al., 2021, inter alia), extractive QA (Wu et al., 2020; FitzGerald et al., 2018; Li et al., 2020, inter alia), abstractive QA (Zhao et al.. 2022a; Du and Ji, 2022) and conditioned gener- ation (Lu et al., 2022; Huang al., 2022b; Hsu et al., 2022, inter alia) to enhance more expensive information extraction or semantic parsing tasks. Recent studies also transformed these technolo- gies to specialized domains such as medicine (Xu et al.. 2023) and software engineering (Zhao et al.. 2022b) where model generalization and lack of an- notations are more significant challenges. There has also been foundational work which studies the informativeness of supervision signals in such set- tings (He ct al., 2021).
However, the aforementioned studies not de- signed for discriminative tasks with very large deci- sion spaces, and do not apply directly due to issues such as high inference costs (NLI) and requiring decisions to be inclusive to the input (QA). We in- stead propose dense retriever that naturally serves as a proper and efficient form of indirect supervi- sion for large-space decision making.
NLU with Large Decision Spaces Many concrete NLU tasks deal with large decision spaces, in- cluding description-based recommendation in e- commerce (Gupta et al., 2021; Xiong al., 2022) and Web search (Gupta et al., 2021), user intent detection in task-oriented dialog systems (Zhang et al., 2020b: Ham et al., 2020), and fine-grained semantic typing (Choi et al., 2018; Chen et al., 2020), etc. Previous studies for such tasks either train classifiers (Yu et al., 2022; Lin et al., 2023a) or rely on contrastive learning from scratch (Zhang et al., 2021; Xiong et al., 2022), which are gener- ally impaired by insufficiency of training data and hard generalize to rarely seen and unseen labels. These challenges motivate us to explore a practi- cal solution with indirect supervision from a dense retriever.
The technical contributions of this work are three-fold. First, we present novel and strong solution, DDR, for NLU tasks with large-space decision making that leverages indirect supervision from dense retrieval. Second, we provide semanti- cally meaningful decision thesaurus construction that further improves the decision-making ability of DDR. Third, we comprchensively verify the ef- fectiveness of DDR on tasks of fine-grained text classification, semantic typing and intent detection where the size of decision spaces range from hun- dreds to hundreds of thousands.
"For example, NLI-based inference leads to k times more inference cost where & being the size of the decision space, which is prohibitive for large-space decision making tasks.

---

DPR trained for open-domain QA
Label Thesaurus
Input T
Indirect supervision
Liwade pins regnonica
Grack Beda Arsitan
Extreme Multi-fobel Clossifccation Lincoln ltnker/ LINK-en: February 12, 1809 American
En nanuie
inilialize
inilialize
April 15. 1865) lawyer, politician, and statesman who served...
Input Text Encoder
sim(x.)
Label Thesaurus Entry Encoder E
Intent "my refund is not here yet"
Thesaurus Construction
? &r & Re * th to
fine Entity Typing:
Adncebtatie Tanage
He retired by the lover hause 1890.
you dotiee
WordNet A. LaGoOA Citiabrase for Enggish
Figure I: Overview of DDR that reformulates general large-space decision making tasks as dense retrieval. Label thesaurus constructed with detailed descriptions from publicly available resources and the dual-eneoder learns to maximize similarity between embeddings of input and label thesaurus entry with indirect supervision from a pre-trained dense retriever.
3
Method
returns ranking list of passages PR. where labels represented by higher ranked passages more likely to be relevant to the input. As demonstrated in Fig. 1, DDR also leverages indirect supervision from open-domain QA by initializing parameters of retriever with those from DPR.
We first describe the reformulation of large-space decision making task as dense retrieval ($3.1). We then introduce several automatic ways to construct label thesaurus of high quality ($3.2). Lastly, demonstrate the dual-encoder architecture of DDR (illustrated Fig. 1) retrieve decisions ($3.3).
3.2 Thesaurus Construction
3.1 Preliminary
Depending on the target task, the decision the- saurus can provide knowledge about the decision space from various perspectives. Given label names, we create informative descriptions for each label automatically without extra expenses. We can refer to publicly available resources for construct- ing thesaurus of high quality. To name few: 1) Lexicographical knowledge bases: dictionaries such as WordNet (Miller, 1995) and Wiktionary provide accurate definitions as well as plentiful use cases to help understand words and phrases.
The mechanism of dense retrieval for general decision-making in large spaces can be formal- ized as follows. Given a textual input, such as "Illy Ground Espresso Classico Coffee: Classico, classic roast coffee has lingering sweetness and delicate..." for description-base based product recom- mendation, a model learns to infer their correspond- ing labels by mapping input to textual descriptions a large label thesaurus (such as names, brands, component, etc.), instead of simply predicting the label index.
2) Large Language Models: LLMs trained on in- structions like ChatGPT and Vicuna- 13B (Chiang et al., 2023) able generate comprehensive and reasonable descriptions of terms when they are prompted for explanation.
Formally speaking, given textual input x, de- cision space covering L labels and label thesaurus that contains corresponding entries (label descrip- tions) D = dd1,d2,-.. d2...., dr.). We first split every entry into text passages of maximum length (e.g.. 100 words) as the basic retrieval units and get M total passages P = {P1, P2..... PM }2. A retriever R. : (z, P) + PR is function that takes as input a text sequence and the label thesaurus P and
3) Training examples: directly adopting input text when labels with detailed information also appear as example input, or aggregating input texts from multiple examples assigned with the same label if they share very high-level semantic similarities.
We describe thesaurus construction in practice for each investigated downstream task in $4.
'Since a descl-desriiptive entry, such as those in XMC tasks, is sometimes too long to encode with a Transformer eneoder, we break it down into multiple passages to avoid the quadratic dependency w.r.t input length.
"https://en "https://en.wikktionary www.wwiwrttttiiiggg.....o.

---

3.3 Learning to Retrieve Decisions
Karpukhin et al., 2020). If not otherwise specified. train with positive and in-batch negatives to ob- tain a weak DDR model firstly; we then retrieve label thesaurus with the weak model for training examples and leverage wrongly predicted label de- scription as hard negatives, which are augmented to training data to obtain a stronger DDR.
Encoders Since a key motivation of DDR is to leverage indirect supervision from retrieval tasks, we adopt the dual-encoder architecture similar to a dense retriever such DPR (Karpukhin et al., 2020), where two sentence encoders with special tokens (BERT (Devlin al., 2019) by default), Fiz and Ep. are leveraged to represent input text x and passage p individually.". The similarity between input and passage is computed using dot product of their vectors denoted by sim(x,p)
We note that, different from open-domain QA where one positive pair can be considered per time with Eq. 1, considering multiple positive pairs is very important for tasks where the decision space though large but follows highly skewed distribu- tions, when in-batch negatives are adopted for model update. In the mini-batch, multiple inputs may share same labels that are popular in the head of long-tail tail label distribution. In this case, these in- put instances have multiple positive passages: one randomly sampled from label passages as normal and some from positives of other in-batch inputs.
Training The objective of the reformulated large- space decision making problem is to optimize the dual encoders such that relevant pairs of input text and label description (thesaurus entry) should have higher similarity than irrelevant pairs in the embed- ding space. Hence, the overall learning objective is a process of contrastive learning.
To tackle single-label classification where each instance contains one input z;, one relevant (posi- tive) passage x.." along with irrelevant (negative) passages Pij. we minimize the negative log likeli- hood of the positive label description
Inference After training is completed, we encode all passages in label thesaurus once with Ep and in- dex them using FAISS offine (Johnson et al., 2019). Given new input, we obtain its embedding with E, and retrieve the top-k passages with embed- dings closest to that of input.
L(zi, (LBisPP,PP.11 .P,1
Prin)
(1)
(sim(zen"))
Experiments
lolog eximlx)p)) )
+ Ej= simm
We evaluate DDR on three NLU tasks with large deicsion spaces in two criteria, 1) a minority of label space is observed during training: Extreme Multi-label Classification (54.1) and Entity Typing ($4.2), or 2) limited amounts of examples are avail- able per label: Intent Classification (54.3). We investigate decision spaces of up to hundreds of thousands. Dataset statisties are shown in Appx. Tab. 1.
We extend to multi-label classification tasks where each input has multiple positive passages and more than one of them participates in model update si- multancously. Accordingly, the cross entropy loss is minimized to encourage similarity between m positive pairs instead:
L(x;, P. LL,,P)
Plme Ps.1 Pin))
4.1
Extreme Multi-label Classification
log
En= sim(snpts) + Ej= sim(wspis))
Task Many real-world applications can be formu- lated as eXtreme Multi-label Classification (XMC) problems, where relevant labels from the of an extremely large size are expected to be predicted given text input. Considering the ever-growing la- bel set with newly added products or websites and the time-consuming example-label pairs collection, we follow the few-shot XMC setting (Gupta et al., 2021; Xiong et al., 2022) to choose relevant labels from all available seen and unseen labels. 5
Positive, negative and hard negative passages: the decision thesaurus entries that describe true labels in thesaurus for each input are deemed as positive entries
positive. In-batch negatives for other input in the same mini-batch, making the computation more efficient while achieving better performance compared with random pas- sages or other negative entries (Gillick et al., 2019;
Forr example, in cases where a BERT model is used, the [CLS] token is used for representation of input text and the decision thesaurus entries
"By randomly sampling 1% and 5% labels from the large decision space, we only use their instances for model training and predict on the test set covering the whole label space.

---

Dataset
Setting
Train Dev
Test
Labels
and scalability, and SiameseXML (Dahiya et al., 2021a) based on novel probabilistic model meld Siamese architectures with high-capacity ex- treme classifiers; 3) methods specifically designed to improve performance with few-shot labels avail- able, including ZestXML (Gupla et al., 2021) that learns to project data point's features close to the features of its relevant labels through highly spar- sified linear transform, and MACLR (Xiong et al., 2022) that pre-trains Transformer- based encoders with a self-supervised contrastive loss.
Extreme Multi-label Classification
1%
6227
LF-Amazon-1314
134,835 131,073
5%
29,719
1%
13,920
LF AF-o
177,515 312,330
5%
62,704
Ultra-fine Entity Typing
UFET
Crowd 1998 1998 1998
10.331
Intent Classification
5-shot 385
BANKING
1540
3080
77
10-shot 770
5-shot 320
Implementation Details The label space of LF- WikiSee is composed of relevant pas- sage titles, hence the corresponding full content should be available in Wikipedia and use them as the label thesaurus." For relevant product ad- vertisement and recommendation, concrete product description provides sufficient information as the label thesaurus for Ammazon-131K We extract product textual descriptions mainly from Amazon data in XMC benchmark (Bhatia et al., 201 6) and public Amazon product meta datasets (Ni et al., 2019), leading to 95.67% of labels well docu- mented. As introduced in $3.2, we then use Chat- GPT 10 to obtain relevant information for remain- ing undocumented labels 11,
HWU
1076
1076
64
10-shot 640
5-shot 750
CLINC
3000
4500
150
10-shot 1500
Table Statistics and experimental settings on datasets with label spaces ranging from 64 to 320K.
Datasets and Metries The public benchmark dataset L.F-Amazon-1318 collects pairs of relevant products with textual descriptions for commercial recommendation, while L.F-WiiiSeeAloo-31 aims to link relevant Wikipedia passages for refer- ence (Bhatia et al., 2016; Gupta et al., 2021).
We use metrics widely adopted by XMC (Chien et al., 2023) and IR (Thakur et al., 2021) literature: precision@" @k for the proportion of the top-k pre- dicted labels to be true labels, and recall@k for the proportion of true labels found in the top-k predictions. 8
We perform label thesaurus construction on Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz with 32 CPUs and 8 cores per CPU. It takes 12.12 minutes to construct label thesaurus for LF- WikiSSeAAls-3300K, while 28.24 minutes for LF- Amazon-1318 131K. We leverage the same infrastruc- ture to prepare label thesaurus in following tasks unless otherwise specified.
Baselines We compare DDR with strong XMC basclines in three categories: 1) Transformer-based models that learn semantically meaningful sen- tence embeddings with siamese or triplet network structures, including Sentence-BERT (Reimers and Gurevych, 2019) and MPNet (Song et al., 2020); 2) competitive methods originally proposed for scal- able and accurate predictions over extremely large label spaces, including XR-Linear (Yu et al., 2022) that recursively learns to traverse an input from the root of a hierarchical label tree to a few leaf node clusters, Astec (Dahiya et al., 2021b) with four sub-tasks for varying trade-offs between accuracy
Due to a lack of explicit knowledge about the connection between the majority of labels and their examples for training, DDR only learns to distin- guish positive example-label pairs from in-batch negatives without further negative mining. For in- ference, with the constructed label thesaurus, we first retrieve the large label space with test ex- ample based on Sentence-BERT embeddings of the title and the first sentence in an unsupervised way. After training DDR with few-shot labels, we mimic the practice of DPR+BM25 for open-
DPR was trained the clean, text-portion articles from the Wikipedia dump question answering. Lists such as the "See also" section have been removed during pre- processing. Hence example-label pairs in LF- WikiiieeeAess 320K are unseen during DPR training and it fair to fine-tune DPR on this task.
Sinne the specific dump version is unelear, we lonk for different resources (e.g.. latest dump, train and test passages, Wikipedia API search) and cover 99.95% of labels in the end. "ht tps: tps://chatt openai. com// com/ "Similarly, for following tusks, we obtain label informa- tion by prompting M. if the label definition or description is unavailable from existing resources.
*Following (Xiong et al., 2022), we use A = {1,3, 5} for precision @k and k = (1,3, 5, 10, 100) for recall

---

1% Labels
5% Labels
Method
Precision @3
Recall @s
Precision @3
Recall @
@1
@
@
@
@
10
@ 100
@1
@
@
@ 10
@ 100
LF-Amazon- -mlK
XR-Linear
1.53
0.57
0.36
0.67
0.75
0.78
0.81
0.92
5.09
2.09
1.32
2.36
2.86
3.02
3.18
3.74
Astec
0,94
0.44
0.29
0.55
0.78
0.84
0.91
1.13
3,94
1.92
1.26
2.31
2.34
3.66
4.00
4.96
XiL
1.45
0.56
0.35
0.84
0.96
1.00
1.03
1.16
5.36
2.23
1.41
3.15
3.89
4.08
4.27
4.82
ZestXML.
10.10
9.19
7.34
5.63
14.46
18.61
23.73
32.69
12.33
10.00
8.71
6.84
17.19
21.97
28.19
46,49
Sentence-BERT
12.64
9.82
7.80
6.97
15.34
19.74
25.33
43.53
15.47
12.24
9.64
8.63
19.23
24.40
30.82
49.22
MPNet
14.78
11.55
8.97
8.28
18.24
22.84
28.54
45.89
15.03
11.88 9.28
8.47
18.74
23.69
29.93
48.84
MACLR
18.74
16.07
12.52
10.73
25.44 31.89
39.17
57.55
19.56
16.19 12.64
11.15
25.65
32. 18 39.63 58.45
DDR (w/o thesaurus)
15.39
10.92 7.87
8,25
19.15 20
31,51
54.32
21.21
13.77 10.23
12.02
24.18 30.29
38.19 58.75
DDR (scratch)
19.55
14.48
11.50
10.87
22.50
29:00
37.80
56.53
23.77
16.48
12.75
13.31
25.54
31.85
40.46 60.30
DDR (full)
22.39
16.38
12.40
12.57
25.64
31.44
38.44
57.72
25.22
17.96 13.63
14.18
27.94
34.40
41.72
61.17
LF-WikiSee Also-320K
XR-Linear
1.24
0.57
0.37
0,42
0.58
0.63
0.68
0.76
4.69
2.20
1.46
1.82
2.41
2.63
2,82
3.42
Astec
1.25
0.60
0.41
0.69
0,98
1.11
1.27
1.56
5.90
2.80
1.86
3.26
4.49
4.95
5.49
6.83
SiameseMML
1.81
0.75
0.48
1.03
1.26
1.33
1.41
1.67
6.83
3.15
2.06
3.88
5.15
5.56
6.02
7.09
ZestXML.
8.74
6.78
5.41
4.68
9.70
12.21
15.73
24.98
10.06
8.11
6.60
5.33
11.49
14.74
19.57
40.46
Sentence-BERT
16.30
12.62
10.08
9.30
18.92
23.78
30.40
52.92
18.47
14.19
11.29
10.82
21.55
26.77
33.92
57.02
MPNet
17.14
12.64
9.96
9.98
18.98 23.45
29.67
50.75
18.59
13.99 11.08
10.89
21.12 26
32.82
54.70
MACLR
19.09 14.57 11.53
11.39
22.34 27.63
34.81
57.92
20,99
15.57 12.26
12.59
23.94 29.41
36.78
59.81
DDR (w/o thesaurus)
17.52 15.93 14.23
4.89
13.11 18.88
25.93
46.93
18.14 16.65
15.39
5.12 14.10
21.81
30.00
52.54
DDR (scratch)
21.26
15.81 13.08
11.16
22.52
29.55
40.14
64,
25.04
18.95 15
12.
25.70
32.
18
40.24
66.22
DDR (full)
24.70
18.45
14.69
13.04
26.31 32.99
41.08
65.39
27.78
19.98 15.66
15.03 28.87 35.47 43.35 68.66
Table 2: Results of few-shot XMC where the training subset covers 1% (left) and 5% (right) labels from the whole set. DDR outperforms the second best MACLR in both settings of two datasets. Indirect supervision from DPR boosts performance against training from scratch (the second row from the bottom), while label thesaurus construction improves accuracy over those using textual label names (the third row from the bottom).
domain QA (Karpukhin al., 2020) for reranking, where prediction scores on test set from DDR and Bentence-BERY are linearly combined as the final ranking score.
fine-grained words phrases that describe the type(s) of that specific mention (Xu al., 2022). Consider the sentence "He had blamed Israel for failing to implement its agreements." Besides per- son and male, the mention "He" has other very specific types that can be inferred from the context, such as leader or official for the "blamed" behavior and "implement its agreements" affair. Ultra-fine entity typing has a broad impact on various NLP tasks that depend on type understanding, including coreference resolution (Onoe and Durrett, 2020), event detection (Le and Nguyen, 2021) and relation extraction (Zhou and Chen, 2022).
Results We report performance of DDR with(out) indirect supervision from dense retrieval for open- domain QA in Tab. 2. On average, DDR boosts performance on LF-AAaaon-113 K by 8.50% and Wl-Wikiseeelloo320K by 21.73% compared with the second best baseline MACLR. When only 1% labels are seen during training, we observe that DDR shows much better transferability than other tra-trdi models, c.g., average performance improved by 12.59% over MACLR and 39.80% over MPNet. As more labels are available for fine-tuning the model, the performance gain from DDR over others becomes even more significant: 17.64% over MACLR and 43.38% over MPNet under 5% labels setting.
Datasets and Metrics We leverage the UFET dataset (Choi et al., 2018) to evaluate benefits of DDR with indirect supervision from dense retrieval. Among 10,331 entity types, accurately selecting finer labels (121 labels such as engineer) was more challenging to predict than coarse granine labels (9 labels such as person), and this issue is exacerbated when dealing with ultra-fine types (10,201 labels such as flight engineer). Following recent entity typing literature (Li et al., 2022; Du et al., 2022), we train DDR on (originally provided) limited
4.2 Ultra-fine Entity Typing
Task Entities can often be described by very fine grained-types (Choi et al., 2018) and the ultra-fine entity typing task aims at predicting one or more

---

Method
Precision
Recall
F1
WordNet and Wiktionary, which covers 99.99% of the whole label set. The time cost for prior label thesaurus construction is around 2.07 minutes.
BiLSTM
48.1
23.2
31.3
LabelGCN
50.3
29.2
36.9
LDET
51.5
33.0
40.1
Initialized with the original DPR checkpoint pre- trained on open-domain QA datasets, DDR firstly
Box-TTypes
52.8
38.8
44.8
LRN
54.5
38.9
45.4
UniST
50.2
49.6
49.9
optimizes the model given positive example-label pairs and in in-btcch negatives. With the fine-tuned model, we then perform dense retrieval the label for each training example, keeping label docu- ments with high scores but not in the true label set as hard negatives. DDR further updates the model with these additional hard negatives. For inference of mult-label entity type classification, we adopt labels with retrieval scores higher than a threshold that leads to the best Fl score the development set.
MLMET
53.6
45.3
49.1
LITE
52.4
48.9
50.6
Context- TE
53.7
49.4
51.5
DDR (wo thesaurus)
51.6
46.0
48.6
DDR (scratch)
53.0
50.5
$1.7
DDR (full)
51.9
52.3
52.1
Table 3: Results of UFET lask. DDR outperforms competitive baselines with (upper 5 methods) without (lower 3 methods) inductive bias from task pre-training.
crowd-sourced examples without relying on distant resources such as knowledge bases or head words from the Gigaword corpus. We follow prior stud- ies (Choi et al., 2018) to evaluate macro-averaged precision, recall and Fl.
Results In Tab. 3, we show performance of DDR and other entity typing methods. DDR obtains the state-of-the-art score over the the best base- line training from scratch (LRN) by 6.7 and the best baseline with inductive bias from the language model (Context-TE) by 0.6.
Baselines We consider two categories of compet- itive entity typing models as baselines: 1) meth- ods capturing the example-label and label-label relations, e.g., BiLSTM (Choi al., 2018) that concatenates the context representation Iearned by bidirectional LSTM and the mention repre- sentation learned by a CNN, LabelGCN (Xiong et al., 2019) that learns to encode global label co-occurrence statistics and their word-level sim- ilarities, LRN (Liu et al., 2021a) that models the coarse-to-fine label dependency as causal chains, Box4Types (Onoe et al., 2021) that captures hierar- chies of types as topological relations of boxes, and UniST (Huang et al., 2022a) that conduct name- based label ranking; 2) methods leveraging induc- tive bias from pre-trained models for entity typ- ing, e.g., MLMET (Dai et al., 2021) that utilizes the pretrained BERT to predict the most probable words for "[MASK]" earlier incorporated around the mention type labels, LITE (Li et al., 2022) and Context-TE (Du et al., 2022) that both lever- age indirect supervision from pre-trained natural language inference.
4.3 Few-shot Intent Classification
Task As a fundamental element in task-oriented dialog systems, intent detection is normally con- ducted in the NLU component for identifying a user's intent given an utterance (Ham et al., 2020). Recently, accurately identifying intents in the few- shot setting has attracted much attention due data scarcity issues resulted from the cost of data collection as well as privacy and ethical concerns (Lin et al., 2023b). Following the few-shot intent detection benchmark (Zhang et al., 2022), we focus on the challenging 5-shot and 10-shot settings.
Datasets and Metrics To evaluate the effective- ness of DDR for NLU with large decision spaces, pick three challenging intent datasets with relatively large number of semantically simi- lar intent labels. Banking77 (Casanueva al., 2020) is single-domain dataset that provides very fine-grained 77 intents in a Banking domain. HWU64 (Liu et al., 2021b) is a multi-domain (21 domains) dataset recorded by a home assis- tant robot including 64 intents ranging from set- ting alarms, playing music, search, movie rec- ommendation. CLINC150 (Larson al., 2019) prompts crowd workers to provide questions commands in the manner they would interact with an artificially intelligent assistant covering 150 in- tent classes over 10 domains. We report accuracy
Implementation Details The dataset UFET first asked crowd workers to annotate entity's types and then used WordNet (Miller, 1995) to expand these types automatically by generating all their syn- onyms and hypernyms based on the most common sense. Therefore, we automatically obtain label thesaurus entries from definitions and examples in

---

Banking77
HWU64
CLINC150
Method
5-shot
10-shot
5-shot
10-shot
5-shot
10-shot
w/ Data Augmentation
ICDA-X'S (+1x)
80.29
86.72
81.32
85.59
91.16
93.71
ICDA-S (+4x)
81.95
87.37
81.97
86.25
91.22
93.98
ICDA-M (+16x)
84.01
88.64
81.84
87.36
91.93
94.71
ICDA-L (+64x)
83.90
89.12
81.97
86,94
92.41
94.73
CCD--X
83.90
89.79
82.45
87.41
92.62
94.84
w/o Data Augmentation
USE
76.29
84.23
77.79
83.75
87.82
90.85
ConveRT
75.32
83.32
76.95
82.65
89.22
92.62
USE+ConveRT
77.75
85.19
80.01
85.83
90.49
93.26
Conv BERT
83.63
83.77
92.10
+ MLM
83,99
84.52
92.75
+ MLM Example
84.09
83.44
92.35
+ Combined
85.95
86.28
93,97
DNNC
80,40
86.71
80.46
84.72
91.02
93.76
CPFT
80.86
87.20
82.03
87.13
92,34
94.18
Pretraining
76.75
84.83
76.02
82.96
88.19
91.55
Context-TFF
80.76
85.53
DDR (w/o thesaurus)
78.86
84,16
80.30
83.74
88.86
91.58
DDR (scratch)
82. 11
85.97
80.76
84.71
88.58
91.16
DDR (full)
83.86 (X L)
88.25 (M)
84.29 (X L)
86.34 (5)
92.71 (XL)
94.58 (M)
Table 4: Few-shot Intent Detection Accuracy three benchmark datasets. DDR achieves much higher accuracy than existing strong learning baselines without data augmentation, while competitive with ICDA requiring a considerable amount augmented data. ICDA prepares additional synthetic data with the scale ranging from the same amount (ICDA-XS) to 128 times (ICDA-XL) of the original few-shot train size. In the bottom mark performance of DDR with the most comparable ICDA variant. Results not available in original papers are marked as
also
for this single-label classification task.
from six public intent datasets to discriminate se- mantically similar utterances.
Baselines There are two families of intent learn- ing algorithms to cope with limited amounts of example-label pairs: 1) Classifiers based on sentence embeddings from PLMs, including USE (Yang et al., 2020) embeds sentences from 16 languages into a single semantic space using a multi-task trained dual ducoden, ConveRT (Hen- derson et al., 2020) that pretrains a dual-encoder model on Reddit Data by maximizing the similar- ity score of embeddings of positive input-response pairs; ConvBERT (Mehri et al., 2020) that adopts BERT-base model pre-trained on large open- domain dialogue corpus; 2) methods leverage se- mantic similarity between utterances from different users, e.g. +ommiincd (Mehri and Eric, 2021) that extends the ConvBERT model with task- adaptive masked language modelling (MLM) and infers the intent of the utterance based on the simi- larity to the examples corresponding to each intent, DNNC (Zhang et al., 2020a) that leverages BERT- style pairwise encoding train binary classifier that estimates the best matched training example for a user input, CPFT (Zhang et al., 2021) that con- ducts contrastive pre-training on example sentences
Different from DDR and prior baselines, ICDA (Lin et al., 2023a), most recent work tack- les the challenge of limited data for intent detection by generating high-quality synthetic training data. It first fine-tunes PL. on small seed of training data for new data point synthesization, and then employs pointwise V-information (PVT) based fil- tering remove unhelpful ones. 12
Implementation Details Based on the observation that utterances labeled by identical intents share very similar sentence-level semantic meanings, we first utilize the whole set of unlabeled training ex- amples represent their corresponding pseudo labels predicted by Sentence-BERT (Reimers and Gurevych, 2019). We perform prediction on a sin- gle NVIDIA RTX A5000 GPU and it takes 4.17 minutes to complete label thesaurus construction for Banking77, 6.15 minutes for HWU64 and 7.61
Althoough the learning and augmentation direction are orthogonal hence incommensurable. we display results with augmented training data as well to refiect the advantage of DDR over ICDA in terms of their required scale of augmented synthetic data to achieve similar good performance.

---

Sentence-BEi (Ist round)
DDR (2nd round)
We appreciate the reviewers for their insightful comments and suggestions. The logo for DDR used in this paper is sourced from the Wikipedia page"! 13 under CC BY-SA 3.0, for which we appre- ciate the contribution of the Wikipedia community. from dense retrieval. Moreover, by representing decision spaces with thesaurus, we provide rich semantic knowledge of each label to improve the understanding of labels for dense decision retriev- ers. Experiments on benchmark datasets show the effectiveness of our method on decision spaces scaling from hundreds of candidates to hundreds of thousands of candidates. Future work can extend our method more large-space decision making tasks, especially in the low-resource setting.
Dataset
Setting
5-shot
84.29
83.86
Banking77
10-shot
88.02
88.25
5-shot
82.81
84.29
HWL66
10-shot
85.69
86.34
5-shot
91.82
92.71
CLINC150
10-shot
94.07
94.58
Table 5: Impact on intent detection from label thesaurus constructed with predictions from Sentence-BERT and DDR for the unlabeled whole set of the training set.
A Acknowledgement
minutes for CLINC150. After training DDR with this first version of label thesaurus, we then update pseudo labels for each training example with pre- dictions from DDR for the recond-round training. Regardless of the basis for label thesaurus con- struction, DDR is trained for two phases in each round: only few-shot positive example-label pairs and in-batch negatives are used in the first phase, while labels wrongly predicted for training exam- ples from the prior phase are used as additional hard negatives in the second phase. After the sec- ond round of training, the latest DDR makes pre- dictions on the whole set of training examples for the final label thesaurus construction, from which DDR retrieves the label with the highest score as the prediction.
Nan Xu is supported by the USC PhD Fel- lowship. Fei Wang is supported by the Annen- berg Fellowship and the Amazon ML Fellowship. Mingtao Dong is supported by the Provost's Re- search Fellowship. Muhao Chen is supported by the NSF Grant IIS 2105329, the NSF Grant ITE 2333736, the DARPA MCS program under Con- tract No. 166001192409 with the United States Office Of Naval Research, a Cisco Research Award, two Amazon Research Awards, and Keston Re- search Award. The computing of this work has been partly supported by a subaward of NSF Cloud- bank 1925001 through UCSD.
Results Tab. 4 presents performance of different intent detection methods 5- and 10-shot set- ting. Without extra training data synthesization, we observe significant performance gain from the proposed DDR over existing baselines. Although ICDA obtains enhanced accuracy by increasing data augmentation scale, DDR is able to achieve comparable performance without crafting 4x even 128x synthetic instances. In Tab. 5, we additionally study the quality of constructed label thesaurus in accordance with the prediction the unlabeled whole training set from Sentence-BERT and DDR after the first round of training. We find that with a more accurate pseudo connection between example and label leads to a higher quality of label thesaurus construction, and finally benefits intent detection.
Limitations
The proposed DDR leverages the dual-encoder ar- chitecture with the inner dot product to compute embedding similarity, obtaining statc-of-the-a per- formance on three investigated challenging large- space decision making tasks. More expressive mod- els for embedding input and label thesaurus, such as joint encoding by cross-encoder is not dis- cussed. Moreover, other ways model the connec- tion between input and label thesaurus entry, such as using
weweennee----eueenee language model to generate the label name given input text and label thesaurus entry, is not explored yet. We believe adopting more advanced dense retrieval algorithms can further promote performance for large-space decision making. We leave this an exciting fu- ture direction.
5 Conclusions
In this paper, we focus on discriminative NLU tasks with large decision spaces. By reformulat- ing these tasks as learning-to-retrivw tasks, we are able to leverage rich indirect supervision signals
"https://en.wwikieed" wwtp:/ippeiia..og/iiiiiiiiieee

---

References
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Prabhu, and M. Varma. 2016. The extreme classi- fication repository: Multi-label datasets and code.
Tnigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38 38-45,00 On- line. Association for Computational Linguistics.
Jiangshu Du, Wenpeng Yin, Congying Xia, and Philip S Yu. 2022. Learning to select from multiple options. arXiv preprint 2rxxv:2222.00300
Muhao Chen, Hongming Zhang, Haoyu Wang, and
Dan Roth. 2020. What are you trying do? se- mantic typing of event processes. In Proceedings of the 24th Conference on Computational Natural Lan- guage Learming, pages 531-542, Online. Association for Computational Linguistics.
Xinya Du and Heng Ji. 2022. Retrieval-augmentt generative question answering for event argument extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
pages
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL pars- ing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Vol- une 1: Long Pupers), pages 2051 12060, Melhourne, Australia. Association for Computational I Linguistics.
Eli Chien, Jiong Zhang, Cho-Jui Hsieh, Jyun- Yu Jiang, Wei-Cheng Chang, Olgica Milenkovic, and Hsiang- Fu Yu. 2023. Pina: Leveraging side information in extreme multi-labo classification via predicted instance neighborhood aggregation. arXiv preprint arXiv:2305. 12349.
Daniel Gillick. Sayali Kulkarni, Larry Lansing, Alessan- dro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Con- ference on Computational Natural Language Learn- ing (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Linguistics.
Funsol Choi, Omer Levy, Yejin Choi, and Luke Zettle- moyer. 2018. Ultra-fine entity typing. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87-96, Melbourne, Australia. Association for Computational Linguistics.
Nilesh Gupta, Sakina Bohra, Yashoteja Prabhu, Saurabh Purohit, and Manik Varma. 2021. Generalized zero- shot extreme multi-label learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 527-535.
Kunal Dahiya, Ananye Agarwal. Deepak Saini, K Gu- ruraj. Jian Jiao, Amit Singh, Sumeet Agarwal, Pu- rushottam Kar, and Manik Varma. 2021a. Siame- sexml: Siamese networks meet extreme classifiers with 100m labels. In International Conference on Machine Learning, pages -330- PMLR.
Donghoon Ham, Jeong- -wann-GGannaa Lee. Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of the 58th Anmual Meeting of the Associ- ation for Computational Linguistics, pages 583-592, Online. Association for Computational Linguistics.
Kunal Dahiya, Deepak Saini, Anshul Mittal, Ankush Shaw, Kushal Dave, Akshay Soni, Himanshu Jain, Sumeet Agarwal, and Manik Varma. 2021b. Deep- xml: A deep extreme multi-label learning framework applied to short text documents. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 3 31-39.
Hangfeng He, Mingyuan Zhang, Qiang Ning, and Dan Roth. 2021. Foreseeing the benefits of incidental supervision. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 1782-1800, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from a
Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung- Tssung-Hiien Wen, and Ivan Vulić, 2020. ConveRT: Efficient and accurate conversational repre- sentations from transformers. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 2161-2174, Online, Association for Computa- tional Linguistics.
masked language model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Confer- ence Natural Language Processing (Volume I:
Long Papers), pages 1790-1799, Online, Association for Computational Linguistics.

---

I-Hung Hsu, Kuan- Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai- Wei Chang, and Nanyun Peng. 2022. DEGREE: A data-efficient generation-based event extraction model. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1890-1908, Seattle, United States. Association for Computational Linguistics.
Papers), pages 6634-6647, Online. Association for Computational Linguistics.
Bangzheng Li. Wempeng Yin, and Muhao Chen. 2022. Ultra-fine entity typing with indirect supervision from natural language inference. Transactions of the Association for Computational Linguistics, 10:607- 622.
Xiaoya Li, Jingrong Feng, Yuxian Meng. Qinghong
James Y. Huang, Bangzheng Li, Jiashu Xu, and Muhao Chen. 2022a. Unified semantic typing with mean- ingful label inference. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642-2654, Seattle, United States. Association for Computational Lin- guistics.
Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5849-5855 On- line. Association for Computational Linguisties.
Yen-Ting Lin, Alexandros Papangelis, Seokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani Tur. 2023a. Selective in-context data augmentation for intent de- tection using pointwise v-information. arXiv preprint 230.:033..
Kuan-Hao Huang. I-Hung Hsu, Prem Natarajan, Kai- Wei Chang, and Nanyun Peng. 2022b. Multilin- gual generative language models for zero-shot cross- lingual event argument extraction. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 4633-4646, Dublin, Ireland. Association for Computational Linguistics.
Yen-Ting Lin, Alexandros Papangelis, Scokhwan Kim, Sungjin Lee, Devamanyu Hazarika, Mahdi Namazi- far, Di Jin, Yang Liu, and Dilek Hakkani-Tuus Tur. 2023b. Selective in-context data augmentation for intent de- tection using pointwise V-information. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguisties, pages 1463-1476, Dubrovnik, Croatia. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. JEEE Transactions on Big Data, 7(3):535-547.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov. Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6770 6781, Online. Association for Computational Linguistics.
Qing Liu, Hongyu Lin, Xinyan Xiao, Xianpei Han, Le Sun, and Hua Wu. 2021a. Fine-grained entity typing via label reasoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4611-4622, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph J. Peper. Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 131 1311-1316, Hong Kong, China, Association for Computational Linguistics.
Xingkun Liu, Arash Eshghi. Pawel Swietojanski, and Verena Rieser. 2021b. Benchmarking natural lan- guage understanding services for building conversa- tional agents. In Increasing Naturalness and Flexi- bility in Spoken Dialogue Interaction: 10th Interna- tional Workshop on Spoken Dialogue Systems, pages 165-183. Springer.
Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma. and Muhao Chen. 2022. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575 6594, Abu Dhabi. United Arab Emirates. Association for Computational Linguistics.
Duong Le and Thien Huu Nguyen. 2021. Fine-grained event trigger detection. In Proceedings of the 16th Conference of the European Chapter of the Associ- ation for Computational Linguistics: Main Volume, pages 2745-2752, Online. Association for Computa- tional Linguistics.
Qing Lyu, Hongming Zhang, Elior Sulem, and Dan Roth. 2021. Zero-shot event extraction via transfer learning: Challenges and insights. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages Association for Computational Linguistics.
Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021. Learning dense representations of phrases at scale. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long
Online.

---

Shikib Mehri and Mihail Eric. 2021. Example-driven intent prediction with observers. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2979-2992, Online. Association for Computational Linguisties.
of information retrieval models. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA Coreference resolution as query- based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953-6963, Online. Association for Computational Linguistics.
Shikib Mehri, Mihail Erie, and Dilek Hakkani-Tuu 2020. Dialoglue: A natural language understanding benchmark for task-oriented dialogue. arXiv preprint arXiv:2009.13500.
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 773-784, Minneapolis, Min- nesota. Association for Computational Linguisties.
George A Miller 1995. Wordnet: lexical database for english. Communications of the ACM, 38( 1 1):39-41.
Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019. Justifying recommendations using distantlyllabeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP)) pages 188-197, Hong Kong. China. Association for Computational Lin- guistics.
Yuanhao Xiong, Wei-Cheng Chang, Cho-Jui Hsich, Hsiang-Fu Fu Yu, and Inderjit Dhillon, 2022. Extreme Zero-Shot learning for extreme text classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5455-5468, Seattle, United States. Association for Computational Linguistics.
Yasumasa Onoe, Michael Boratko, Andrew McCallum. and Greg Durrett. 2021, Modeling fine- fine-grained entity types with box embeddings. In Proceedings of the 59th Annual Meeting of the Association for Compu- lational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Vol- ume Long Papers), pages 2051-200 2064, Online. As- sociation for Computational Linguistics.
Jiashu Xu, Mingyu Derek Ma, and Muhao Chen. 2023. Can nli provide proper indirect supervision for low- resource biomedical relation extraction? In Proceed- ings of the 2023 Conference of the Association for Computational Linguisties: Human Language Tech- nologies.
Yasumasa Onoe and Greg Durrett. 2020. Interpretable entity representations through large-scale typing. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 612-624, Online. As- sociation for Computational Linguistics.
Nan Xu, Fei Wang. Bangzheng Li, Mingtao Dong, and Muhao Chen. 2022. Does your model classify en- tities reasonably? diagnosing and mitigating spuri- ous correlations in entity typing. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 8642-8658, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Nils Reimers and Tryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP. pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego. Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sen- tence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 87-94, Online. Association for Computational Linguistics.
Dan Roth. 2017. Incidental supervision: Moving be- yond supervised learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelli- gence, February 4-9, 2017, San Francisco, Califor- nia, USA, pages 4885-4890.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857. 16867.
Wenpeng Yin. Muhao Chen, Ben Zhou, Qiang Ning, Kai-Wa Wei Chang, and Dan Roth. 2023. Indirectly supervised natural language processing. ings of the 61 Annual Meeting of the Association for Computational Linguistics ( (Volume 6: Tutorial Ab- stracts), pages 32-40 40, Toronto, Canada, Association for Computational Linguistics.
Proceed-
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation

---

Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking sero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI-P-JCCIIP), pages 39 14-39223, Hong Kong, China. Association for Computational Linguistics.
EACL 2023, pages 1092-1106, Dubrovnik, Croatia. Association for Computational Linguistics.
Wenting Zhao, Konstantine Arkoudas, Weiqi Sun, and Claire Cardie. 2022a. Compositional task-oriented parsing as abstractive question answering. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4418 24424, Seattle, United States. Association for Computational Linguistics.
Hsiang-Fu Yu, Jiong Zhang, Wei-Cheng Chang, Jyun- Yu Jiang, Wei Li, and Cho-Jui Hsieh. 2022. Pecos: Prediction for enormous and correlated output spaces. In Proceedings of the 28th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, pages 4848 4849.
Zhongkai Zhao, Bonan Kou, Mohamed Yilmaz Ibrahim, Muhao Chen, and Tianyi Zhang. 2022b. Knowledge- based version incompatibility detection for deep learning. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 247-251.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Pro- ceedings of the 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, pages 1503- 1512.
Wenxuan Zhou and Muhao Chen. 2022. An improved baseline for centecee-eveel relation extraction. In Pro- ceedings of the 2nd Conference of the Asia-Pacity Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 161-168, Online only. Association for Computational Linguistics.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021. Few-shot intent detection via contrastive pre-training and fine-tuning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 1902-1912, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020a. Discrimina- tive nearest neighbor few-shot intent detection by transferring natural language inference. In Proceed- ings of the 2020 Conference on Empiricul Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Yao Wan, Zhiwei Liu, Ye Liu, Caiming Xiong, and Philip Yu. 2022. Are pre-trained transformers robust in intent classi- fication? a missing ingredient in evaluation of out- of-scope intent detection. In Proceedings of the 4th Workshop NLP for Conversational Al, pages 12- 20, Dublin, Ireland. Association for Computational Linguistics.
Jianguo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wang, Philip Yu, Richard Socher, and Caiming Xiong. 2020b. Find or classify? dual strategy for slot-value predictions
multi-domain dialog state tracking. In Proceedings of the Ninth Joint Confer- ence on Lexical and Computational Semantics, pages 154-167, Barcelona, Spain (Online). Association for Computational Linguistics.
Ruohong Zhang, Fau-Shian Wang. Yiming Yang, Dong- han Yu, Tom Vu, and Likun Lei. 2023. Long-tailed extreme multi-label text classification by the retrieval of generated pseudo label descriptions. Findings of the Association for Computational Linguistics:


Reviews:
[{"review_id": "DevaSJCzbh", "reviewer": "Reviewer_YGUm", "summary": "", "questions": "What is the computational cost for constructing the thesaurus in each scenario?\n\nCan the thesaurus be expanded at any point?\n\nTo accomodate more tasks, presumably the thesaurus would have to grow continually along with the growing label space. Can the proposed thesaurus scale in such scenarios?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback and would like to clarify some concerns as follows:\n\n**1% and 5% setting of Extreme Multi-label Classification task on LF-Amazon-131K and LF-WikiSeeAlso-320K**\n- One of our goals is to demonstrate the effectiveness of indirect supervision in **low-resource settings**. We therefore follow the low-shot setting introduced in [1] where the training subset covers 1%/5% labels from the whole set, instead of using a full-shot setting where training data are enough.\n- To comprehensively evaluate benefits of dense retrieval under different scenarios of large decision spaces with limited per-label training data, we also consider 1) thousands of labels with different granularities for entity typing in Section 4.2, and 2) few-shot setting on intent classification in Section 4.3. \n\n[1] Xiong, Yuanhao, et al. \"Extreme zero-shot learning for extreme text classification.\" arXiv preprint arXiv:2112.08652 (2021).\n\n**Details about thesaurus construction**\n\nWe describe thesaurus construction details in Implementation Details paragraph for three tasks from lines 354 to 365 in Section 4.1, from lines 452 to 460 in Section 4.2 and from lines 542 to 551 in Section 4.3. We will also release the code for thesaurus construction soon.\n\n**Construction cost  of label thesaurus**\n\nWe perform label thesaurus construction on Intel(R) Xeon(R) Gold 5217 CPU @ 3.00GHz with 32 CPUs and 8 cores per CPU. For labels thesaurus constructed based on training samples in few-shot intent classification (last three datasets), we predict pseudo labels with Sentence-BERT on a single NVIDIA RTX A5000 GPU. The concrete time cost per dataset is as follows:\n- LF-WikiSeeAlso-320K: 59.12 min\n- LF-Amazon-131K: 28.24 min\n- UFET: 2.07 min\n- Banking77: 4.17 min\n- HWU64: 6.45 min\n- CLINC150: 7.61min\n \n**Expansion (scalability) of label thesaurus**\n\nYes, the thesaurus is freely expandable. As discussed from lines 54 to 63 in Introduction, simply training a classifier is not adaptable to unseen labels. The proposed DDR, however, can expand thesaurus by adding the new label as a new entry in the same approach to initial (automated) construction. \n\nThe design of DDR is sufficiently flexible to support continual learning as the label space grows in real-world applications: 1) fine-tuning DDR when new instances and labels are added, and 2) encoding new label texts into thesaurus with fast inference supported by FAISS.\n\n**Ablation study of label  thesaurus**\n\nFor all three investigated tasks with large decision spaces, we’ve reported results of not using the thesaurus but only textual label names as shown in rows DDR (w/o thesaurus) in Table 1, 2 and 3. We observe that the proposed DDR (full) benefits greatly from the constructed thesaurus, obtaining much higher accuracy than DDR (w/o thesaurus).\n\n**Thesaurus construction wrt. Dataset**\n\nTraditional classifiers for NLU tasks with large decision spaces have to be trained separately for different datasets or decision spaces. However, our label thesaurus is based on the concrete label space and then constructed automatically according to publically available resources. It could be reused for different tasks/datasets with label overlapping, e.g., training subset covers 1% and 5% labels for extreme multi-label classification tasks discussed in Section 4.1. It is also freely expandable.\n\n\n**Novelty over standard dense retrieval methods or retrieval-augmented decision making models**\n\n- Difference from standard dense retrieval methods: this paper investigates how to leverage indirect supervision from dense passage retrieval to address NLU with large label spaces. Specifically, we provide semantically meaningful decision thesaurus construction that further improves the decision-making ability of DDR.\n- Difference from retrieval-augmented decision making models: retrieval-augmented approaches seek to retrieve auxiliary knowledge from external resources to augment the contextual information of an autoregresive LM, for which we do not see obvious connection to our work that adapts pretrained bi-encoders for decision retrieval.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The issue of rare labels is important, and the proposed model has good performance in limited-label scenarios.\n\nThe construction of a thesaurus based on external sources including lexical knowledge bases and large language models is interesting and shown to be effective.", "reasons_to_reject": "LF-Amazon-131K and LF-WikiSeeAlso-320K results are reported for 1% and 5% labels, but this two settings alone are not entirely convincing.\n\nThe model depends heavily on the thesaurus, but ablation or detailed discussion of thesaurus construction and its impact on performance is lacking. Besides, thesaurus must be constructed for each dataset, such as Wikipedia or Amazon products data.\n\nOverall, I do not think the method has significant novelty over standard dense retrieval methods or retrieval-augmented decision making models.", "questions_to_authors": "What is the computational cost for constructing the thesaurus in each scenario?\n\nCan the thesaurus be expanded at any point?\n\nTo accomodate more tasks, presumably the thesaurus would have to grow continually along with the growing label space. Can the proposed thesaurus scale in such scenarios?", "paper_topic_and_main_contributions": "This paper proposes a retrieval-based method for classification. The method is motivated by the issue of rare labels in long-tailed label distributions and generalization to unseen labels.\n\nThe frames decision-making as a retrieval over a thesaurus (as well as corpus of training data), and shows performance gains in limited-label scenarios.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rNvPg6PPGu", "reviewer": "Reviewer_E5JK", "summary": "", "questions": "1. Could you please clarify the source used as the 'label thesaurus' in the intent classification task?\n2. What does the 'w/o thesaurus' setup in Table 1, Table 2, and Table 3 refer to?\n3. It is noted that large language models are mentioned as potential thesaurus sources, but they were not utilized in the experiments.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank your helpful suggestions and would like to clarify following details:\n\n**Label thesaurus construction for the intent classification task**\n\n- There are two rounds of label thesaurus construction as mentioned in lines 542-551.\n    - The first round: In 5- and 10-shot settings, we only have 5 or 10 samples per label, but abundant unlabeled examples. Therefore, we predict labels for examples and then concatenate example texts assigned with the same pseudo label to construct label thesaurus. Specifically, example labels are predicted according to their embedding similarity with the limited labeled samples based on Sentence-BERT.\n    - The second round: After training DDR with this first round of label thesaurus, we then update pseudo labels for each training example with predictions from DDR and start the second-round training with the new label thesaurus. \n- Normally, the second round of pseudo labels from DDP are more accurate than those from Sentence-BERT. We further show the benefits of this thesaurus construction process in Table 4, where label thesaurus after the second round leads to better classification performance than that after the first round.\n\n**'w/o thesaurus' setup in Table 1, Table 2, and Table 3**\n\n'w/o thesaurus means no thesaurus construction but using only label names to represent the decision. \n\n**Utilization of Large Language Models in experiments for label  thesaurus construction**\n\nWe use LLMs as a supplementary resource for thesaurus construction if necessary. For instance, we can find descriptions of 99.95% labels from LF-WikiSeeAlso-320K (line 358) from lexicographical knowledge bases. We then use the LLM (in this context, ChatGPT) to obtain relevant information for remaining 0.05% labels.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces an innovative approach, Dense Decision Retrieval (DDR), to tackle discriminative NLU tasks using dense retrieval.\n2. The proposed DDR model exhibits consistent improvements over baseline methods in the three tasks.\n", "reasons_to_reject": "1. The authors may consider addressing some missing descriptions and typos that are affecting the readability of the paper.", "questions_to_authors": "1. Could you please clarify the source used as the 'label thesaurus' in the intent classification task?\n2. What does the 'w/o thesaurus' setup in Table 1, Table 2, and Table 3 refer to?\n3. It is noted that large language models are mentioned as potential thesaurus sources, but they were not utilized in the experiments.\n", "paper_topic_and_main_contributions": "The paper presents a novel approach called Dense Decision Retrieval (DDR) to address discriminative NLU tasks using dense retrieval. The authors argue that traditional classifier methods struggle with tasks involving a large number of labels or unseen labels due to data sparsity. To overcome this, they propose representing the label space with descriptive labels known as the 'label thesaurus' and employ the indirect supervision of DPR to retrieve label thesaurus entries. The results demonstrate that the DDR model consistently outperforms baselines on three tasks where the number of training labels is much smaller than the entire label space.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "030 - descriminative → discriminative\n045 - having more decision labels naturally leads to data scarcity\n267 - thesauras → thesaurus\n580 - descriminative → discriminative"}, "extra_text": {}}, {"review_id": "PdTISS8g0o", "reviewer": "Reviewer_FPkP", "summary": "", "questions": "- It seems that the proposed solution may face challenges in generalization when applied to scenarios outside the scope of the Wikipedia or WordNet collection. Can you offer further justification for this limitation?\n- The relevance between the experimental tasks and \"decision making\" appears to be limited. Could you please provide additional justification for this connection?\n- In sec 3.2, the LLM approach is mentioned as a potential method for thesaurus construction but is not well explored in the study. Could you please elaborate on the reasons for this decision?\n- [minor] Given the impressive reasoning capabilities demonstrated by recent LLMs, do you believe that a generation-based framework could be a more promising solution compared to a retrieval-based framework? Why?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your suggestions and would like to clarify some details as follows:\n\n**Relevance to \"decision making\"**\n\nFrom lines 194 to 205 in Section 3.1, we give a formal definition of the NLU tasks with large decision spaces, where ranking labels’ relevance to input text is considered as a decision making process. We understand that clarifying the difference from decision making in planning or reinforcement learning studies is necessary, and will do so at the beginning of the paper.\n\n**Generalizable construction process of label thesaurus**\n\n- We propose a **general solution** to address large label spaces of NLU tasks by reformulating it as a learning-to-retrieve task. It allows the flexibility of adding new labels from different task learning resources, and reusing shared ones among them.\n\n- Moreover, there is **no restriction** on strategies for label thesaurus construction, and we provide three possible ways from lines 217 to 230 in Section 3. We can rely on resources including definition/explanation for general concepts such as Wikipedia and WordNet. Besides, LLMs could be used to generate essential information for labels missing in existing lexicographical knowledge bases.\n\n**Investigated tasks with large label spaces**\n\nOur experiments covered various application areas: 1) product and document recommendation, 2) entity typing for type understanding, 3) intent classification  in task-oriented dialog systems. These tasks also covered  label space sizes ranging from hundreds, tens of thousands, to hundreds of thousands. In addition to these representative large-decision-space NLU tasks, we are happy to work on more evaluation on tasks of this kind upon the release of our implementation. \n\n**Technical contributions** \n\nIn this paper, we present a novel and strong solution, DDR, for NLU tasks with large-space decision making that leverages indirect supervision from dense retrieval. We also provide semantically meaningful decision thesaurus construction that further improves the decision-making ability of DDR. \n\n**Scenarios outside the scope of the Wikipedia or WordNet collection**\n\nAs mentioned from lines 221 to 230 in Section 3.2 Thesaurus Construction, we can refer to two other possible resources for label thesaurus construction: Large Language Models and Training examples. LLMs could be used to generate essential information for labels missing in existing lexicographical knowledge bases, while training examples could be used for thesaurus construction if instances with the same label share sentence-level semantic similarities.\n\n\n**Generation-based framework for NLU tasks with large label spaces**\n\nFor now, LLMs still face critical, unresolved challenges to address NLU tasks with large label spaces. Specifically, a large number of label names cannot fit in the limited context of autoregressive LMs. Accurately matching next token probability to multiple tokens of label names is also non-trivial, and labels rarely seen in the pre-training corpus of LLMs are less likely to be generated. Hence, using PLMs and retrieval-based decision retrieval still represent a much more practical solution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed task, large-space decision making, is important and may opens up opportunities for addressing more real-world challenges.\n- The presented solution is clear and paper is well written\n- Experiments are comprehensive\n", "reasons_to_reject": "- The experimental tasks appear to lack direct relevance to decision making, and it seems that the construction of the thesaurus heavily relies on these specific tasks rather than being a general construction method applicable to various open-domain tasks.\n- One of the main challenges in addressing large-space decision making lies in the vastness of the decision space, coupled with the lack of naturally-existing of ground truth explanations for each decision. However, the experiments seem to have been conducted with carefully designed tasks, where classification labels align well with existing knowledge bases like Wikipedia and WordNet. This controlled environment may not fully reflect the real-world decision-making scenarios as the authors mention in their motivation.\n- Technical novelty and contribution is limited.", "questions_to_authors": "- It seems that the proposed solution may face challenges in generalization when applied to scenarios outside the scope of the Wikipedia or WordNet collection. Can you offer further justification for this limitation?\n- The relevance between the experimental tasks and \"decision making\" appears to be limited. Could you please provide additional justification for this connection?\n- In sec 3.2, the LLM approach is mentioned as a potential method for thesaurus construction but is not well explored in the study. Could you please elaborate on the reasons for this decision?\n- [minor] Given the impressive reasoning capabilities demonstrated by recent LLMs, do you believe that a generation-based framework could be a more promising solution compared to a retrieval-based framework? Why?\n", "paper_topic_and_main_contributions": "The paper introduces a novel solution called Dense Decision Retrieval (DDR) for addressing the challenges of large label spaces in discriminative natural language understanding (NLU) tasks. The traditional approach of predicting fine-grained decisions as logits is replaced by a bi-encoder retrieval architecture that learns to retrieve decision thesaurus. Experimental results demonstrate that DDR outperforms baselines in various tasks like multi-label classification, entity typing, and intent classification.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper utilized dense retrieval to address discriminative NLU tasks. The proposed task is well motivated and may yield some more real-world impact. The paper is well written in general. Reviewers praised the comprehensive experiments with consistent improvements of the proposed model over baselines. On the other hand, two reviewers raised concerns about limited technical novelty of the proposed method given the existing work. The readability of the paper can also be improved. Overall, I think this paper proposes a solid approach (while not super innovative) to a well-motivated task with convincing experiments. I hope the reviewers' comments on the weaknesses will help improve the quality of the next version of the paper.
Decision: Accept-Findings