GLEN: Generative Retrieval via Lexical Index Learning
Authors: Sunkyung Lee
Minjin Choi
Jongwuk Lee
Venue: EMNLP 2023 Main (2023)
Keywords: Generative retrieval
Document retrieval
Lexical index

Abstract: Generative retrieval shed light on a new paradigm of document retrieval, aiming to directly generate the identifier of a relevant document for a query. While it takes advantage of bypassing the construction of auxiliary index structures, existing studies face two significant challenges: (i) the discrepancy between the knowledge of pre-trained language models and identifiers and (ii) the gap between training and inference that poses difficulty in learning to rank. To overcome these challenges, we propose a novel generative retrieval method, namely Generative retrieval via LExical iNdex learning (GLEN). For training, GLEN effectively exploits a dynamic lexical identifier using a two-phase index learning strategy, enabling it to learn meaningful lexical identifiers and relevance signals between queries and documents. For inference, GLEN utilizes collision-free inference, using identifier weights to rank documents without additional overhead. Experimental results prove that GLEN achieves state-of-the-art or competitive performance against existing generative retrieval methods on various benchmark datasets, e.g., NQ320k, MS MARCO, and BEIR. The code is available at https://github.com/skleee/GLEN.

GLEN: Generative Retrieval via Lexical Index Learning
Sunkyung Lee', Minjin Choi', Jongwuk Leej Sungkyunkwan University, Republic of Korea {sk1027, zxcvxd, jongwuklee) edu
Abstract
ment identifiers. As depicted in Table 1, existing studies are categorized into two pillars: identiieer types and identifier learning strategies.
Generative retrieval shed light new paradigm of document retrieval, aiming to di- rectly generate the identifier of relevant doc- ument for query. While it takes advantage bypassing the construction of auxiliary in- dex structures, existing studies face two signif- icant challenges: (i) the discrepancy between the knowledge of pre-trained language models and identifiers and (ii) the gap between train- ing and inference that poses difficulty in learn- ing to rank. overcome these challenges, we propose novel generative retrieval method, namely Generative retrieval via LExical iNdex learning (GLEN). For training, GLEN effec- tively exploits a dynamic lexical identifier using a two-phase index learning strategy, enabling it learn meaningful lexical identifiers and relevance signals between queries and docu- ments. For inference, GLEN utilizes collision- free inference, using identifier weights to rank documents without additional overhead. Exper- imental results prove that GLEN achieves state- of-the-art or competitive performance against existing generative retrieval methods on var- ious benchmark datasets, e.g., NQ320k, MS MARCO, and BEIR. The code is available at https://gi /hpptbb//gtuub..... http:/ttt........eoeeeetee.e som/skkeee//LEN
Identifier types. Some canonical works employ mu- meric identifiers for document representation, e.g., hierarchical clustering using document representa- tions (Tay et al., 2022: Wang et al., 2022) and prod- uct quantization (Zhou et al., 2022). However, nu- merie identifiers struggle to fully exploit the knowl- edge of pre-trainen language models (PLMs) due to semantic discrepancy between natural language and numeric identifiers. Other studies pre-define lexical identifiers using titles (Lee al., 2023) URLs (Zhou al., 2022; Ren et al., 2023). Al- though they can narrow the semantic gap between PLM knowledge and identifiers, such information may be inadequate for representing documents and does not exist depending on the dataset.
Identifier learning strategies. Depending on the strategy of training identifiers, we refer to an iden- tifier as being static if it does not change during training. Meanwhile, when an identifier evolves during training, we refer to it as being dynamic. Static identifiers may lead to performance bottle- neck in generalizing for unseen documents during training. To overcome this limitation, Sun et al. (2023) proposed method to dynamically learn nu- merie identifiers. However, it still non non-trivial to learn appropriate identifiers due to the task discrep- ancy between training and inference; the models focus generating the identifier during training, but they need to rank documents during inference.
1 Introduction
Generative retrieval has emerged as an innova- tive approach to document retrieval (Metzler et al., 2021). Unlike conventional retrieval methods fol- lowing the teteee--rrrreeetee--aaittt pipeline, unifies an entire search process. Specifically, di- rectly generates the identifier of a relevant docu- ment for a given query. By formulating the entire search process as a it bypasses an auxiliary index structure and can be optimized through end-to-end learning.
To this end, we introduce a new generative re- trieval method, namely Generative retrieval via L.xiccl iNdex learning (GLEN) using the dynamic lexical identifier in right- bothtm cell in Table 1. The key novelty of GLEN is (i) define lexical identifiers from documents using the knowledge of PLMs, (ii) to learn them from relevance between queries and documents, and (iii) to effectively rank documents with the same identifier for inference.
problem,
Despite these benefits, generative retrieval faces major challenges in how to define and train docu-
"Equal contribution Conresponding author

---

Training of GLEN. utilizes a two-phase index learning strategy define lexical identifiers and to learn them dynamically. First, in the keyword- based ID assignment phase, GLEN defines identi- fiers from documents and learns them. To alleviate the diserepancy between the knowledge of PLMs and the semantics of identifiers, we depict identi- fiers in the pre-trained vocabulary space leverag- ing self-supervised signals by extracting key terms from documents. The model can learn how map its knowledge to the unique nature of identifiers.
Numeric
Lexical
GENRE (2021),
DSI (2022),
SEAL (2022),
Ultron-PQ (2022),
CorpusBruin (2022).
Static
NCT (2022).
Ultron-URL. (2022),
DSI-QG (2023)
NpDecoding (2023),
TOME (2023)
Dynamic
GENRET (2023)
GLEN (Ours)
Table I: Category of existing generative retrieval models
based on (i) identifier types and (ii) identifier learning strategies. GLEN introduces a dynamic lexical identifier.
Then, the ranking-based ID refinement phase is used to effectively learn dynamic identifiers. We directly incorporate query-document relevance in learning through the elaborate design of two loss functions. Specifically, GLEN explicitly learns query-document relevance using pairwise ranking loss to capture the ranking relationships and point- wise retrieval loss to learn the relationship between a query and a relevant document. Thus, GLEN can generate identifiers that better encapsulate the sub- tle semantics of the query-document relationship.
2 Related Work
2.1 Document Retrieval
Document retrieval aims to seek relevant docu- ments to the user query from large document corpus. Most existing methods have followed the "innee--etriceee-thee--aat pipelinn.. Traditional sparse retrieval methods (Robertson and Walker, 1994; Formal et al., 2021; Choi al., 2022) rely on the inverted index utilizing term matching sig- nals. On the other hand, dense retrieval meth- ods (Karpukhin et al., 2020; Xiong et al., 2021; Khattab and Zaharia, 2020) calculate the vector similarity of dense representations via an approx- imate nearest neighbor index. Although dense trieval has shown a remarkable performance, the model cannot be optimized end-to-end and has a drawback in the cost of the external index structure.
Inference of GLEN. employs collision-free in- ference using an identifier weight deal with the document identifier collision problem, i.e., the same identifier can be assigned to multiple docu- ments they are semantically similar. A simple solution is to force a different identifier for those documents. However, it can make the identifier too long or potentially interfere with the semantic learn- ing of the identifier. Instead of enforcing unique- ness during training, we leverage the document identifier logits during inference to rank the col- lided documents. Notably, this sunple-yet-effectiv solution avoids high computational costs by using the generation logit as the weight.
2.2 Generative Retrieval
Apart from traditional retrieval, generative retrieval uses only a unified model (Metzler et al., 2021) that directly generates an identifier of a relevant document for a given query. As shown in Table 1, we categorize the existing methods according to how they define and train identifiers.
In summary, our key contributions are follows, (i) We propose GLEN, which learns lexical identi- fiers in a dynamic fashion. To our knowledge, is the first generative retrieval method using learning- based lexical identifiers. (ii) We devise a two-phase index learning strategy with keyword- based ID as- signment and ranking-based ID refinement to gener- ate identifiers reflecting query docymeet relevance. (iii) We present a collision-fie free inference via rank- ing using identifier weight while effectively pre- serving identifier semantics. (iv) We evaluate the ef- fectiveness of GLEN on three benchmark datasets: Natural Questions (Kwiatkowski al., 2019), MS MARCO Passage Ranking (Nguyen et al., 2016), and BEIR (Thakur et al., 2021).
Numeric identifier. Tay al. (2022) proposed the differentiable search index (DSI), firstly demon- strating that document retrieval can be accom- plished with a single model. assigns numeric identifiers in various ways, e.g., atomic, naive, and semantic. Especially, semantic identifiers are ob- tained by hierarchical k-means clustering over doc- ument representations to capture the document semantics. Wang et al. (2022) and Zhuang et al. (2023) additionally introduced the prefix-aware weight-adaptive decoder and data augmentation us- ing query generation, respectively. To improve the semantic deficiency of numeric identifiers, Zhou et al. (2022) employed product quantization. Most

---

Two-phase lexical Index learning
Collision-tree Inference
Dasy
petrn
ølyepte:
Olympie ganes- 1ist
yyyppi list 0Tympic-
a ylyptcc
ganes
1st
2.5
1.1
-02
Keyword
Indexing model
Retrieval model
Indexing
Indexing model
Retneval model
Document 2 - olynpt Myyyypc-ggammen c-ganes- 1st
extractor
model
3.5
0.7
0.9
Profe negative compiing
Document
Query
Relevant document
Query
Document 3 olympic Cmpiccgares--- -0.2
LEst alynate Gares host cities This is Tist host cities of the plampid Games, both summer and winter, since the rodern @lympte began
Positive
document
2.7
22
Negative
Keyword-based aesignment
banklng-based ID refinement
Figure I: Overview of training and inference for GLEN, For training, the keyword-based ID assignment phase is performed, which learns identifiers via self-supervised signals, followed by the ranking- based ID refinement phase to learn identifiers dynamically. For inference, GLEN generates identifiers for query, and the documents are ranked with the logits when the collision occurs. The number below the identifier token indicates the logit for each token.
recently, Sun et al. (2023) proposed an identifier learning framework to overcome the limitations of static identifiers. However, numeric identifiers inherently suffer from the difficulties of leverag- ing the knowledge of PLM due to gap between natural language and numeric values.
3.1 Task Formulation
Generative retrieval aims to autoregressively gen- erate the identifier of the relevant document for given query. Specifically, it involves computing the probability P(zlg) of generating a document identifier = for the query q.
Lexical identifier. Bevilacqua et al. (2022) pro- posed method to consider n-grams in a document identifiers using the FM-Index structure. Zhou al. (2022) and Ren al. (2023) utilized URLs as document identifier, while Chen al. (2022), Lee al. (2023), and Cao al. (2021) defined title as an identifier They can leverage the knowledge PLMs to decode identifiers, enjoying the benefit of pre-trained vocabulary space. However, external information such as URLs and titles may not exist depending on the datasets and may not adequately represent the document. To overcome these limi- tations, we define lexical identifiers by extracting keywords from documents and dynamically refine them by directly optimizing query-documen rele- vance.
P(z10) II P(zig,
(1)
where is the number of tokens in the identifier. To address the key challenges of generative trieval: (i) how define identifiers and (ii) how to train query-document relevance, we propose a dynamic lexical identifier by defining it using key- words and refining it through relevance.
3.2 Model Architecture
We propose a novel generative retrieval method, Generative retrieval via LExical iNdex learning (GLEN). Specifically, it consists of two compo- nents: (i) An indexing model takes document d as input to generate a document identifier z, and (ii) retrieval model takes query q as input to generate the identifier of relevant document.
3 Proposed Method
We describe the process of deriving identifier z from document d using the indexing model, and the retrieval model can proceed in the same way. Both models are initialized with the pre-trained language model with the Transformer architecture (Vaswani et al., 2017) and share parameters. For the index- ing model, a document representation is defined follows.
this section, we formulate the generative docu- ment retrieval task (Section 3. and present GLEN (Section 3.2), as depicted in Figure 1. To tackle the challenges of identifier design and training strategy, GLEN adopts two-phase lexical index learning (Section 3.3). For inference, we devise collision- free inference using identifier logits (Section 3.4). While maintaining its simplicity, GLEN handles identifier collisions where semantically similar doc- uments share the same lexical identifier.
d Dec((nc(d)), der),
2, arg max (d, e,); JE(I....|VII
(2)

---

where d, R is the final hidden representation of the decoder at time t. An embedding vector ej E I is the j-th vector of the word embedding matrix E E IR v where m is the dimension of embedding vectors, and [V] is the dimension of vocabulary space. Let Enc(-) and Dec(-) represent the transformer encoder and decoder, respectively, For GLEN, we define the probability of generating identifier from document d as follows.
sequeence-t--sequence cross-entropy loss.
n
Lkey =
Σ
log P(t koy d, dct),
(4)
where key is token at t-th step of key, In addition, also utilize query as input and train to infer keywords of its relevant documents.
3.3.2 Ranking- ID Refinement is crucial to incorporate query- rele- vance and learn how to generate identifiers of rel- (3) evant documents from queries in training. To this end, we design two losses: (i) pairwise ranking loss for learning ranking and (ii) pointwise retrieval loss for learning the query-identifier relationship. Consequently, GLEN can dynamically learn how to generate lexical identifiers from the relevance signal.
P(zld) = II
P( det),
t=1
P(z = =ild, det) = Softmax,(d; EE),
where P(ze = jld, det) denotes the probability that is the j-th token in the vocabulary space, and Softmax,(() is the j-th element of the softmax function output. For the original transformer de- coder, the output of each step, i.e., Ret, is fed for the next step. Here, we use the final hidden rep- resentations det for the decoder input instead of Zec. This method ensures that the decoder input does not fluctuate even if the document identifier fluctuates during training, allowing for stable train- ing. (Empiricallyy we observed about 14.4% gain in Recall 1. See Section 5.2 for details.)
Pairwise ranking loss. First, we introduce pair- wise ranking loss, incorporating query-document relevance into identifier learning. It helps the model to represent queries as close to relevant documents and far from irrelevant documents d C N, where N is set of negative documents obtained via prefix prefix-aware aware dynamic negative sampling, which will be described later. The pairwise ranking loss is defined follows.
3.3 Two-phase Lexical Index Learning
Cpir log exp(rellg, ot exp(rel(q,d") d'))
To effectively train the lexical identifier, we intro- duce a two-phase training strategy: keyword-based based ID assignment to learn the semantics of the corpus and the characteristies of identifiers and ranking- based ID resinement adjust appropriate identi- fiers that encapsulate relevance signals.
Σ exp(rel(g,d
EN
(5)
For pairwise ranking loss, we define the relevance score of query q and document d as described.
rel d) Σ q r
3.3.1 Keyword-based ID Assignment
(6)
A document identifier should be concise yet infor- mative, unlike typical natural language sentence. Due to its unique nature, it is challenging learn from scratch to assign appropriate identifiers to documents. We bridge the generation task gap be- tween the pre-trained language model task and the identifier generation task by training the model to generate representative keywords. Specifically, we choose top-n tokens with the highest tf-idf scores using BM25 (Robertson and Walker, 1994) as the keyword identifier _kyy for the document. This en- sures that the model can construct the semantics of the document from self-supervised signals ex- tracted from the corpus and naturally learns the nature of identifiers. The model learns it using
r = Softmax ....EET /TT/)- .,,
where qe E R. is the final hidden representation of the decoder for query time Note that It used as a document representation, not de. In the inference phase, the query should generate the identifier. In this regard, we exploit the identi- fier representation for representing documents, thus mitigating the gap between training and in- ference. In addition, since arg max(-) to calculate 2i in Eq. (2) is non-differentiable, Softmax (-) and temperature 7.
get r, with
If T is low enough, it yields similar effect arg max(-). However, when the model not suffi- ciently trained, r, may become collapsed regardless

---

of the document. As such, we adopt an annealed temperature, i.., max(10-55, exp(-t)) where denotes the training epochs. (See Section 5.2 for the effectiveness of annealing).
target document, i.e., we fetch documents with the same prefix for the first n tokens. If the resulting set of documents does not reach the desired count of Nneg. we opt for documents with the same prefix at the first n tokens. We repeat it by reduc- ing the length of the prefix until the set reaches Nueg documents. Our approach iteratively samples documents based on the prefix. However, the cost was negligible in our experiments, and we found it effective for ranking, as shown Section 5.2.
Pointwise retrieval loss. To ensure the model can capture the relationship between the query and the relevant document identifier, we design a pointwise retrieval loss as follows:
Cpiint
Σ t=1
(2 lg, 9<t)
3.4 Collision-free Inference
+ Adisi dist(w", (o0,, w wa))
The inference process of GLEN is straightforward: (i) we proceed over the documents for assigning identifiers to the document offline, and (ii) infer the identifiers of relevant documents from the query online. We finally assign a dynamically learned identifier each document predicted by the model. We also employ constrained decoding generate only the valid identifiers, and ranked list of docu- ments obtained by beam search.
T for & n1, 1 qi ez
where 2 indicates the identifier predicted from the positive document d' and 2, is a t-th token of =_. ez E R" is the word embedding vector of 2,. The first loss term is cross-entropy loss that maps query q to the identifier 2+ of relevant documents. It alleviates the gap between training and inference in that mapping from queries to iden- tifiers is performed in the inference. The second loss term utilizes the identifier logits w wd and allows the model to learn the relative importance of the identifier tokens, e.g., "Olympic" is more important than "list" in the example of Figure 1. Here, Adist is the hyperparameter to adjust the im- portance between the pointwise loss terms. For distance function dist(-), adopt cosine distance.
If an identifier is assigned to documents by learn- ing, the documents with similar semantics may be mapped to the same identifier, i.c., identifier colli- sion. It incurs that documents with conflicting iden- tifiers cannot be ranked. Existing studies (Wang et al., 2022: Tay et al., 2022) have appended addi- tional digits (e.g., X-X-0, X-X-1) address this problem, but such manually defined identifiers may distort the subtle semantics of the identifier. On the other hand, we do not force the identifier to be unique for semantic learning of identifiers.
The final loss is the sum of the pairwise ranking loss and the pointwise retrieval loss:
We introduce a novel solution, collision ranking using identifier logit, resolve the collision issue at inference time. Specifically, we utilize logit of each step in generating lexical identifier from query q (or document d). The relevance between a query and document using identifier logits is defined as follows.
L = Cpair + Apoint point
(8)
where Apoint is the hyperparameter to control the im- portance of the pointwise retrieval loss. It enables end-to-end optimization of the retrieval task.
Prefix-aware dynamic negative sampling. To im- prove top-ranking retrieval performance robustly, we devise prefix-un aware dynamic negative sampling for the pairwise ranking loss (Eq. (5)). As pointed out Zhan et al. (2021), dynamic hard negatives, which are sampled during training based on re- trieval results of the model itself, can effectively improve the ranking performance. To reflect the nature of the autoregressive model, we obtain set of negative documents A based the identi- fier prefix. Concretely, we determine the candidate negatives for each document in the following man- ner. Given an identifier length of n, we first take the documents that have the same identifier as the
rel d) = cos(w", w()).
(9)
For each query, we first rank the document identi- fiers via P(z]g) = II (q: e 9: e,) If multiple documents share a single identifier, they are ranked using rel, d). In this way, the colli- sion problem can be avoided without unnecessary intervention in the semantic learning of identifiers. In particular, it has the advantage that there is only a negligible additional cost for ranking since the weights of the document identifiers w", wd are al- ready used to compute P(zld), P(zlg).

---

4 Experimental Setup
Table 1. (i) Static numeric identifier. DSI (Tay et al., 2022) uses a sequence-lo-sequenee model to generate numeric identifiers built by hierarchi- cal k-means clustering. DSI-QG (Zhuang et al., 2023) and NCI (Wang et al., 2022) are built upon DSI while adopting augmented data via query generation and prefix-aware weight-adaptive de- coder, respectively. (ii) Static lexical identifier. GENRE (Cao al., 2021) utilizes title as iden- tifier. SEAL (Bevilacqua et al., 2022) generates arbitrary n-grams to retrieve relevant documents, utilizing the FM-Index structure. TOME (Ren et al., 2023) performs retrieval by generating the docu- ment URLs via a two-stage generation architecture. (iii) Dynamic numeric identifier. GENRET (Sun et al., 2023) learns how to assign numeric iden- tifiers based a discrete auto-encoding scheme. (iv) Dynamic lexical identifier. To our knowledge, GLEN is the first work to employ the dynamic lexical identifier. For details of sparse and dense retrieval models, see Section A.I.
4.1 Datasets
Natural Questions (NQ320k) (Kwiatkowski et al., 2019) consists of 320k query-document relevant pairs, 100k documents, and 7,830 test queries, which has been actively used in existing gener- ative retrieval methods (Tay et al., 2022; Wang al., 2022). We also follow the setup in Sun et al. (2023), splitting the test set into two sub- sets: seen test and unseen test. seen test consists of queries where the annotated target documents are included in the train set, while unseen test con- sists of queries where no labeled documents are included in the train set. MS MARCO passage ranking (MS MARCO) (Nguyen al., 2016) is large-scale benchmark dataset with 8.8M pas- sages collected from Bing's results and IM real- world queries. We use the official development set consisting of 6,980 queries with a full corpus, i.e. 8.8M passages, following Ren et al. (2023). BEIR (Thakur et al., 2021) is benchmark dataset for zero-shot evaluation on diverse text retrieval tasks. Following Sun et al. (2023), we assess on Arguana (Arg) (Wachsmuth et al., 2018) and NF- Corpus (NFC) (Boteva et al., 2016). For train data, we follow published train data constructed by Wang et al. (2022) for NQ320k for fair comparison. For MS MARCO, we use the MS MARCO training set, which consists of 500k queries and randomly split 1,000 queries for validation. For the generated query of the MS MARCO, we used the published predicted queries!.
4.4 Implementation Details
We initialized GLEN with T5-base (Raffel et al., 2020). The batch size is set to 128, and the model is optimized for up to 3M steps and 30K steps using the Adam optimizer with learning rates 2c- 4 and 5e-5 for keyword-based ID assignment and banking-based
ranking- based ID refiement, respectively. We use beam search with constrained decoding and beam size of 100 for inference. For two-phase lexical index learning, we set the length of document id after tuning among {2. 3, 5, 7. 10) for NQ320k and MARCO, respectively. Apoint and Adist are set to 0.5 and 0.5 after tuning in (0. 0.25, 0.5, 1, 2, 4). respectively. For т, we set it to le-5 with temperature annealing. We set the number of negative documents per query Nung to 8 after tuning in (0, 1, 2, 4, 8) and adopted in-batch negatives, where all passages for other queries in the same batch are considered negative. Further details for model architecture and training hyper- parameters can be found in Section A.I.
3,
4.2 Metries
We report Recall and MRR for NQ320k following existing works (Sun et al., 2023). MRR 10 and nDCG@ 10 the official metric of MS MARCO Passage Ranking and BEIR, respectively.
4.3 Baselines
We compare GLEN with three types of base- line models, including two sparse retrieval mod- els (BM25 (Robertson and Walker, 1994) and DocT5Query (Nogueira and Lin, 2020)), four dense retrieval models (DPR (Karpukhin et al., 2020), ANCE eXionget al., 2021), Sentence-TS, T55 (Ni et al., 2022a), and GTR (Ni et al., 2022b)), and six generative retrieval models. For generative re- trieval methods, we categorize them following the
5 Experimental Results
5.1 Main Results
Evaluation on NQ320k. Table 2 presents the re- trieval performance on the NQ320k. The key obser- vations are as follows: (i) GLEN shows outperform- ing performance among sparse and dense baselines and competitive performance with generative re-

---

NQ320K (7,830)
(6,075) MRR@
Unseen (1.755)
Model
R@1
R@
MRR 100
@@
R
MRR 100
Sparse & dense retrieval
BM25 (1994)
29.7
60.3
40.2
29.1
59.8
39.5
32.3
61.9
42.7
DocTSQuery (2020)
38.0
69.3
48.9
35.1
68.3
46.7
48.5
72.9
57.0
DPR (2020)
50.2
77.7
59.9
50.2
78.7
60.2
50.0
74.2
58.8
ANCF (2021)
50.2
78.5
60.2
49.7
79.2
60.1
52.0
75.9
60.5
Se (2022a)
53.6
83.0
64.1
53.4
83.9
63.8
56.5
79.5
64.9
GTR base (2022b)
56.0
84.4
66.2
54.4
84.7
65.3
61.9
83.2
69.6
Generative retrievul
GENRE (2021)
55.2
67.3
59.9
69.5
83.7
75.0
6.0
10.4
7.8
DSI (2022)
55.2
67.4
59.6
69.7
83.6
74.7
1.3
7.2
3.5
SEAL (2022)
59.9
81.2
67.7
DSI-QG (2023)
63.1
80.7
69.5
68.0
85.0
74.3
45,9
65.8
52.8
NCI (2022)
66.4
85.7
73.6
69.8
88.5
76.8
54.5
75.9
62.4
GENRET (2023)
68.1
88.8
75.9
70.2
90.3
77.7
62.5
83.6
70.4
TOME (2023)
66.6
GLEN (Ours)
69.1
86.0
75.4
88.9
78.5
57.6
75.9
63.9
Table 2: Performance comparison for the proposed method and baseline models for NQ320k. The best generative retrieval model is marked bold, and the second best model is underlined. number parentheses indicates the number of queries. We refer to the results of baselines reported by Sun al. (2023) and Ren et al. (2023). Results not available are denoted as
MS MARCO Dev (6,980) MRR@
BEIR (nDCG@ (10)
Model
Model
Average
NFC Arg
Sparse & dense retrieval
Sparse retrieval
BM25 (1994)
18.4
BM25 (1994)
32.0
31.5
32.5
DocTSQuery (2020)
27.2
DocTSQuery (2020)
33.9
34,9
32.8
GTR-base (20226)
34.8
Generative retrieval
Generative retrieval
DSI (2022)
6.5
DSI (2022)
3.1
NCI (2022)
2.6
4.3
DSI-QG (2023)
GENRET (2023)
12.1
12.1
12.1
NCI (2022)
GLEN (Ours)
16.8
17.6
15.9
GLEN (Ours)
20.1
Table 4: Zero-shot performance for the proposed method and baseline models for the BEIR dataset. The best gen- erative retrieval model is marked bold, and the second best model is underlined. Average means the average accuracy over two datasets. We refer to the results of baselines reported by Sun et al. (2023).
Table Performance comparison for the proposed method and baseline models for MS MARCO passage dev. The best generative retrieval model is marked bold, and the second best model is underlined. We refer to results of baselines reported by Pradeep et al. (2023).
trieval methods. Specifically, GLEN outperforms the best competitive generative retrieval model by +1.5% and +3.3% on Recall@) in the NQ320k full and seen test set, respectively. (ii) While GLEN performs second best among generative models on unseen tests, it surpasses other models in seen tests, indicating that the ranking-based ID refinement effectively optimizes the identifiers. (iii) The gen- erative retrieval methods with dynamic identifiers exhibit higher performance than those with static identifiers. confirms their ability to capture the intricate semantics of documents and queries via end-to-end optimization.
Evaluation on MS MARCO. Table shows the retrieval performance on the MS MARCO Passage Ranking set. GLEN yields a clear improvement over the best competitive generative retrieval meth- ods and BM25 (Robertson and Walker, 1994) by 15.6% and 9.3% in MRR @ 10, respectively. Exist- ing generative retrieval methods still struggle to memorize the knowledge of the corpus and thus often fail to work in large-scale corpora (Pradeep al., 2023). In contrast, GLEN successfully works on large-scale corpora owing to learning identifiers via directly learning the relevance of queries and documents.

---

Dataset (Metric)
Query subset (# queries)
Collision ranking
Random ranking
NQ320k (7,830)
Model
R@1
R@10
MRR@ 100
NQ320k
All (7,830)
69.1
68.6
GLEN
69.1
86.0
75.4
(R@1) 1)
Collision (441)
41.5
31.9
w/o keyword
48.0
77.2
58.3
MARCO
All (6,980)
20.1
18.7
w/o annealing
67.6
85.2
74.2
(MRR @ 10)
Collision (2.170)
17.5
12.9
decoder input Eege
60.4
83.7
69.1
w/o pairwise loss
67.3
84.7
73.8
Table 5: Performance comparison of GLEN with differ- ent solutions for collision in inference. Random ranking denotes a randomly ranked result for colliding docu- ments. We report an average of 10k runs.
w/o pointwise loss
68.3
85.8
74.8
random negatives
68.3
85.5
74.7
Table 6: Ablation study of GLEN. Note that keyword means keyword-based ID assignment, and annealing means temperature annealing for 7 in Eq. (6).
Prefix-aware BM25 Random None
68.5
1intta 100
75
R81
68
MItta
71.55
5.2 In-depth Analysis
67.5
74
Effect of collision-free inference. As shown in Table 5, we validate the effectiveness of the colli- sion ranking by comparing with random ranking, which randomizes the ranking of colliding doc- uments. For a thorough comparison, we further constructed a subset (i.e., collision) by collecting queries where at least one labeled document is col- liding documents. For NQ320k and MS MARCO datasets, we found that collision ranking improves performance by 30.2% and 35.5%, respectively, random ranking for the collision query subset. MS MARCO dataset has higher ratio of collision queries due its larger corpora than NQ320k (e.g., 8.8M vs. 109K). verifies that colliding documents are effectively ranked without additional cost using identifier weight, while the semantics of identifiers are well preserved.
67
73.5
3000 4000
2000
Training steps
Training steps
(a) Recall @
(b) MRR @ 100
Figure 2: Performance comparison GLEN depending the negative sampling strategy by training step for ranking-based ID refinement phase on NQ320k.
Zero-shot evaluation on BEIR. We thoroughly investigate the generalization capability of GLEN via the zero-shot performance on the BEIR (Thakur al., 2021) dataset after training on NQ320k, re- ported in Table 4. GLEN shows the best average accuracy in generative retrieval methods, surpass- ing the best competitive generative model by 38.8% on average. We also observe that the dynamic iden- tifiers (i.e., GLEN and GENRET) consistently out- perform static identifiers, showing that they are more effective at capturing the complex semantics of documents and can be generalized in a zero-shot setting.
Effect of prefix-aware dynamic negative. Figure depicts the effect of prefix-aware dynamic neg- ative sampling along the training step. The prefix- aware dynamic negative exhibits the most effee- tive performance for robust ranking, showing 1.0% and 1 1.2% gains in R@ over the random negative and BM25 negative sampling, respectively. Further- more, prefix-aware negatives delivered a 1.2% per- formance improvement in R @ 1, compared not using hard negatives. It highlights that the nature of an autoregressive model effectively reflected via the prefix, and dynamically sampled negatives are conducive to learning.
For dynamic identifiers, GLEN outperforms GENRET in zero-shot evaluation. The difference in performance stems from two primary distinc- tions: (i) identifier types and (ii) solutions to iden- tifier collisions. GLEN can assign a generalized identifier to a new document by leveraging knowl- edge from the PLM, while GENRET may have dif- ficulty allocating numeric identifiers to new docu- ments. Besides, the same identifier can be assigned to semantically similar documents, especially if the documents are out-of-domain. It often happens since models not trained differentiate be- tween them. To rank these same identifier docu- ments, GLEN introduces collision-free inference to break the tic, while GENRET cannot distinguish between them and places them randomly.
A Ablation study. Table 6 presents the effective- ness of various strategies in GLEN on NQ320k. (i) The proposed keyword-based ID assignment phase remarkably improves accuracy by 44.0% in R Notably, successfully allows the model to learn the unique nature of identifiers based on sulf-super signals, thus enabling it to leverage

---

Query
would you represent ge cell cycle of neuron
Relevance
Original title
GLEN ID
NCT ID
Keyword ID
V
Ge phase
(#1) phase-phase-cell
(#14) 22-17-18-4
phase-cells-nutri
X
G2 phase
(#2) phase-phase-cell
(#2) 21-28-3-8
phase-phase-cell
x Cell cycle checkpoint
(#3) point-check-cell
(49) 1-27-21-1
point-check-cell
x Neuron
(#57) neurons-nervvos-nnerr
(#1) 1-27-2-1
neurons-nerrous-nner
x
Neurotransmission
(#62) neuro-trans-apt
(#3) 1-27-2-2
neuro-trans-apt
Table 7: A retrieval example of GLEN and NCT on NQ320k. The number in parentheses denotes the rank of
document for each model. Keyword ID is the extracted identifier used at the keyword-bass ID assignment phase of GLEN, Note that the tokenization process for GLEN ID and Keyword ID simplified.
the knowledge of PLM. (ii) The temperature nealing for identifier representation (in Eq. 6) contributes to stable training, yielding 2.2% gain in @@ (iii) Replacing decoder input der to acts the accuracy drops by 12.6%, suggesting that det for decoder input enhances stable training. (iv) The proposed pairwisc ranking loss and the pointwise retrieval loss contribute to the accuracy compared to adopting single loss by up 2.6% in R@ 1. study focusing
our knowledge, it is the first work introducing dy- namic lexical identifier for generative retrieval. Ex- perimental results demonstrate that GLEN achieves state-of-the-art or competitive performance among generative retrieval methods.
Limitations
This work proposes new generative retrieval ap- proach, GLEN, that dynamically learns lexical identifiers. Though verified the performance of GLEN on large corpus, it still exhibits a performance gap with longstanding conventional retrieval methods (e.g., ColBERTv2 (Santhanam et al., 2022), LexMAE (Shen et al., 2023)), which still hold state-of-the-art performance. This implies that generative retrieval still faces limitations in learning large-scale corpus. It may require using large models designing training schemes, leaving many research problems to be explored. addition, we experimentally verified the proposed model a zero-shot setting. We showed that it outperforms the generative retrieval method but still performs less than sparse retrieval. It suggests that generative retrieval still suffers from limited generalization compared to well vell-designed dense or sparse retrieval models.
Case study. Table exhibits on identifier clucidate how generative retrieval is performed. We take one query from NQ320k and show the retrieval results from GLEN and NCL. Our observations are as follows: (i) GLEN can assign the same identifier to documents with similar se- mantics (e.g.. "GO phase" and "G2 phase"), but it effectively ranks them via collision-free inference. It indicates that GLEN successfully learns the sub- de semantics of document-identtffer relationships. (ii) GLEN refines identifiers through the refinement phase, changing from a keyword ID "phase-cells- nutri" to GLEN ID "phase paasse-phae--elll (iii) Static numeric identifiers in NCT fail to reflect the seman- tics of the documents. Although some documents are semantically similar (e.g., "GO phase" and "G2 phasc"), they have completely different identifiers.
6 Conclusion
Ethics Statement
In this paper, we proposed novel lexical index learning method. namely Generative retrieval via LExical iNdex learning (GLEN).. To effectively tackle the critical challenges of generative retrieval, we adopt dynamic lexical identifier learning framework that can mitigate (i) the discrepancy between the knowledge of pre-trained language models and identifiers and (ii) the discrepancy be- tween training and inference. GLEN successfully enjoys the benefits of a dynamic lexical document identifier via delicately devised two- two-phase index learning scheme and collision-free inference. To
This work complies with the ethics of ACL. The sci- entific artifacts we used are available for research with permissive licenses. The use of the artifacts in this paper adheres to their intended use.
Acknowledgments
This work was supported by Institute of Informa- tion & communications Technology Planning & Evaluation (UTPP) grant funded by the Korea gov- ermment (MSIT) (No. 2019-0-00421, No. 2022-0- 00680, and RS- 2-203-002219919

---

References
Donald Metzler, Yi Tay, Dara Bahri, and Mare Najork. 2021. Rethinking search: making domain experts out of dilettantes. SIGIR Forum, 55(1); 13:1):13
Michele Bevilacqua, Giuseppe Ottaviano, Patrick S. H. Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Generating substrings as document identifiers. In NeurlPS.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NewrlPS.
Vera Boteva, Demian Gholipour Ghalandari, Artem Sokolov, and Stefan Riezler. 2016. A full-text learn- ing to rank dataset for medical information retrieval. In ECIR, pages 716-722.
Jianmo Ni, Gustavo Hernández Åbrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022a. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. In Findings of the ACI, pages 1864-1874.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In ICLR.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gus- tavo Heráández Ábrego, Ji Ma. Vincent Zhao, Yi Luan, Keith B. Hall, Ming-We Chang, and Yinfei Yang. 2022b. Large dual encoders are generalizable retrievers. In EMNLP, pages 9844 9855.
Jiangui Chen, Ruging Zhang, Jiafeng Guo, Yiqun Liu, Yixing Fan, and Xueqi Cheng. 2022. Corpusbrain: Pre-train generative retrieval model knowledge- intensive language tasks. In CIKM, pages 191-200.
Eunseong Choi, Sunkyung Lee, Minjin Choi, Hyeseon Ko, Young-In Song, and Jongwuk Lee. 2022. Spade: Improving sparse representations using a dual doe- ument encoder for first-stage retrieval. In CIKM, pages 272-282.
Rodrigo Nogueira and Jimmy Lin. 2020. From doc2query to doctutiquery. Online preprint.
Ronak Pradeep. Kai Hui, Jai Gupta, Ádám Dániel Lelkes, Honglei Zhuang, Jimmy Lin, Donald Met- zler, and Vinh Q Tran. 2023. How does generative retrieval scale to millions passages? CoRR.
Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: sparse lexical and expan- sion model for first stage ranking. In SIGIR, pages 2288-221 2292.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res.,
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Tevatron: An efficient and flexible toolkit for neural retrieval. In SIGIR, pages 3120-3124.
Ruiyang Ren, Wayne Xin Zhao, Jing Liu, Hua Wu, Ji- Rong Wen, and Haifeng Wang. 2023. TOME: A two-stage approach for model-based retrieval. In ACI. pages 6102-6114,
Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. 2021. Scaling deep contrastive learning batch size under memory limited setup. In Repl ACI-- IJCNLP, pages pages 316-321.
Stephen E. Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR, pages 232-241.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In EMNLP. pages 6769-6781.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In NAACL, pages 3715- 3734.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over BERT. In SIGIR, pages 39-48.
Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiao- long Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2023. Lexmae: Lexicon Lexicon-bottleneenn bottlenecked pretrain- ing for large-scale retrieval. In ICLR.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wer Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trunx. Assoc. Comput. Linguistics, 7:452- 466.
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. Learning to tokenize for generative retrieval. CoRR.
Hyunji Lee, Jae Young Kim, Hoyeon Chang, Hanseok Oh, Sohee Yang, Vladimir Karpukhin, Yi Lu, and Minjoon Seo. 2023. Nonparametric decoding for generative retrieval. Findings of the ACL, pages 12642-12661.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer memory as a differentiable search index. In NewrIPS.

---

Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In NeurIPS Datasets and Benchmarks.
2021) is an evaluation benchmark dataset of 18 pub- licly available datasets from diverse text retrieval tasks and domains, which is widely used for eval- uating the generalization capabilities of models. The task of Arguana and NFCorpus is argument retrieval and bio-medical information retrieval, re- spectively.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In NeurlPSS, pages
A.1.2 Metrics
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In ACL, pages 241-251.
To measure the effectiveness, cepted metries for information retrieval, including recall, mean reciprocal rank (MRR), and normal- ized discounted cumulative gain (nDCG), mean average precision (MAP) with retrieval size K. Recall is defined as Σe rel, where is the po- &
widely
Yujing Wang, Yingyan Hou, Haonan Wang. Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A neural corpus indexer for document retrieval. In NeurIPS.
sition in the list, l is the number of relevant doc- uments and rel; {0, 1} indicates whether the i-th document is relevant to the query not. MRR is defined as [Q] E rank;" > where rank, refers to the rank position of the first relevant document for the i-th query. nDCG considers the order of re- trieved documents in the list. DCG@K K is defined as K 2nd Li=1 (og2(++11) where rel, is the graded relevance of the result at position i. nDCG is the ratio of DCG to the maximum possible DCG for the query, which occurs when the retrieved documents are presented in decreasing order of relevance.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang. Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neigh- bor negative contrastive learning for dense text re- trieval. In ICLR.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense re- trieval model training with hard negatives. SIGIR, pages 1512.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang, and Ji-Rong Wen. 2022. Ultro: An ulti- mate retriever on corpus with a model-based indexer. CoRR.
A.1.3 Model architecture
GLEN is based on the transformer-based encoder- decoder architecture. The number of transformer layers is 12, the hidden size is 768, the feed-forward layer size is 3072, and the number of self-attention heads is 12 for the encoder and decoder. We imple- mented GLEN using PyTorch based on the Teva- tron library 2 (Gao et al., 2023) and adopted the gradient cache (Gao et al., 2021) to accommodate large batch size with limited hardware memory.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei,
Ming Gong, Guido Zuccon, and Daxin Jiang. 2023. Bridging the gap between indexing and retrieval for differentiable search index with query generation. In Gen-IR@SIGIR,
A
Appendix
A.1
Additional Experimental Setup
A.1.4 Baselines
A.1.1 Datasets
BM25 (Robertson and Walker, 1994) is the tradi- tional sparse retrieval model using lexical match- ing. DocTSuuery (Nogueira and Lin, 2020) ex- tends the document terms by generating relevant queries from the documents using T5 (Raffel al., 2020). DPR (Karpukhin al., 2020) is bi-encoder model trained with in batbatch negatives, which re- trieves the documents via a nearest neighbor search. ANCE (Xiong et al., 2021 ii a bi-encoder model trained with asynchronously selected hard training negatives. Sentence-T5 (Ni et al., 2022a) is similar hhttp://tevatron.a//
Natural Questions (NQ320k) (Kwiatkowski et al., 2019) is curated from Natural Questions. The dataset consists of a Wikipedia article and query in the form of a natural language question. MS MARCO passage ranking (Nguyen et al., 2016) is a dataset released by Microsoft in 2016 for reading comprehension and adapted in 2018 for retrieval. For validation of MS MARCO, 1,000 queries and use corpus consisting of ran- domly sampled 100 passages from BM25 (Robert- son and Walker, 1994) top 1000 passages for each query, ie,. |D|- x 100,000. BEIR (Thakur et al.,
randomly split

---

Datasets
NQ320k
MS MARCO
Metrics
R@1
R@ 10
MRR@ 100
MRR@ 10
w/o refinement
66.9
84.9
73.6
6.5
w/ refinement
69.1
86.0
75.4
20.1
Table Ablation study of GLEN the ranking-based ID refinement phase.
DPR but utilizes T5 (Raffel al., 2020) backbone. GTR (Ni al., 2022b) is scaled-up bi-encoder model with a fixed-size bottleneck layer based on Sentence-T5, -5, which is a state-of-the-at art dense retrieval model. For BM25, we followed the official guide" for reproducing. For NQ320k and BEIR, we refer to the results reported by Sun et all (2023) and Ren et al. (2023). For MS MARCO, refer the results from Pradeep al. (2023). The results of NCI are obtained based the publicly released checkpoint by Wang et (2022).
A.1.5 Reproducibility
The weight decay is le-4. We set the max sequence length for a query to 32, the max sequence length for document to 156, and the dropout rate to 0.1. We conducted all experiments on desktop with NVidia RTX V100, 512 GB memory, and single Intel Xeon Gold 6226.
A.2 Effect of Ranking- Bassedg-based ID Refinement
Table 8 reports the effect of the lanking-based ID refinement phase of GLEN on NQ320k and MS MARCO. We observed that the refinement phase led to a performance gain of 3.3% for NQ320K and 209.4% for MS MARCO, respectively. This un- derscores the significance of the refinement phase, which trains on both pairwise ranking loss and pointwise retrieval loss key component in dy- namic identifier learning. Also, shown that the ranking-basse based ID refinement phase is effective, es- pecially for the large-corpus set (i.e., MS MARCO). This is due to the fact that learning the mapping relations berween documents and predefined iden- tifiers becomes more challenging as the number of documents increases.
Phttp://pyserni..ia


Reviews:
[{"review_id": "VZLVmgIfBr", "reviewer": "Reviewer_CqNg", "summary": "", "questions": "- If we already need to have a two-step process for learning/assigning document identifiers and learning the relevancy, this is already very similar to any other dense retrieval model, which first trains the model and then indexes the documents. What is the advantage and motivation of using a generative retrieval model?\n- Have you looked into the version of not learning the document identifier but directly assigning them using tokens with high tfidf scores (like the supervision of the ID training step)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable comments that will definitely improve our paper. Also, thank you for acknowledging the novelty of our work. First, we would like to address three points mentioned in the ‘Paper Topic And Main Contributions’ part. \n  \n***\n### 1. Comparison and analysis between GLEN and GENRET [1]\n\nThanks to the reviewer for pointing it out. We will include the following analysis to provide\nclear motivation for our work. We have analyzed why the performance difference between GENRET and GLEN depends on the dataset, i.e., NQ320k and BEIR. The **main differences between GLEN and GENRET** can be summarized as follows.\n\n|  Model  | Identifier type | Identifier collision solution          |\n|--------|-----------------|----------------------------------------|\n| GLEN   | lexical         | Collision-free ranking using ID weight |\n| GENRET | numeric         | Random ranking                         |\n\nBased on this analysis, we hypothesize that the discrepancy is due to the following reasons: **(1) the generalization capability of lexical identifiers**, and **(2) the collision-free inference of GLEN for new documents**.\n\nFirst, lexical identifiers are likely to have better generalization capability in a zero-shot setting than numeric identifiers. The key to generalization is to assign appropriate identifiers to documents that represent out-of-domain documents well. **GLEN can assign a generalized identifier to a new document by leveraging knowledge from the PLM, whereas GENRET may struggle with it**. For instance, if a new document belongs to the medical domain and the training set lacks that domain, GLEN can assign domain-specific lexical terms as identifiers. This is because GLEN’s lexical identifier units are pre-trained T5 vocabulary tokens, while GENRET’s numeric identifier units are numbers and are only trained to represent training documents. Therefore, the performance difference between GLEN and GENRET is not significant for the trained dataset, NQ320k. In contrast, it seems significant for the untrained dataset, BEIR.\n\nSecond, the collision-free inference of GLEN would work better in BEIR, which performs retrieval for new documents. If multiple documents have the same identifier, **GLEN breaks the tie using the logit of the identifier, while GENRET ranks them in random order**. (Refer to Section 5.2 of [1]) For in-domain documents, the model can be trained to distinguish between similar documents, but for out-of-domain documents, the same identifier can be given if they are semantically similar. GLEN and GENRET differ in their collision handling, hence the performance gap seems to be relatively larger for BEIR than NQ.\n\n\n\n\n### 2. Role of ranking-based ID refinement phase (i.e., pairwise ranking loss and pointwise retrieval loss)\n\nFirst, we would like to clarify that the **ranking-based ID refinement phase is very crucial** to our main idea. The pairwise ranking loss and pointwise retrieval loss are key to dynamic identifier learning, which is our main idea.\n\nSuppose there is only a keyword-based ID assignment phase. In that case, the model will only learn to map documents or queries to predefined identifiers, which is highly similar to the learning process of other models that utilize static identifiers (e.g., NCI [2]). We believe that the ranking-based ID refinement phase is essential to learn appropriate identifiers for query-document matching. It is shown by the following **experimental results** as follows.\n\n\n| NQ320k           | Recall@1 | Recall@10  |  MRR@100 |\n|--------------|----------|------------|---------|\n| w/o refinement phase    |   66.8 |     84.3 | 73.3 |\n| w/ refinement phase   |   68.2 |     85.9 |   74.8 |\n\n| MS MARCO | MRR@10     |\n|-------------|------------|\n|  w/o refinement phase  |      4.0 |\n|  w/ refinement phase | 18.0 | \n\nAlso, it is shown that the **ranking-based ID refinement phase is effective, especially for the large-corpus set (e.g., MS MARCO)**. This is because learning the mapping relations between documents and predefined identifiers becomes more difficult as the number of documents increases.\n\nWe'd like to thank you for the suggestion. It is a very interesting idea to apply our ranking-based ID refinement to other models, such as NCI. We are in the process of implementation, and we believe it might improve the accuracy of those models as well.\n\n\n\n### 3. Extension of collision-free approach using other retrieval models\n\nWe appreciate your suggestion and have observed very interesting results. In alignment with your proposal, we have investigated the **utilization of other retrieval models to effectively deal with collisions**. We present the results obtained using BM25 model below.\n\n\n| NQ320k           | Recall@1 | Recall@10  | MRR@100 |\n|----------------|----------|------------|---------|\n| random rerank  |   67.7 |     82.4 |  74.5 |\n| BM25 rerank    |   68.9 |     86.0 |  75.3 |\n\n| MS MARCO |  MRR@10     |\n|----------------|------------|\n| random rerank  |       16.6 | \n| BM25 rerank    |      19.2 | \n\nThe experimental results demonstrate that BM25 reranking provides a 1.8% gain in NQ (R@1) and a 15.6% gain in MARCO (MRR@10) compared to random reranking.\nWe believe that this is a very simple yet effective way to improve the performance of the model, and it is expected that utilizing other dense retrieval models will further improve the effectiveness. However, unlike the proposed collision-free inference, which requires little additional computational cost or storage overhead, **utilizing an additional retrieval model has the limitations of (1) additional retrieval latency and (2) the need for an auxiliary index structure**. Therefore, we adopt an identifier weight-based reranking that eliminates the need for additional cost.\n\n***\nNext, we would like to answer the ‘Questions For The Authors’ part in the following.\n\n\n### 4. What is the advantage and motivation of using a generative retrieval model?\n\nGLEN is similar to dense retrieval in that it learns relevance signals between queries and documents through ranking loss during training. However, GLEN is only trained with additional losses, and the inference phase is identical to existing generative retrieval methods. That is, GLEN, like generative retrieval models, does not require an auxiliary index structure for inference. In this regard, the advantages of generative retrieval are as follows.\n\n**(1) Efficiency in terms of memory storage**. The memory footprint of a generative retrieval model depends solely on the model parameters. This contrasts with dense or sparse retrieval models, which require additional storage space to store documents depending on document collection size (e.g., the inverted index or ANN index).\n\n**(2) Efficiency in terms of retrieval latency**. For online retrieval latency, the generative retrieval model only requires the time to generate an identifier for a query. According to [1], for the representative models, GTR-base [3] and GENRET, the online retrieval latency is 1.97 seconds and 0.16 seconds (beam size=100), respectively, but the performance is similar at 57.6 and 58.1 based on MRR@10.\n\n**(3) Effectiveness via end-to-end optimization**. Generative retrieval can optimize the entire retrieval process in an end-to-end manner, thus enabling effective retrieval.\n\nDespite these advantages, the generative retrieval model still has a performance gap with existing state-of-the-art conventional retrieval methods. In this sense, there is still much room for further research. Furthermore, we will enhance the clarity of our paper by providing a more explicit comparison between dense retrieval and GLEN.\n\n### 5. Have you looked into the version of not learning the document identifier but directly assigning them using tokens with high TF-IDF scores (like the supervision of the ID training step)?\n\nWe appreciate your interesting suggestion. Here is the experimental result when the document identifier is not dynamically learned, but a predefined keyword-based identifier is assigned based on a high TF-IDF score. Note that for predefined identifiers, we cannot utilize the proposed collision-free inference since the logit of the identifier does not exist. So, for a fair comparison, we report the results of random reranking for both pre-defined identifiers and learned identifiers. \n\n| NQ320k   | Recall@1 | Recall@10  | MRR@100 |\n|--------------|----------|------------|---------|\n| GLEN with pre-defined identifier | 67.6 | 85.9 |  74.4 |\n| GLEN with learned identifier | 67.7 |     82.4 |   74.5 |\n\n| MS MARCO |  MRR@10     |\n|-------------|------------|\n| GLEN with pre-defined identifier |  6.8 | \n| GLEN with learned identifier |      16.6 | \n\n\nExperimental results show that utilizing learned identifiers improves performance over predefined identifiers for NQ and MS MARCO. In particular, in large-scale corpus like MS MARCO, it is more effective to assign identifiers through learning rather than predefining them. We conjecture that learning the mapping relationship between documents and predefined identifiers becomes more difficult as the number of documents increases.\n\n\n***\nHere is our response to ‘Typos Grammar Style And Presentation Improvements.’\n\n### 6. Comparison with other SOTA dense retrieval models (e.g., ColBERT)\n\nYour constructive suggestions are consistent with our views. We also believe that it is necessary to compare the performance with state-of-the-art sparse or dense retrieval models such as ColBERT in addition to GTR-base [3]. We are currently implementing ColBERT on the same code framework as GLEN. We will compare our experimental results with GLEN in the future for completeness.\n\n\n\n### 7. Clarifying descriptions and correcting citations\n\nWe appreciate your detailed comments about our paper. (1) In terms of the word 'identifier'. We would like to revise the description of predicted, generated, and generic identifiers to match their meaning. (2) For the naming rule of notation, we basically use bold for vectors and italics for scalars and text sequences. We will add additional explanations to the text to make it clearer. (3) We will modify citations to follow published versions for [4, 5, 6, 7].\n\n### References\n- [1] Weiwei Sun et al., Learning to Tokenize for Generative Retrieval. CoRR abs/2304.04171 (2023)\n- [2] Yujing Wang et al., A Neural Corpus Indexer for Document Retrieval. NeurIPS 2022\n- [3] Jianmo Ni et al., Large dual encoders are generalizable retrievers. EMNLP 2022\n- [4] Luyu Gao et al., Tevatron: An efficient and flexible toolkit for neural retrieval. SIGIR 2023\n- [5] Hyunji Lee et al., Nonparametric decoding for generative retrieval. ACL (Findings) 2023\n- [6] Ruiyang Ren et al., TOME: A two-stage approach for model-based retrieval. ACL 2023\n- [7] Shengyao Zhuang et al., Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation. Gen-IR@SIGIR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Provide a novel approach for determining the document identifier in the generative retrieval model. ", "reasons_to_reject": "Lack of detailed analysis to its dynamically-generated numeric identifier counterpart (GENRET). ", "questions_to_authors": "- If we already need to have a two-step process for learning/assigning document identifiers and learning the relevancy, this is already very similar to any other dense retrieval model, which first trains the model and then indexes the documents. What is the advantage and motivation of using a generative retrieval model?\n- Have you looked into the version of not learning the document identifier but directly assigning them using tokens with high tfidf scores (like the supervision of the ID training step)?", "paper_topic_and_main_contributions": "This paper discusses a generative retrieval approach that uses a learned, dynamic lexical identifier for the retrieval candidates. The novelty of the work comes from the learning process of the dynamic lexical identifier and how this learning process interacts with the ranking mechanism. This is certainly an interesting work that deserves a spot at the conference. However, it has some drawbacks that would definitely benefit from some revision. \n\n1. The closest prior work is probably GENRET, which learns a numeric document identifier instead of a lexical one. The authors also acknowledge the similarity but did not provide more comparisons and analyses between them. In the result section, the authors generally conclude that both the proposed GLEN and GENRET outperform models using static identifiers, which is a fine conclusion(e.g., Line 485 and 508). However, the novelty and the motivation of the work come from the leverage of using lexical identifiers to better utilize the language model. The difference is certainly small in NQ320k but much larger in BEIR. It would be much more complete if the authors provided more analysis into the reasons why we are seeing such discrepancies in the two collections. \n2. The authors added additional steps, such as pairwise and pointwise ranking losses, to enhance the ranking ability of the model. These are more independent of the central idea proposed in the paper, which makes it hard to compare with other works. Since the authors did not reimplement other models (like a lot of other papers), it is likely to be something that the authors cannot fix. However, this raises the question of what if we also apply these training techniques to other models. \n3. The collision-resolution approach is an interesting one. If I understand this correctly, this is basically falling back to a dense retrieval approach that uses the intermediate representation to break the tie (due to the identifier being the same). It would be interesting to see if we use a simple BM25 or other dense retrieval models to break the ties, which is much more practical, instead of random ranking in Table 7. \n\nIn summary, while there is room for this paper to improve, I believe these are generally fixable. The paper, as it is, has already provided a good set of evidence for the effectiveness and novelty of the work. Therefore, I vote for acceptance for this paper.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The word “identifier” is overloaded in the paper. It can mean “predicted,” “generated,” or just generic IDs throughout the paper. It would be better to specify what exactly it should refer to whenever you mention it.\n- There are quite a lot of notations in the paper; some are similar to each other but with a different symbol because of the version or usage. It would be nice to explain the naming rule if there is one so it is easier for the readers to follow.\n- It would be nice to include ColBERT in the comparison since it is generally the state-of-the-art retrieval architecture regarding effectiveness.\n- Some papers cited are already accepted in some conferences. It is a better practice to cite their published/latest version instead of the preprint one."}}, {"review_id": "jtgy2AJDqZ", "reviewer": "Reviewer_XCay", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your constructive comments. Also, thank you for considering our work as a potentially impactful, innovative, and promising methodology. We address the points you mentioned in the ‘Reasons To Reject’ part.\n\n***\n\n###  1. Does it require re-indexing documents compared to non-generative retrieval?\n\n**No**, GLEN's dynamic identifier learning does not require re-indexing documents compared to non-generative retrieval. The dynamic learning is an essential point that distinguishes GLEN from other generative models, but the **inference process is still in the nature of generative retrieval**. In other words, instead of building an inverted index or storing dense vectors in non-generative retrieval, GLEN only requires the document identifier for each document to perform a search. \n\n### 2. Side effects of dynamic hard negative\n\nCompared to static hard negative sampling (e.g., BM25 hard negatives), **our dynamic hard negative sampling can be more costly in terms of time efficiency**. For each document, static hard negatives can be sampled from a predefined negative document set beforehand. Therefore, sampling during training time takes very little time. On the other hand, our dynamic hard negatives require additional time to sample documents with the same prefix as the positive document. (In our experiments, this was negligible.) We appreciate your feedback, and we will revise the paper to clarify this point.\n\n### 3. Implementation of indexing model and retrieval model\n\n**The two models have the same architecture**, and as mentioned on line 232, they also **share parameters**.  We will modify this to enhance clarity and avoid any confusion.\n\n### 4. Significance test for Table 6 (ablation study)\n\nWe thank the reviewer for pointing this out. We conducted Bonferroni correction pairwise t-tests following [2,3]. Statistical significant differences (p<0.01) with Bonferroni correction between the GLEN and variants are denoted as ∗. **We will add the results** in the paper.\n\n|             Model            |       Recall@1       | Recall@10 | MRR@100 |\n|----------------------------|---------------|----|-------|\n|             GLEN             |       68.3      | 85.7 |   74.9  |\n|       w/o keyword-based      |       46.9*     | 76.5*|   57.5 *| \n|         w/o annealing        |       67.5      | 85.2 |   74.1  | \n|  w/o adaptive decoder input  |       60.4*     | 83.7 |   69.1 *| \n|   w/o pairwise ranking loss  |       67.3      | 83.8 |   73.4  | \n| w/o pointwise retrieval loss |       67.0      | 85.8 |   74.1  | \n|          random neg          |       67.9      | 85.1 |   74.3  |\n\n\n\n### References\n- [1] Vladimir Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020\n- [2] Norbert Fuhr, Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. SIGIR Forum 51(3): 32-41 (2017)\n- [3] Leonid Boytsov et al., Deciding on an adjustment for multiplicity in IR experiments. SIGIR 2013", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is simple but potentially impactful. The methodology is innovative as it is the first to explore a new dimension, i.e., Lexical dynamic generative retrieval. The methodology is also well depicted and structured, with clear formula and math notations to assist understanding.\n\nThe obtained results look very promising, especially in zero-shot / cross-domain effectiveness (according to Table 4). As generative retrieval is a newly explored retrieval model, I look forward to more development in this direction.\n\nImportantly, the authors claim they are willing to share source code upon acceptance, this would be convenient for other researchers to replicate and better inspect this work.", "reasons_to_reject": "* Since the dynamic learning is a critical dimension to differentiate this model from others (in terms of novelty), it should be emphasized the side effects of adapting dynamic hard negatives. Does it require re-indexing documents compared to non-generative retrieval?\n\n* Although being similar to the indexing model, the implantation of retrieval model is missing. Is it initialized with the same weight as the indexing model? How about using only the indexing model? (e.g., using the indexing model of the last training time step as retrieval model)\n\n* Table 6 is informative, but it would be great to perform significance test for reader to better gauge the importance of each component. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes GLEN, a generative retrieval model based on lexical document identifier. The proposed model is trained with the objectives to both align language model vocabulary and semantic representations. To deal with lexical document ID conflicts, the model uses a retrieval model to select best candidate using weight matches.\n\nThe main contribution of this work, in my opinion, is showing generative retrieval model is able to achieve top effectiveness (especially in zero-shot and cross-domain retrieval) while taking a different approach from the previous best generative retrieval models using numeric identifiers.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TLOO5fcBWq", "reviewer": "Reviewer_L2RC", "summary": "", "questions": "Please refer to reasons to reject.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking valuable time to review our paper. We deeply appreciate the comments on the novelty of the proposed method and its strengths in comparison to related works.  We address the three concerns mentioned in the 'Reasons To Reject' part.\n\n### 1. The expressiveness of the identifier\n\nWith the example identifier '*Olympic game list*,' it may seem insufficient to capture the meaning of the entire document. Nevertheless, during the ranking-based ID refinement phase, GLEN is trained to **generate the most appropriate identifier for query-document matching**. Therefore, we conjecture that GLEN encodes the document entirely and generates suitable identifiers that capture the core meaning of the document.\n\nIn addition, to improve identifier expressiveness, **we can increase identifier length**. In our experiments, we observed a **25.0% performance gain** in MRR@10 for MS MARCO passage when we increased the identifier length from 3 to 10, as shown below.\n\n\n\n|           Model         |     MRR@10    |\n|-----------------------|-------------|\n|           BM25          |      18.4     |\n|        DocT5Query       |      27.2     |\n|         GTR-base        |      34.8     |\n|            DSI          |       3.1     |\n|          DSI-QG         |      11.8     |\n|            NCI          |      17.4     |\n|      GLEN (length=3)    |      18.0     |  \n|     GLEN (length=10)    |      22.5     |  \n\n\n**GLEN (length=10) outperforms BM25** and the generative retrieval baselines. However, as the identifier length increases, decoding time also increases, which can lead to a trade-off between expressiveness and efficiency. (For MS MARCO, GLEN (length=10) takes approximately 2.27 times longer than GLEN (length=3) for retrieval.) Therefore, choosing the appropriate identifier length is nontrivial to balancing expressiveness and efficiency. We have chosen a length of 3.\n\n\n\n### 2. Effectiveness of GLEN in NQ320k (Table 2)\n\nFrom Tables 2 and 4, we can see the performance difference between GENRET and GLEN **depending on the dataset**, i.e., NQ320k and BEIR. That is, the performance difference between GLEN and GENRET is not significant for the trained dataset, NQ320k, but it seems to be significant for the out-of-domain dataset, BEIR. Based on these results, we conjecture that GLEN performs better in more general situations, although both models showed similar accuracy on the NQ320k with a small corpus size.\n\nWe hypothesize that the performance differences across datasets come from the following reasons. First is the **generalization capability of lexical identifiers**. GLEN can assign a generalized identifier to documents by leveraging knowledge from the PLM, whereas GENRET may struggle with it. This is crucial when processing new documents or when the corpus size is large. Second is the **collision-free inference** of GLEN. If multiple documents have the same identifier, GLEN breaks the tie using the logit of the identifier, while GENRET ranks them in random order. This collision issue can become more prevalent with out-of-domain corpus sets or larger corpus sizes and must be handled.\n\nConsequently, we believe that the effectiveness of GLEN is verified, especially in the zero-shot setting.\n\n### 3. Comparison of GLEN and GENRET for MS MARCO (Table 3) and BEIR (Table 4)\n\n#### (1) MS MARCO passage dataset\n\n**We strongly agree that an experimental comparison between GLEN and GENRET** on the MS MARCO passage is needed, and we are working on it.\n\nSince the results of GENRET on MS MARCO passage full corpus set are not publicly available, **we have been trying to reproduce GENRET for a long time**. First, we requested the code for the authors and received a response that they would publish it on Github later. However, as of now, there is no code yet. (https://github.com/sunnweiwei/GenRet)\n\nTherefore, we have tried to implement GENRET but failed to replicate the experimental results of the paper. Specifically, we observed that our GENRET implementation generates the same IDs for all documents after training. When we contacted the author about this issue, the author told us that the distribution of docid values would be highly uneven without codebook initialization and docid re-assignment technique, both of which were introduced in the GENRET paper for docid diversity. We are trying to solve the issue by focusing on the two techniques. **After reproducing GENRET, we will compare the results in detail with GLEN**. We sincerely understand your concern about the lack of comparison to GENRET in Table 3 and will do our best to address it.\n\n#### (2) BEIR dataset\nTable 4 shows the comparison with GENRET on the BEIR dataset, and the experimental results show that **GLEN has a higher generalization capability than GENRET**. For further details, please refer to Table 4. We will also compare performance on other retrieval tasks in BEIR after reproducing GENRET.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The new ID generation method seems novel and provides new insights for future research.\n2. The writing is good and easy to follow and the experiments are extensive\n3. The collision-free inference can well handle documents with conflicting identifiers without additional cost.", "reasons_to_reject": "1. The new identifier generation is interpretable, however, the expressiveness of the identifier is doubtful, for example, 'olympic game list' can only represent part of the information hidden in document.\n2. The experimental result in Table 2 cannot verify the effectiveness of this method considering the strong baseline GENRET.\n3. Also, the experiment result in Tables 3 and 4 does not include the SoTA baseline like GENRET, which may not be fair.", "questions_to_authors": "Please refer to reasons to reject.", "paper_topic_and_main_contributions": "This paper proposes a new generative retrieval method that utilizes tokens as doc id representation instead of numbers or characters. The new id generation method bridges the gap between the knowledge of PLM and the identifier. Experimental result demonstrates its effectiveness.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses a generative retrieval approach that uses a learned, dynamic lexical identifier for the retrieval candidates. The proposed model, named Generative retrieval via LExical iNdex learning (GLENT), exploits a dynamic lexical identifier using a two-phase index learning strategy. The work is novel and interesting. It had some drawbacks, that were clearly identified by the reviewers and that were well addressed in the rebuttal.
Decision: Accept-Main