Auto Search Indexer for End-to-End Document Retrieval
Authors: Tianchi Yang
Minghui Song
Zihan Zhang
Haizhen Huang
Weiwei Deng
Feng Sun
Qi Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Document Retrieval
Generative Retrieval
End to End
TL;DR: We propose a fully end-to-end document retrieval pipeline, Auto Search Indexer (ASI), which supports both end-to-end docid assigning and end-to-end document retrieval within a joint framework.

Abstract: Generative retrieval, which is a new advanced paradigm for document retrieval, has recently attracted research interests, since it encodes all documents into the model and directly generates the retrieved documents. However, its power is still underutilized since it heavily relies on the ``preprocessed'' document identifiers (docids), thus limiting its retrieval performance and ability to retrieve new documents. In this paper, we propose a novel fully end-to-end retrieval paradigm. It can not only end-to-end learn the best docids for existing and new documents automatically via a semantic indexing module, but also perform end-to-end document retrieval via an encoder-decoder-based generative model, namely Auto Search Indexer (ASI). Besides, we design a reparameterization mechanism to combine the above two modules into a joint optimization framework. Extensive experimental results demonstrate the superiority of our model over advanced baselines on both public and industrial datasets and also verify the ability to deal with new documents.

Auto Search Indexer for End-to-End Document Retrieval
Tianchi Yang', Minghui Song', Zihan Zhang*,, Haizhen Huang,
Weiwei Deng, Feng Sun, Qi Zhang
STCA, Microsoft Corporation
baanchhiyagg.innguuisonyy zihzaa, hhaan, dedeng, sunfeng, z qi halljmmcoooft.cmm
hhuang,
Abstract
as assigning unique integers for documents (Tay et al., 2022; Zhou et al., 2022). Given the pre- processed docids, a Transformer-based model is employed to encode the document-docia mapping information into its parameters, and meanwhile is trained to generate relevant docids directly from a given query. As such, by adding preprocessing phase, turns the whole index-retrieve process into generation task.
Generative retrieval, which is new advanced paradigm for document retrieval, has recently attracted research interests, since it encodes all documents into the model and directly gen- erates the retrieved documents. However, its power is still underutilized since it heavily re- lies on the "preprocessed" document identifiers (docids), thus limiting retrieval performance and ability to retrieve new documents. In this paper, we propose a novel fully end-to-end re- trieval paradigm. It can only end-to-end learn the best docids for existing and new doe- uments automatically via semantic indexing module, but also perform end-to-end document retrieval via an encoder-decooder-basee genera- tive model, namely Auto Search Indexer (ASI). Besides, we design reparameterization mech- anism to combine the above two modules into a joint optimization framework. Extensive ex- perimental results demonstrate the superiority of our model over advanced baselines on both public and industrial datasets and also verify the ability to deal with new documents.
Despite the great success of these methods, the power of generative retrieval is still underutilized since they rely on the pre-processed docids. thus leading the following limitations. (1) New docu- ments cannot be seamlessly retrieved by an existing trained indexer: On the one hand, docids are pre- processed so that new documents cannot obtain their docid assignments directly from the retrieval model. On the other hand, even if their docids are obtained by the same "pre- prre-processing" policy, these new docids are usually unknown semanties to the retrieval model. (2) Existing preprocessing policies are confined to one-to-one mapping be- tween documents and docids. Accordingly, only one single document can be retrieved for each re- trieval calculation. It deviates from the intention of the retrieving- sarieving-ranking framework, i.e., groups of relevant documents are expected to be efficiently re- trieved in the retrieving stage (Guo et al., 2022). We argue to assign similar documents with same docid, which supports retrieving more documents the same computational cost. (3) The preprocessing phase is independent of the index-retrieve process. Consequently, the caused semantic gap between the docids in preprocessing phase and the embedding space in index-retrieve process limits the perfor- mance of generative retrieval. However, it is not trivial automatically learn the best docids within joint framework. since the docids, which is served as the generation ground-truths, cannot maintain the gradient flow because they must appear in dis- crete form by an argmax function. Therefore, the
1 Introduction
Search engines are widely deployed on web appli- cations to meet users' daily information require- ments (Wang et al., 2022). Given a user query, search engines usually first retrieve candidate doc- uments from huge document collection and then rank them to return a ranking list. Consequently, the performance and efficiency of document re- trieval essential to the final search quality.
Recently, new end-to-end document retrieval framework named Generative Retrieval is proposed to develop differentiable indexer, which directly maps given query to the relevant document identi- fiers (docids) via a seq2seq model (Tay et al., 2022). Specifically, some policies are first applied to pre- process all the existing documents for docids such
"oorreppondigg author.
The authors contribute cqually.

---

docid learning process and the index-retrieve pro- cess are still independent of each other even if they are integrated together.
2 Related Work
Studies about document retrieval can be roughly di- vided into three categories: sparse retrieval, dense retrieval and generative retrieval, which are briefly introduced as follows.
In this paper, propose novel fully end-to- end generative retrieval paradigm, Auto Search In- dexer (ASI). It combines both the end-to-end learn- ing of docids for existing and new documents and the end-to-end document retrieval into generative model based joint optimization framework. Specif- ically, we model document retrieval problem as seq2seq task, i.e., with user queries as input, it out- puts the docids of retrieved documents. Then, we design a semantic indexing module, which learns to automatically assign docids existing / new docu- ments. Besides, we design two semantic-oriented losses for it, which makes semantically similar doc- uments share the same docids and assigns different docids to dissimilar documents. As such, the new document will be assigned an existing docid based on its content, or new docid but belonging to the same semantic space other docids. Further- more, a reparameterization mechanism is proposed enable gradient to flow backward through the semantic indexing module, thus supporting joint training for all modules. Extensive experiments on public and industrial datasets show that the pro- posed ASI outperforms the statc-of-the-art (SOTA) baselines by significant margin in document re- trieval, and demonstrate that our semantic indexing module automatically learns meaningful docids for documents.
2.1 Sparse Retrieval
Early studies are mostly based on inverted index and retrieve documents with term matching metries, e.g., TF-IDF [45]. BM25 (Robertson and Zaragoza, 2009) measures term weights and computes rel- evance scores based on TF-IDF signal. Recent studies design to leverage the word embeddings to help build inverted index (Zheng and Callan, 2015; Dehghani et al., 2017: Dai and Callan, 2020b,a), To alleviate the mismatch problem between query and document words, which is the key weak point for sparse retrieval, researchers attempt augment possible terms before building the inverted index, c.g., Doc2Query (Nogueira et al., 2019b).
2.2 Dense Retrieval
In another line to relieve the mismatch problem. solutions based deep learning first embed the queries and documents to dense vectors and then retrieve documents per vector similarity (Lu et al., 2020; Ma et al., 2022; Ni et al., 20226; Zhan et al.. 2022; Li et al., 2023a; Wang ct al., 2023). These methods especially benefit from recent advances in pretrained language models (PLMs). For instance, SimCSE (Gao et al., 2021) is simple but effec- tive contrastive learning framework that employs BERT (Devlin al., 2019) or RoBERTa (Liu al., 2019). To improve the inference latency, dense re- trieval methods are usually equipped with approxi- mate nearest neighbor (ANN) (Subramanya et al.. 2019) or maximum inner product search (MIPS) algorithms (Shrivastava and Li, 2014) to retrieve relevant documents within a sub-linear time cost.
The contributions are summarized as follows:
To the best of our knowledge, we are the first to propose fully end-to-end pipeline, Auto Search Indexer (AST), which supports both end-to-end docid assigning and end-to-end document retrieval within joint framework.
To this end, we propose a semantic index- ing module as well as two novel semantic- oriented losses to automatically assign docu- ments with docids, and develop reparame- terization mechanism make the individual modules optimize jointly.
2.3 Generative Retrieval
Recently, an alternative architecture proposed to end-to-end map user queries to relevant docids with Transformer-basee autoregressive model. Specif- ically, Tay et al. (2022) and Wang et al. (2022) propose to preprocess the documents into atomic identifiers or hierarchical semantic identifiers with hierarchical k-means algorithm. Differently, SEAL (Bevilacqua et al., 2022) devise to leverage all n- grams in passage as its identifiers. Chen et al. (2022) similarly retrieves evidence by returning
Extensive experiments demonstrate that our ASI can learn the best docid for documents, and meanwhile achieves the best document retrieval compared to the SOTA methods on both public dataset and real industrial dataset.

---

Training Phase
DocID Prob
f
automatically assign docids to existing / new doc- uments. To encode semantics into the docids, we creatively design a discrete contrastive loss and a sequence-oriented reverse cross-entropy loss for the semantic indexing module, which helps to as- sign semantically meaningful docids and break the limitation of one-to-one mapping between docu- ments and docids. Moreover, reparameterization mechanism proposed to enable gradient flowing through the indexing module to support joint opti- mization, thus saving the decoder from falling into meaninglessly mimicking the indexing module. other words, the decoder thus gains the ability to surpass the indexing module on document retrieval.
Sequence Loss
Query
Encoder
Decoder
frt Reverse Loss
Doc
(CLS) Emb
Doe Emb
DeelD
L
Indexing
Contrulive
Query Lmb
Query11>
Inference Phase
DoelD, Doell,
Heam Search
Lookup
Encoder
Decoder
Query
Figure I: Illustration of the proposed model ASI.
sentence identifiers, namely GERE. Ultron (Zhou et al., 2022) designs both keyword-based identifiers and semantic- identifiers and develops a three- stage training workflow. SE-DSI (Tang et al., 2023) proposes to use summarization texts as document identifiers. MINDER (Li et al., 2023b) assigns doc- uments multiview identifiers. Concurrently with work, Sun et al. (2023) also investigated dif- ferent framework, GenRet, which uses codebook and discrete auto-encoder with progressive train- ing to learn the doc-id assignments within retrieval stage, while it relies on a clustering- based initial- !
3.2 Basic Architecture
Noticing the outstanding advances of generative PLMs, ASI adopts a seq-to-seq framework. Specifically, ASI forms "query-toddoiid" paradigm, i.e., ASI takes the user query input and generates several relevant docids, which are represented
sequence of id tokens. To this end, with the help of transformer- based encoder- decoder architecture, the query a is encoded by its encoder, and the generation probability is estimated by its decoder as follows,
In word, they suffer from the docid pre- processing phase, forming fake end-to-end frame- work. In this paper, we propose fully end-to-end paradigm ASL. It not only supports end-to-end doc- ument retrieval by generative model, but also end-to-end Iearns the best docids for documents within a joint optimization framework.
h, Deccoder het),
(1)
P(id; q, q,ides; Qea) Softmax(h, (2)
where id, denotes the i-th token in the currently given docid id of length m, which is obtained by the semantic indexing module. θ denotes the trainable parameters in encoder-decoder architec- ture. W E Vd the linear parameter to clas- sify the hidden state h; into the docid vocabulary of size Via. Here we treat all of the docids as different tokens from the encoder vocabulary, and therefore the encoder and decoder do not share the vocabu- lary space to improve decoding efficiency.
3
Our Proposed Method
3.1 Overview
In this subsection, present overview of our novel Auto Search Indexer. The basic idea is, as illustrated in Figure I, to build a fully end-to-end pipeline to automatically learn the meaningful do- cids for documents, perform end-to-end document retrieval, and combine them into a joint framework.
Finally, to maximize the target docid sequence likelihood, we adopt cross-entropy loss to optimize the following generation objective function,
In detail, our ASI adopts encoder-decoder ar- chitecture encode the user query q and directly generate relevant docids id( i 1, 2.... Distin- meth-
(e(De)) Σ (q.d)ED
Σ log iid
guished from existing preprocessing-baxee ods, semantic indexing module is integrated to
(3)
'Compared with ASI, GenRet additionally relies on a pro- gressive training scheme and unique docid assignment, lead- ing to limited training and inference efficiency for large-scale corpora.
where the target docid id is obtained based on doc- ument d by the semantic indexing module, which will be described next.

---

c Controstive Loss
module to softly encourage the assignment of dif- ferent docids different documents, rather than utilizing manual rules force it.
id
Query
Linear
Argmax
id
Discrete Contrastive Loss First of all, the premise to assign similar documents with same docid is that the encoder should learn semantic- based representations for documents. Accordingly, propose discrete contrastive loss to help leam different embeddings for documents of different semantics, where the "different" is measured by query-document pairs.
by Reverse CE Loss
td"
Doc d
Figure 2: Illustration of the proposed semantic indexing module and two semantic-oriented losses.
3.3 Semantic Indexing Module
Formally, given set of training examples D = {g, d)} composed of query-document pairs, the discrete contrastive objective function is as follows,
Previous works focus on the end-to-end retrieval phase, while neglecting to build an end-to-end learning framework for docid indexing, i.e., they have to "preprocess" the documents for docids (Tay et al., 2022; Wang et al., 2022; Zhou et al., 2022). Therefore, they can hardly deal with new docu- ments, which are common and unavoidable in prac- tical applications. To tackle the above problem, in this subsection, we propose semantic indexing module to automatically assign docids existing new documents, as illustrated in Figure 2.
C-= > max ridd ) iid )aa,,
(6)
(q.))D (g.d (D
where the docid id' and pscudo query id id are cal- culated by (5), a is hyperparameter of margin. Note that it is hard to measure the distance between two discrete ids and calculate the corresponding gradient, thus we use the probability distribution of ids calculate distances. Formally,
Specifically, given the input document d E D and its representation æ from encoder2. the seman- tic indexing module assigns its corresponding i-th id token based on the following probability distri- bution,
rid" idd)
(7)
Σ, [P(id])g; Oci) P(idy/d: Oc.)ll
P(id, d; Ozj)) Linear;(x)) id, arg maxP(idd;ld Oci),
(4)
(5)
where the distribution P(-) refers to Eq.
where Oe denotes the model parameter in encoder and semantic indexing module. Note that this se- mantic indexing module could have been more elab- orately designed, while this paper focuses on the "fully" end-to-end framework, and hence this mod- ule is designed from simple point of view.
Sequence-Oriented Reverse Cross-Entropy Loss Intuitively, considering minimizing cross- entropy loss is usually used to make a variable closer to given label, we can conversely maxi- mize it to make id token away from the given label (Pang al., 2018). Considering that docid is formed sequence of id tokens, it is not nec- essary guarantee that every token of the two sequences different, but only one the tokens is different. Therefore, given a pair docids, we can find the maximum value from the cross-entropy of all id token pairs, which means this pair is most likely to become different. Then, our sequence- oriented reverse cross-entrope loss is proposed to maximize this maximum value.
3.4 Semantic-Orienter Losses
Existing indexing policies commonly enforce that each docid uniquely refers one document, which reduces the retrieval efficiency. We argue to assign similar documents with same docid, thus support- ing retrieve more documents the same compu- tational cost.
To this end, as depicted in Figure 2, we propose a discrete contrastive loss and a sequence-oriented reverse cross-entropy loss for semantic indexing We represent by its [CLS] embedding.
Formally, given two documents d,,, dg E DD let id and ådt denote their docids, respectively. We can diversify their docids by minimizing the fol-

---

lowing loss function,
Table Statistics Datasets
(.(Oei) = = -- de d
c(i (Oci), (8)
Datasets
ADS
MSMARCO
where C(1A)) ((ei)) = max CrossEntropy (P(id) dy; (ei,
Train
50M
367K
Expansion
32M
Valid
10K
5.2K
# Does
13.1M
3.2M
For mini-batch training, the document pairs are selected from batch.
where detach() makes a tensor detached from the backpropagation. As such, STE utilizes the gradi- ent of P(id, d) to replace the gradient of argmax. Finally, (3) is rewritten as follows,
3.5 Reparameterization Mechanism
The above semantic indexing module is expected to learn the best docids jointly with Leeg. How- ever, the obtained docids are utilized generation ground-truths, which cannot maintain the gradi- ent flow with the help softmax but must appear in discrete one-hot form by a gradientless argmax function. Consequently, the gradient is not able to propagate back from decoder to the semantic indexing module, thus the optimization of decoder and semantic indexing module is still decoupled. It means that directly regarding the docids of one-hot format the training target probably makes de- coder fall into meaninglessly mimicking rather than surpassing the indexing module, as 89; Inspired by DALL-E (Ramesh et al., 2021), we devise simple but effective reparamcterization mechanism to support our end-to-end learning framework via Straight-Throush Estimator (STE) (Hinton, 2012). Specifically, suppose the semantic indexing module outputs the i-th docid éd; for document d, we can derive the formula according to the chain rule as follows,
(o.di)))= (q.))DD
Σ Σ log (idd id Oeds))
(13)
where Oeai denotes the trainable parameters encoder, decoder and indexing modules.
3.6 Model Training & Inference
For model training, we merge the above objective functions together as follows,
0amy 03_
f= Jeho Jphr;
(14)
where "/ and "/r are the scaling coefficients. In the inference (retrieval) phase, when user query is inputted retrieve documents, we apply the encoder to encode the query, then adopt beam search on the decoder to generate relevant docid, and finally look up the document-docid mapping to output the retrieved documents.
@Lwey @Leey Oid; 00; Oid; (iddd. a0
OP(id
(10)
As for when new document appears and should be incorporated into the existing document collec- tion, we apply the encoder followed by the seman- tic indexing module to assign a docid to it. is worth noting that this docid belongs to the same semantic space others. As result, even if it has never appeared in the training document collection. there is no need retrain the model for this docid.
where id, denote the one-hot vector of id,, and aid; the mid-term [dl is non-differential. STE
suggests defining the non-diffcrential term as "1",
thus we have
@Leyy
0Lseg
OP (d)
Lar max P(idj(d)
00:
Oid;
00
(11)
Experiments
To this end, the outputted one-hot vector is repa- rameterized in forward propagation as follows,
4.1 Experimental Setup
id, = id, detach( P(id, Oci))+ P(id, d: Oc:),
4.1.1 Datasets
(12)
We evaluate the empirical performance on pub- lic dataset and an industrial dataset for document retrieval. The statistics the data are reported Table and brief introduction is as follows.
'Note that the gradient of the two semantic-oriente losses also rely on this reparameterization mechanism. Here we only take the sequence loss as an example to illustrate the reparamcterization mechanism.

---

MS MARCO Document Ranking Task (Nguyen al., 2016) is a large-scale dataset for machine reading comprehension, which con- tains a total of 3.2 million candidate documents. We use the official split of the dataset. Besides, for fair comparison, following Zhou et al. (2022), we apply Douury (Nogueira et al., 2019a) for query generation.
these two variants as Ultron-URL and Ultron-QQ. respectively. NCI invents a tailored prefix-aware weight-adaptive decoder architecture, better suited to its hierarchical clustering- search-based generator. GenRer (Sun et al., 2023) uses codebook and discrete auto-encoder with progressive training to learn the doc-id assignments within retrieval stage.
docids and beam
ADS It is a real-world large-scale dataset col- lected from Bing² sponsored search engine, which provides organic web results in response user queries and then supplements with sponsored ads. We collect query-ad pairs where the ads are the concatenation of the title and abstract from the sponsored ads corresponding to the user query.
4.1.3 Metrics
We evaluate model performance with the follow- ing common metrics for document retrieval. Re- call K (R@ 1/5/10) treats the cases as true posi- tives that the decoder generates the same docids as the assignments of the semantic indexing mod- ule. Moreover, for dataset ADS, Quality Score between the query and the retrieved documents is measured by an online quality estimation tool in Bing. Considering that ASI allows each docid to point to multiple documents, we report the micro-/ macro-averaged Quality Score, denoted as Mi-QS and Ma-QS respectively. Besides, we also report the average number of retrieved documents for each query when generating Top10 docids, denoted DIQ.
4.1.2 Baselines
To validate the effectiveness of our proposed ASI, we compare it with the following three groups of strong document retrieval baselines.
Sparse Retrieval BM25 (Robertson and Zaragoza, 2009) is a difficult-to-beat baseline which uses TF-IDF feature to measure term weights. DocT5Query (Nogueira et al., 2019a) utilizes T5 to generate pscudo query for document to expand document information and then applies BM25 for document retrieval.
4.1.4 Detailed Implementation
In terms of model architecture, we build ASI with 6-layer encoder and 6-layer decoder, where the en- coder is initialized with pretrained 6-layer BERT and the decoder is optimized from scratch since its vocabulary changed to id tokens. For encoder- only or decoder-only baselines, i.e., RepBERT, Sentence-TS. DPR, we set the layer number 12 for fair comparison. For encoder-decoder models, we set encoder/decoder layer number 6, i.c.. 12 layers in total. For other settings, we set the max length of input sequence 64 for ours and follow the settings for baselines in Zhou et al. (2022). We set the margin a=3, loss coefficients =..,, the length of docid m=4 and the range of each id token is [0, 256). For model training, we set batch size 4096 and lcarning rate lc-4 with AdamW opti- mizer. For inference, we apply vanilla beam search without constraints? and the beam size as 10.
Dense Retrieval We select four representative methods for comparison, i.e., RepBERT (Zhan al., 2022), Sentence-TS (Ni et al., 20226), DPR (Karpukhin et al., 2020), SimCSE (Gao al, 2021) and GTR (Ni et al., 2022a).
Generative Retrieval DSI (Tay et al., 2022) is the first generative retrieval framework to directly output docids with the query as input. We com- pare ASI with two DSI variants, which construct docids with random unique integers and hierarchi- cal clusters, namely DSI-Atomic and DSS-Semantic, respectively. DSI-QG (Zhuang et al., 2022) bridges the gap between indexing and retrieval for differ- entiable search index with a query generation tech- nique. SEAL (Bevilacqua al., 2022) regards all n-grams contained in documents as their identi- fiers. Ultron (Zhou et al., 2022) designs a three- stage training workflow where the docids are built by reversed URL or product quantization (Zhan et al., 2021) on document embeddings. We denote
"gaunernst/bert-Le ---7t----------ccaee
"Note that there is no need to adopt methods such as con- straint beam search to enforce that the generated docid must have corresponding documents. On the one hand, as stud- ied in Section 4.4, new documents might occupy new docids. Consequently, the "invalid" docids have certain guiding sig- nificance for us to expand the coverage of documents. On the other hand, we have also counted the proportion of generated invalid docids, which is only about 0.58%.
"htess//microooff.gite htt.://moo..........oo/oooonnnae io/msmarce/DDttussss hhaps://wwwwbing.com

---

Table 2: Performance on datasct MS MARCO. The best results are shown in bold and the third best underlined. "f" denotes that the performance is referred from Zhou et al. (2022). Sun et al. (2023) or Tang et al. (2023) and "8" denotes we reproduce by their official implementations.
Table 3: Performance on dataset ADS.
Mi-QS
Model
R@D
R@5
R@ 10
DIQ
Ma-QS
Documents in Training & Validation Set (r/13M)
SEAL
0.0444 0.1463 0.1998 0.5291
Model
Params
R@1
R
R@:10
MRR@ 10
SimCSE
0.0934 0.2853 0.3891
0.3122
10
BM25F
0.1894
0,4282
0,5507
0,2924
SimCSEooo
0.2768 0.4593 0.5008
0.5290
443
DocTSQueryi
0.2327
0.4938
0.6361
0,3481
0,5087
RepBERTi
220M
0.2525
0,5841
0.6918
0,3848
0.5053
ASI
0.3952 0.6542 0.7259
1344
Sentence- T5j
220M
0.2727
0,5891
0.72115
0,4069
0.4857
DPR
220M
0.2908
0,6275
0,7313
0,4341
SimCSE:
110M
0.2867
0.6470
0.7322
0,4390
Full Documents Collection (~~69MM)
GTR- BTR--Buse1
110M
0.4620
0.7930
0.5760
0.4806
AST
142258
DSI-Semanticf
250M
0.2574
0.4358 0.5384
0,3392
0.4661
DST-Atomici
495M
0.3247
0.6301 0.69992
0.44229
DSI-QG
200M
0,2574
0.4358 0.5384
0,3392
NCT:
376M
0.2574
0.4358 0.5384
0.3392
SEAL:
139M
0.2884
0.5683 0.6829
0,4066
fluence of the one-to-many mapping of docid is excluded, our ASI is still significantly better than all the strong baselines. This further demonstrates
DynamicRetrievert
495M
0.2904
0.6422 0.7315
0,4253
Ulrro-URLi
248M
0.2957
0,5643 0.6782
0,4002
Ultron-PQt
257M
0.3155
0,6398
0.7314
0,4535
Ultron-Attmict
495M
0.3281
0.6490
0.7413
0.4686
the superiority of our design.
GenRet |
215M
0.4790
0.7980
0,5810
ASI
125M
0.6121
0,7831
0.8207
0.6857
4.3
Retrieval Performance on ADS
ASI (Expectation)
125M
0,5497
0,7072
0.7414
0.6175
Considering the sparsity the supervision signal in datasets, the document that was not interacted with the given query according to the dataset is not definitely an "inappropriate" retrieval result. Therefore, in this subsection, evaluate ASI on ADS dataset and focus on the retrieval quality.
4.2
Retrieval Performance on MS MARCO
We evaluate the model performance for document retrieval on dataset MS MARCO in this subsection, which is reported in Table 2. The major findings from the results are summarized as follows:
Settings. For datasets, we first conduct ex- periment on the collected dataset, i.e., we train the model on training set and perform document retrieval on the about 13M documents from train- ing & validation set. Furthermore, we also evaluate this checkpoint based on the full collection of about 689M documents in Bing platform. For baselines, select two representative baselines, i.e., Sim- CSE and SEALL. Furthermore, we modify Sim- CSE to support docid retrieval, where we use the average of document embeddings corresponding to every docid as the docid embedding, based on the document-docid mapping relationship learned by our ASI, namely SimCSEascid
(1) ASI significantly outperforms all the compet- itive baselines by a significant margin across four different metries on the dataset MS MARCO. Espe- cially on the R@1 I metric, our ASI almost achieves twice the performance compared with the strongest baseline, i.e., Ultron-Atomie, which validates the superiority of our fully end-to-end pipeline. We attribute this surprising gain to both its tailored design and more suitable docids learned for gener- ative retrieval.
(2) Furthermore, considering that ASI allows each docid point multiple documents, it is somehow unfair for baselines to directly compare on Recall. Therefore, we modify the metrics for fur- ther comparison: For a retrieved docid that points to multiple documents, we randomly sample one single document for it. As such, ASI would also retrieve the same number of documents basc- lines. Besides, in order further eliminate the occasional performance fluctuation caused by the random sampling strategy, we choose to report the expectation of Recall/MRR in Table (refer to ASI (Expectation)). It can be seen that even if the in-
For metrics Recall @ K, as reported in Table 3, ASI outperforms the selected baselines by greatly large margin. It does not rule out that it is because the one-to-many docids reduce the retrieval dif- ficulty, thus we add a comparison between ASI
8 "Among generative retrieval methods, SEAL relies on ngram-based docids so that new documents can be casily incor- porated by simply rebuilding the FM index without retraining the model. In contrast, the others rely on virtual token-bused docids that are generated during preprocessing phase, which causes difficulty in generalizing to new documents.

---

Table 4: Performance on new documents from ADS.
Specifically, there are two possible results when dealing with new texts. One is assign new doc- uments with an existing docid, which means they are new in content rather than semantics. The other is that the new documents are assigned with a new docid, which means they are new in semantics, Hence, we split the validation set into three sub- sets, including "Existing" denoting the documents that are contained in training set, "NewContent" denoting the first category of new documents and "NewSemantic" denoting the second category.
Full Valid
New Content
New Semantic
Metrics
Existing
Sample
10000
1661
8146
193
R@1 1
0.3952
0.4218
0.4088
0.0570
R R@5
0.6542
0.6865
0.6629
0.2073
R@ 10
0.7259
0.7583
0.7333
0.2487
Mi-QS
0,5053
0,5174
0,5041
0,4640
Ma-QS
0.4857
0.5006
0,4872
0.4638
DIQ
1344
1565
1178
925
and SimCSExocid- As shown in the table, there is still a remarkable improvement compared ASI with SimCSEdocid. which validates the superior perfor- mance of our ASI for docid retrieval.
As shown in Table 4, compared with the per- formance on the full validation set of ADS, ASI performs better on group "Existing". Surprisingly. ASI also performs better in "NewContent".. For the "NewSemantic" group, our ASI still achieves desirable performance. These results validate the semantic indexing module does not mechanically copy the docids contained in the training set, but fully learns the relationship between document manties and docids. That's why our model has such satisfying ability to handle new documents. Due to space limitations, we report the performance of baselines on new documents Appendix A.
For Quality Score, SEAL outperforms all the methods including ours. We analyze that this is because SEAL is based on n-grams. Limited by the sparse supervision signal in the dataset, it can- not obtain the expected Recall @ K performance. But the n-grams guarantee that its retrieved docu- ments are of high quality. After adding the docid mappings learned by ASI, SimCSFaocid has greatly improved the Quality Score and even achieves bet- ter performance than ours, which validates the ef- fectiveness of the docid learned by ASI terms of quality. However, they still suffer from low retrieval efficiency. Specifically, SEAL can only generate 10 documents in one generation process (set topk=10 for beam search in decoder), and Sim- CSE also incurs high computational and storage costs even equipped with ANN Search, while our AST could retrieve amount of documents in one generation process without any extra storage. is worth noting that we can retrieve 1344 documents of competitive quality for each query at the same computational cost, which is more than 100 times the efficiency of SEAL.
4.5 Comparison of Variants
We compare our ASI on ADS with the following variants to study the effectiveness of each mod- ule. Specifically, "ASI-Unique" denotes the variant using unique id tokens for different id positions like "0,256,512,768" which is our proposed model; "ASI-Share" denotes the variant using shared id tokens for different id positions like "0,0,0,0"; "w/o Cont" denotes the variant removing the dis- crete contrastive loss; "w/o RCE" denotes the vari- ant removing the sequence-oriented reverse cross- entropy loss; "w/o Repara" denotes denotes the variant re- moving the reparameterization mechanism. We also add a metric Accuracy (Acc) that counts if the semantic indexing module assigns the same docid to the query and document. As reported in Table 5, draw the following conclusions.
Additionally, we apply this semantic indexing module to encode the full collection of documents, and evaluate the quality score. One can see that we can obtain more impressive retrieval efficiency and the price we need to pay is only little bit of acceptable quality degradation. It demonstrates the advantages of our model in terms of both effective- ness and efficiency in practical usage.
As reported in Table 5, "ASI-Unique" slightly outperforms "ASI-Share", indicating the unique id tokens for different id positions carry more se- mantic information. When the discrete contrastive loss is removed, the model completely fails be trained, referring to "w/o Cont", while "w/o RCE" performs better in terms of Recall but the Quality Score drops since more documents are assigned with a same docid (i.e., the D/ID becomes larger).
4.4 Analysis for New Documents
As mentioned before, thanks the semantic index- ing module with two semantic-oriented losses, ASI can better deal with the new documents.

---

Table 5: Comparisons of different variants on dataset ADS.
Variant
R@1 1
R@55
@@ 10
Mi-QS
Ma-QS
DQQ
Acc
DAD
ASI-Unnque
0.3926
0.6701
0.7452
0.5110
0.4946
1028.3
013730
1.1338
ASI-Share
0.3952
0.6542
0.7259
0,5053
0.4857
1343.8
0.3944
1.1539
w/o Cont
0.0000
0.0000
0.0000
0.0000
0.0000
0.0
0.0000
10000
w/o RCE
0,4703
0.7239
0.7826
0,4822
0.4827
1994.7
0.4612
1,1853
w/o Repara 0.3540
0.6375
0.7092
0.4963
0,4875
1212.6
0.4075
1.1492
Table Three docid cases from ADS.
5
Conclusion
Ducid 245,105,,449 149,190 (sapphire rings)
In this paper, we make the first attempt to propose a fully end-to-end pipeline, Auto Search Indexer (ASI), which supports both end-to-end docid in- dexing and end-to-end document retrieval within joint framework. Extensive experiments the public and industrial datasets show that ASI out- performs the SOTA methods for document retrieval. Besides, the experiments also demonstrate the su- periority ASI for handling new documents and verify the effectiveness of its docid assignments.
sapphire engagement rings blue nile propose .. 14k gold sapphire fine rings for jewelry... lab created sapphire ring sterling silver
Docid 2: 245,105, 16,0,,,,1990,, 190 (sapphire earrings)
oblue sapphire diamond carrings... ...
oblue white lab created sapphire earrings sterling silver
sale sapphire stud earrings fine earrings...
Docid 12.187,16,208 (lipstick)
oplum color lipstick target exclusions apply... .. onudestix lip makeup lip products you will onyx professional makeup lipstick ulta beauty mouse...
Limitations
Hierarchical Docids The docid assigned by our ASI does not have hierarchy due to the equivalent multiple linear layers in semantic indexing module. As studied in previous work (Tаy et al., 2022; Wang et al., 2022), hierarchical docids might be more suitable for beam search-based generators. We expect this can be implemented by hierarchical neural clustering-basee indexing module. We left it for future work focus the "fully" end- to-end framework in this paper.
The removal of the reparameterization mechanism causes a certain decrease both Recall and Qual- ity. These results verify the effectiveness of each module. It is more noteworthy that the R@1 of "w/o Repara" is lower than Acc while others are not. It implics that the reparametcrizztiion mecha- nism exactly makes the semantic indexing module be trained jointly with encoder-decoder, so that the decoder achieves superior performance than directly using semantic indexing module.
The docid is represented as integer sequence with no semantic informa- tion that humans can understand. As observed in the case study (referring to 4.6 and B.3), these inte- gers might have become a machine language that only the model itself can understand and use during training and inference. Therefore, we will study how these integer sequences can be interpreted for humans.
Docid Interpretability
4.6 Case Study for Docid Assignment
In this subsection, provide three docid cases from ADS to uncover ASI's docid assignments. As illustrated in Table 6, ASI can assign the same docid to semantically similar documents, e.g., documents of Docid are all about sapphire rings. Besides, ASI also assigns similar docids for similar topics. For example, Docid and Docid 2 differ only in the third position, so the topics are similar, i.e., sapphire rings and sapphire earrings. These cases demonstrate that our proposed model can effectively capture the document semantics and assign meaningful docids for them. Please refer Appendix for more detailed analysis of docid ussignment, where more case analysis can found in Appendix B.3.
Post-Retrieval Filtering Strategy While permit- ting single docid to correspond to numerous doc- uments may improve the efficiency of the Recall phase, it's evident that retrieving excessive num- ber of documents could enlarge significant pressure on the subsequent Ranking phase. This implies that a rest-retrieval filtering might be required to lessen the number of retrieved results.

---

References
Hang Li, Ahmed Mourad, Shengyao Zhuang, Bevan Koopman, and Guido Zuccon. 2023a. Pseudo Rele- vance Feedback with Deep Language Models and Dense Retrievers: Successes and Pitfalls. ACM Trunsactions on Information Systems, 41(3).
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive Search Engines: Generating Sub- strings as Document Identifiers. In Advances in Neu- ral Information Processing Systems.
Yongqi Li, Nan Yang. Liang Wang, Furu Wei, and Wen- jie Li. 2023b. Multiview Identifiers Enhanced Gen- erative Retrieval. In Proceedings of the 61th Annual Meeting of the Association for Computational Lin- guistics. Association for Computational Linguistics.
Jiangui Chen, Ruqing Zhang. Jiafeng Guo, Yixing Fan, and Xueqi Cheng. 2022. GERE: Generative Evi- dence Retrieval for Fact Verification. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2184 2189. Association for Computing Ma- chinery.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretrain- ing Approach. CoRR.
Zhuyun Dai and Jamie Callan. 2020a. Context-Aware Document Term Weighting for Ad-Hoc Search. In Proceedings of The Web Conference 2020, pages 1897-1907. Association for Computing Machinery.
Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twin- BERT: Distilling Knowledge to Strut-Strctturd Compressed BERT Models for Large-Scale Retrieval. In The 29th ACM International Conference on Infor- mation and Knowledge Management, pages 2645- 2652. ACM.
Zhuyun Dai and Jamie Callan, 2020b. Context- Aware Term Weighting For First Stage Passage Retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 5533-15361 Association for Computing Machinery.
Xinyu Ma, Jiafeng Guo, Ruqing Zhang, Yixing Fan, and Xueqi Cheng. 2022. Pre-Train a Discrimina- tive Text Encoder for Dense Retrieval via Contrastive Span Prediction. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 848- 858. Association for Computing Machinery.
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural Rank- ing Models with Weak Supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 65-74, Association for Computing Machinery.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4171-4186, Association for Compu- tational Linguistics.
Laurens van der Maaten and Geoffrey Hinton. 2008 Visualizing data using t-SNE, Journal of machine learning research, 9759):2559-
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. In Proceedings of the Workshop on Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Infor- mation Processing Systems, volume 1773. CEUR- WS.org.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pages 6894-6910. Association for Com- putational Linguistics.
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Her- nandez A Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming- Wei Chang, and Yinfei Yang. 2022a. Large Dual Encoders Are Generalizable Retrievers. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 9855. Association for Computational Linguis- tics.
Jiafeng Guo, Yinqiong Cai, Yixing Fan, Fei Sun, Ruqing Zhang, and Xueqi Cheng, 2022. Semantic Models for the First-Stage Retrieval: A Comprehensive Review. ACM Trans. Inf. Syst., 40(4).
Geoffrey Hinton. 2012. Neural networks for machine learning. Technical report, Coursera.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Deuse Passage Retrieval Question Answering. In Proceed-
Jianmo Ni, Gustavo Hernández Ábrego. Noah Constant, Ji Ma. Keith B. Hall, Daniel Cer, and Yinfei Yang. 2022b. Sentence-TS: Scalable Sentence Encoders from Pre-trained Text-to-Texi Models. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 1864-
for Open-Domaa ings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. Associa- tion for Computational Linguistics.
1874. Association for Computational Linguistics.

---

Rodrigo Frassetto Nogueira, Jimmy Lin, and Al Epis- temic. 2019a. From doc2query to doettttquuer. In Online micro-publication.
Yujing Wang. Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun. Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, 7heng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. 2022. A Neural Corpus Indexer for Document Retrieval. In Advances in Neural Information Pro- cessing Systems.
Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b. Document Expansion by Query Prediction. CoRR.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jum Zhu.
Jingtao Zhan, Jiaxin Mao. Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing Dense Retrieval Model Training with Hard Negatives. In SIGIR '21: The 44th International ACM SIGIR Con- ference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 1503-1512. ACM.
2018. Towards Robust Detection of Adversarial Ex- amples. In Advances in Neural Information Process- ing Systems, volume 31. Curran Associates, Inc.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Gen- eration. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 8821-8831. PMLR.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2022. Learning Discrete Representations via Constrained Clustering for Effec- tive and Efficient Dense Retrieval. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1328-1336. Associa- tion for Computing Machinery.
Stephen E. Robertson and Hugo Zaragoza. 2009, The Probabilistic Relevance Framework: BM25 and Be- yond. Found. Trends Inf. Retr. 3(4):333-389
Anshumali Shrivastava and Ping Li. 2014, Asymmet- ric LSH (ALSH) for Sublinear Time Maximum In- ner Product Search (MIPS). In Advances in Neural Information Processing Systems 27: Annual Con- ference on Neural Information Processing Systems 2014. pages 2321-2329.
Guoqing Zheng and Jamie Callan. 2015. Learning to Reweight Terms with Distributed Representations. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 575-584, Association for Computing Machinery.
Suhas Jayaram Subramanya, Devvrit, Rohan Kadckodi, Ravishankar Krishaswamy, and Harsha Vardhan Simhadri. 2019. DiskANN: Fast Accurate Billion- Point Nearest Neighbor Search on a Single Node. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. Curran Associates Inc.
Yujia Zhou, Jing Yao, Zhicheng Dou, Ledell Wu, Peitian Zhang. and Ji-Rong Wen. 2022. Ultron: An Ultimate Retriever on Corpus with a Model-based Indexer.
Shengyao Zhuang, Houxing Ren, Linjun Shou, Jian Pei. Ming Gong, Guido Zuccon, and Daxin Jiang. 2022. Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation. CoRR.
Weiwei Sun, Lingyong Yan, Zheng Chen, Shuaiqiang Wang, Haichao Zhu, Pengjie Ren, Zhumin Chen, Dawei Yin, Maarten de Rijke, and Zhaochun Ren. 2023. Learning to Tokenize for Generative Retrieval. In Advances in Neural Information Processing Sys- tems.
Yubao Tang, Ruqing Zhang, Jiafeng Guo, Jiangui Chen, Zuowei Zhu Shuaiqiang Wang, Dawei Yin, and Xueqi Cheng. 2023. Bemantec-Enhanced Differen- tiable Search Index Inspired by Learning Strategies. In Proceedings of the 29th ACM SIGKDD Confer- ence on Knowledge Discovery and Data Mining, pages 4913-4913. ACM.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. 2022. Transformer Memory as Differentiable Search Index. In Advances in Neural Information Processing Systems.
Jajja Wang, Weizhong Zhao, Xinhui Tu, and Tingting He. 2023. A novel dense retrieval framework for long document retrieval. Frontiers of Computer Science, 17(4): 774699.

---

Table 7: Performance of SimCSE on new documents
10a
from ADS.
10
Full Valid
New Content
New Semantic
10*
Metrics
Existing
Doc
10²
Sample
10000
1661
8146
193
10
R@1
0.0934
0.1013
0.0901
0.0043
10'
@5
0.2853
0,2888
0.2877
0.0130
R
0.3891
0.3925
0,3884
0.0179
10"
W.1.
0.3122
0.3244
0.3100
0.2987
10
10"
10
10"
DIQ
10
10
# Docid
Table Performance of SEAL on new documents from
Figure 3: Assignment density of docids based on ASI.
ADS.
New Content
New Semantic
tail distribution, which is in line with most real- world data distributions.
Metries
Existing
Valid
Sample
10000
1661
8146
193
R@1
0.0444
0.1295
0.0459
0.0271
B.2
Docid Visualization
R 5
0.1463
0.2073
0.1652
0.0464
R @ 10
0.1998
0.2487
0.2260
0.0656
QS
0.5291
0.5366
0.5290
0.5005
DIQ
10
10
10
10
40
20
A Baseline Performance on new documents
In this section, report the performance new documents of some selected baselines. Specifically, Table 7 and show the performances of SimCSE and SEAL, respectively.
-10
-40
-20
20
40
60
As shown the above tables, the baselines sim- ilarly perform the best on group "Existing" and perform the worst in "NewSemantic". Compared these results with those in Table 4, ASI outper- forms the baselines in the four groups of validation sets on the metric R R@K and shows competitive performance on the metric Quality Score. More im- portantly, we highlight that ASI can simultaneously retrieve far more documents than the baselines with the same computational costs.
Figure 4: T-SNE visualization of docids based on ASI.
In this section, we visualize the assigned docids where the docids are regarded as continuous vec- tor and visualized by t-SNE (Maaten and Hinton, 2008).
As depicted in Figure 4, there is obvious clus- tering phenomenon on the learned docids. On the contrary. the distribution of docids is nearly uni- form. indicates that the proposed semantic index- ing module as well as the two cemantic-orienter losses can evenly learn from different documents. Besides, it verifies the ability of docids to distin- guish different document semantics from the side and also guarantees that each docid has large coverage of semantics.
B Analysis on Docid Assignment
In this section, we make a thorough analysis on the docid assignment ASI on the dataset ADS.
B.1 Assignment Density
In ASI, each docid is allowed to point to several documents. In this section, study the assign- ment density of docids.
B.3 More Cases
We provide more docid cases from dataset ADS in Table 9, 10 and 11.
As illustrated in Figure 3, most of the docids point to a single document, while few of the docids point to hundreds of documents. In other words, the density docid assignments obeys long-
As shown in tables, it can be seen that among the four docid cases Table 9, the first three tokens are of the same while the fourth token is

---

Table 9: Four docid cases about tablecloth from ADS.
Docid I: 14,200,244,50 (green tablecloth)
green tablecloth bed bath beyond what product
help you find clearance... ...
wayfair green outdoor tablecloths you II love 2023 get hy tue feh 7 buy green solid tublecloths online overstock our best table linens decor deals ...
green kitchen tablecloths hed bath beyond clearance ... green modern elegant tablecloths luxury tablecloths bloomingdale's... ...
Docid 2: 11 2,2,0, (gingham tablecloth)
gingham tablecloth serena and lily over 150 new arrivals to explore the fresh start event enjoy
wayfair blue gingham table linens you ll love in 2023 get it by thu feb 16...
wayfair 100 cotton gingham tablecloths you Il love in 2023 am in absolute love with food network woven gingham tablecloth create rustic dining atmosphere
wayfair classic farmhouse gingham tablecloths you II love in 2022 shop wayfair...
Docid 3:
11,200,244,137 (table skirt)
table skirts party tahlecloths target exclusions apply... ... cloth table skirts wayfair.... ...
cloth table skirts pleated wayfair get it by thu fch 9
9ft natural raffia table skirt tropical table skirts for hawaiian decoration tableclothsfactory snap drape table skirting covers clips napkins more snap drape also called sdi brands was founded..
Docid 4: 1,,200224,,55,. (table linens)
sale tablecloths table linens kohl n enjoy free shipping and easy returns every day ... table linens tagged tablecloth page eirene home fashions showing items 57 ...
cyber monday special tablecloth and table linens macy's tablecloth and table linens macy's 's.....
table linens tagged tablecloths elrene home fashions... ...
Table 10: Five docid cases about high neck swimsuit from ADS.
Docid 5:
10,194.75,99 (high neek swimsuit)
high neck swimsuit in living art venus a feminine silhouette that offers fuller coverage women's 's high neek swimsuits Iululemon need it fast use available near you to buy and pick... waterside high neck one piece swimsuit medium hum coverage online sailor blue high neck zip up bikini top bikini venus high neck with zipper the high neck swimsuit in living art venus shop high neck swim dress...
Docid 6: 10, 194,11 8199 (women's 's swim dress)
swim dresses women's 's swimsuit dresses target exclusions apply sale womens swimdress kohl s n enjoy free shipping and easy returns every day
swim dresses women's 's clothing dillard's...
womens swim dress lightinthehox com
swimdress women's 's swimwear macy's 's new plus size cape town
Docid 7: 10,194,203,99 (swimwear)
designer bonpoint swimwear saks fifth avenue designer bonpoint swimwear at saks enjoy free shipping ...
best beverly swim wear coupon codes online stop scarching start saving why scroll when you can save occan dream swimwear shop the world's largest collection of fashion shopstyle shop top ocean ... ashanti swimwear promo eodes deals discounts for free january 2023 install capital one shopping apply ... swimwear and beachwear zimmermann net porter claim your exclusive discount code when you subscribe to ...
Docid 8: 10,194,225,99 ,99 (bikini)
bikini shorts swim shorts for women venus get sexy bikini shorts that keep you on the full coverage swim shorts bikini sweet dreams venus twisted bodice accentuates your curves while adding swim shorts bikini aqua reef venus achieve more modest beach day look in this pair.. women's 's swimsuits micro bikinis beach cover ups asos getting ready for your holidays wherever you women's swimwear bikinis tankinis fatface us the warmest days of the year are on their way
Docid 9: 10,194.231,99 1,99 (women's 's one piece swimsuits)
women's 's one piece swimsuits lululemon need it fast use available near you to buy and pick... one piece swimsuits for women macy's 's one piece swimsuits are a must have for any woman's 's...
black one piece women's swimsuits swimwear macy's 's new women's 's linked in colorblncked occanus one piece women's swimsuits swimwear macy's 's new women's 's bias stripe bandeau one piece swimsuit ... one piece swimsuits for women target exclusions apply n whether you re planning a beach vacation

---

Table 1I: More docid cases starting with "11" id token from ADS.
Docid 10: 11,245,117,200 (bowl)
plastic serving bowls williams sonoma sugg price 59 95 disposable bowls walmart com green walmart com n shop for disposable bowls walmart com plastic bowls round seagreen 2oz 100 count box my cart...
Docid 11: 11,186.19..44 (table number)
top 10 best wedding table numbers gold of 2023 ...
table number cards zazzle new instant downloads weddings invitations cards
letters table numbers efavormart decorations prove be the most important uspect of party
Docid 12: 11.1 11.17.445.92 (marble coffee table)
pierre marble coffee table williams sonoma buy in monthly payments with affirm on orders over 50
madison park signature marble coffee table with its luxurious marble top this madison park... buy modern contemporary marble coffee tables online at overstock our hest living room furniture..
Docid 13: 11,1 70,0,92 (glass coffee table)
buy glass square coffee tables online overstock our hest living room furniture.. st germain glass coffee table serena and lily buy monthly payments with affirm on orders ... glass coffee tables raymour flanigan a coffee table for more than just coffee...
Docid 14: 13,155,1 12,203 (disposable plates)
disposable plates efavormart 5 10 off all folding chair covers... our 10 best disposable plates in the us january 2023 bestproductsrreiees com blue panda disposable plates 48 pack paper plate party supplies...
Docid 15: 11,135,197,34 (plastic tablecloth)
plastic tablecloth rolls in plastic tablecloths walmart com free deals discounts on clear plastic tahlecloth roll w self cutter wide thick disposahle table cover... plastic table cloths etsy...
Docid 16: 11,15 12,34 (paper plates)
vdomdhtm/html our 10 best paper plates in the us january 2023 bestproducctreeviewss.o com black and white paper plates wayfair get by tue feh 14... paper plates white walmart way to celebrate white paper dessert plates 7in 24ct
Docid 17: 11, 12,82,24,11 137 (napkins)
table linens napkins williams sonoma earn 10 back in rewards with the new williams sonoma credit linen napkins when it comes to event décor every detail counts high quality banquet napkins... farmhouse cloth napkins wayfair thought this was a great buy for that many napkins
Docid 18: 11,21,140, 140,203 (dinnerware)
world tableware dinnerware foodservice products webstaurantstore for years restaurant dinnerware wholesale plates lowls dishes step up the elegance your restaurant... tabletop dinnerware serveware kohl enjoy free shipping and easy returns every day ...

---

Table 12: Quantitative similarity comparison.
0.60
0.90
M-Q5
Ma-Q5
TF-IDF Vector
BERT Embedding
0.50
Random
0.0078 0.0270
0.60889 + 0.0820
0.60
ASI
0.2549 0. 1 1148
0.6864 + 0.0469
p-value
6.50 x 10" 201
5.22 x 10
0.30
0.40
R@
R.g
R
different, indicating that this group of docids are similar to each other. Specifically, these four docids are all related to "tablecloths", but each has its own emphasis details, i.e., they focus on "green table- cloth", "gingham tablecloth", "table skirt", "table linens", respectively. Similarly, the five docid cases in Table 10 are all involved "swimsuits", while they focus on the finer-grained topics "high neck swim- suit", "women's 's swim dress", "swimwear", "bikini" and "women's s one piece swimsuits". It is worth noting that the shared id token in Table 10 appears in the first, second and fourth positions, while the different id token appears in the third position. This is because the id tokens in different positions are equivalently assigned by our semantic indexing module.
0.00
5
0.30
(a) Performance
15000
D/Q
D/ID
2.00
1.50
10000
1.00
5000
0.50
0
0.00
(b) Docid Statistics
Figure 5: Performance and docid statistics under differ- docid lengths.
In addition, the two groups of docids in the two tables totally different, which is also in line with the fact that the poarsrr-grained topies of the two are different, i... "tablecloths" and "swimsuits".
random sampling, "ASI" means sampling from the same docid, and "p-value" represents whether the difference between the above two is statistically sig- nificant based on t-test (usually, < 0.01 means extremely significant).
Moreover, provide more cases whose docids start with "11 in Table 11. As illustrated in the table, the topics of the 9 docids are "bowl", "ta- ble number". "marble coffee tab", "glass coffee tab", "disposable plates", "plastic tableclot", "pa- per plate", "napkin" and "dinnerwar", respectively. Together with the cases in Table 9, these docids that start with "11" are all related to "tableware", It implies that one id token will correspond to coarser concept, and group of id tokens will point finer concept. These cases verify the effective- ness of our model to learn semantically meaningful docids for documents.
As shown in Table 12, the similarity between the documents of same docid is significantly larger than that between random documents, which demonstrates the effectiveness of docid assign- ments learned by ASI. We will add the above re- sults in the revision.
C Parameter Sensitivity
There are two essential hyper-parametee in ASI, i.e. the length of docid m, and the range each token. In this section, we study the impact of these two hyper-parameters.
B.4 Quantitative Analysis
C.1 Impact of Docid Length
In this section, we conduct quantitative analysis for the docid assignment. In ASI, the semantic indexing mechanism allows docid to point to mul- tiple documents based the principle of sharing docids for similar documents and distinguishing docids for the dissimilar ones. Therefore, we mea- sure the averaged cosine similarity of document pairs based on TFIDF vector and BERT embed- dings. Specifically,, "Random" means completely
We evaluate the Recall@K and Quality Score for ASI of different docid lengths from 2 to 6 and the docid range is set to 256. As depicted in Fig- ure 5(a), the length of the docid increases, the micro- and macro-averaged Quality Scores achieve much better while the Recall @ 1/5/10 on the con- trary decrease obviously. This is because the model capacity increases with the growth of docid length,

---

0.90
MIQS - MaQs
0.50
0.60
0.30
0.40
R@1
- Rg
Ri@o
0.00
0.30
32
64
128
(a) Performance
D/Q
D/ID
2.00
4000
1.50
1.00
2000
0.50
0
0.00
32
64
128
256
512
(b) Docid Statistics
Figure 6: Performance and docid statistics under differ- ent docid ranges.
while the difficulty of generation for the decoder also increases due to the longer target docid. Con- sequently, ASI could assign different docids to doc- uments based on their finer-grainct semantics (re- ferring to 5(b)), supporting better retrieval quality the expense of recall.
C.2 Impact of Docid Range
We evaluate the performance of ASI with different docid ranges from [O, 32) to [0, 512), where the length is set to 4. As illustrated in Figure 6, there are similar observations to those above for different ranges of docids. Differently, the performance of ASI is not very sensitive to the docid range. We analyze it may be due to the fact that model capacity grows exponentially with docid length, but polynomial with docid range.
All in all, the above two experimental results imply a trade-off practical applications between efficiency and effectiveness. Therefore, in the main body of this paper, we choose to set the length of docid m= and the range of each id token is set to [0, 256).


Reviews:
[{"review_id": "fTEjX2Ymiw", "reviewer": "Reviewer_CdLZ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you a lot for your positive comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and well-structured. The proposed model is novel and addresses one of the fundamental challenges of the new paradigm of generative information retrieval. The baselines are thorough and the results show substantial improvement over all the baseline models including sparse retrieval, dense retrieval, and generative retrieval. ", "reasons_to_reject": "N/A", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an end-to-end generative information retrieval pipeline, Auto Search Indexer (ASI), that supports document-id assignment as well as document retrieval. ASI follows a seq-2-seq architecture that encodes user query q and generates the relevant documents. The proposed approach is novel since the previous work relies on preprocessed doc_ids and trained indexers can not be utilized with new documents. The model is trained with respect to a sequence loss and two semantic-oriented losses. The semantic-oriented losses are 1) contrastive loss between the q_id and pos_doc_id and 2) Reverse Cross Entropy loss between the pos_doc_id and neg_doc_id with the intuition of making the two id token distant from each other. The authors perform extensive experiments on one public and one industrial dataset and the results show solid improvement over the baseline. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p22ekw2Xr1", "reviewer": "Reviewer_rkeY", "summary": "", "questions": "- For multiple documents share the same doc id, have the authors manually check the similarity of those documents?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below, we respond to each of your concerns in detail. \n\n$ $\n\n1. *Regarding the comparison of recall/MRR metrics when one docid matches multiple documents.* \n\n   - Yes, when one docid points to multiple documents, it is indeed somewhat unfair for baselines to directly compare on Recall/MRR. To tackle this issue, as described on Line 488-495, we have **randomly sampled a single document for each of those docids**. Under this simple strategy, our ASI would also **retrieve the same number of documents as baselines** without additional prior knowledge, which means that our ASI did **NOT look for documents at more positions** than the baselines. Besides, in order to further eliminate the occasional performance fluctuation caused by the random sampling strategy, we choose to report the expectation under this random strategy, i.e., ASI(Expectation) in Table 2. As such, a fairer comparison is finally achieved and the results demonstrate that when retrieving the same number of documents, ASI still outperforms the baselines. We will clarify this point in the revision. \n\n$ $\n\n2. *Regarding the document ranking strategy if one docid is associated with multiple documents.* \n\n   - This is a valuable research question. In this paper, we do not employ any ranking strategy, but simply consider all the retrieved documents equally relevant to the query. Specifically, in the evaluation of ASI(Expectation), we simply apply **random** sampling strategy. For the calculation of MRR metric, the documents of the same docid are randomly sorted, and the documents of different docids are arranged in groups according to the order of docid. This is because we focus on the \"fully\" end-to-end framework in this paper, and thus we leave the document ranking strategy for future work. \n\n   - PS: As expected, an effective ranking strategy can greatly benefit the one-to-many id mapping of ASI. Another experimental result on our real applications shows that the relevance score is improved nearly 3 times (0.0576 $\\rightarrow$ 0.1602) when we adopt an online lightweight ranking model as a post-processing filter. \n\n$ $\n\n3. *Regarding the generative retrieval baseline in ADS dataset.* \n\n   - In fact, we have already conducted extensive experiments on MS MARCO dataset, which provides sufficient comparisons for the generative retrieval baselines. \n\n   - Furthermore, considering new documents will frequently appear in practical applications, it is necessary for the retrieval model to be able to handle new documents. Accordingly, on dataset ADS, we aim at a more comprehensive evaluation system such as the retrieval quality (quality score) and the performance on new documents (Section 4.4). To this end, we select SEAL as the generative retrieval baseline on ADS dataset, which relies on ngram-based docids so that new documents can be easily incorporated by simply rebuilding the FM index without retraining the model. In contrast, the other methods (NCI, DSI, DynamicRetriever, Ultron-Atomic, etc.) rely on virtual token-based docids that are generated during preprocessing phase, which makes it difficult for the model to generalize to new documents. We will clarify this point in the revision. \n\n   - As an improper alternative, we preprocess these new documents from validation set *directly* with the training set, although this setting actually cannot be applied in practice as the new documents are not predictable. Here we report the performance of Ultron-PQ under this cheat setting (Ultron-Atomic will cause OOM due to the huge scale of document corpora): \n\n      |         Model         |  R@ 1  |  R@5   |  R@10  | Mi-QS  | Ma-QS  | D/Q  |\n      | :-------------------: | :----: | :----: | :----: | :----: | :----: | :--: |\n      |         SEAL          | 0.0444 | 0.1463 | 0.1998 | 0.5291 | 0.5291 |  10  |\n      |        SimCSE         | 0.0934 | 0.2854 | 0.3891 | 0.3122 | 0.3122 |  10  |\n      | SimCSE$_\\text{docid}$ | 0.2768 | 0.4593 | 0.5008 | 0.5290 | 0.5087 | 443  |\n      |       Ultron-PQ       | 0.0281 | 0.0720 | 0.0854 | 0.4343 | 0.4302 |  10  |\n      |          ASI          | 0.3952 | 0.6542 | 0.7259 | 0.5053 | 0.4857 | 1344 |\n\n   - As shown in the above table, even though Ultron obtains the content of the new documents via cheat settings, it still does not outperform, which proves the superiority of ASI. Besides, we find its performance on ADS is far worse than that on MSMARCO. We analyze that Ultron is based on non-semantic virtual tokens, as also studied in [1], which makes it difficult to handle large-scale corpora. \n\n$ $\n\n4. *Regarding the manual check of document similarity of same docid.* \n\n   - Yes, we have checked. Consequently, as shown in Section 4.6 and Appendix A.3, we have provided several cases to qualitatively uncover ASI's docid assignments. One can see that the documents of the same docid are exactly similar. From a quantitative point of view, the quality score on the ADS dataset in Table 3 can also reflect the similarity of documents of the same docid, since better quality means that the document semantics of the same docid are all relevant to the given query). \n\n   - In addition, we provide a more direct quantitative comparison here: the averaged cosine similarity of document pairs based on TFIDF vector and BERT embeddings. \"Random\" means completely random sampling, \"ASI\" means sampling from the same docid, and \"p-value\" represents whether the difference between above two is statistically significant based on t-test (usually, $p<0.01$ means extremely significant). \n\n      |         | TF-IDF Vector           | BERT Embedding          |\n      | ------- | ----------------------- | ----------------------- |\n      | Random  | 0.007862 $\\pm$ 0.027043 | 0.608963 $\\pm$ 0.082007 |\n      | ASI     | 0.254954 $\\pm$ 0.114874 | 0.686494 $\\pm$ 0.046917 |\n      | p-value | $6.50\\times 10^{-251}$  | $5.22\\times 10^{-200}$  |\n   \n   - As shown in the above table, the similarity between the documents of same docid is significantly larger than that between random documents, which demonstrates the effectiveness of docid assignments learned by ASI. We will add the above results in the revision. \n\n\n\n [1] Pradeep, Ronak, et al. \"How Does Generative Retrieval Scale to Millions of Passages?.\" arXiv preprint arXiv:2305.11841 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Expanding generative retrieval model to new documents is an important research problem.\n\n- It is an interesting idea to generate document ids based on the semantics of document content.", "reasons_to_reject": "- The major concern with this paper is the performance of the proposed model.  Since one doc id could be matched to multiple documents, the recall/MRR metric computation is unfair for baselines. Though the authors compute the expectation of Recall/MRR, essentially the proposed model still look for documents at more positions than the baselines.\n\n- Some key technical details are missing. When one doc id is associated with multiple documents, how those documents will be ranked? Are they considered equally relevant to the query? \n\n- The generative retrieval baseline in ADS dataset is weak. From the experiment in MS MARCO dataset, SEAL is not the best performing generative retrieval baseline. The authors should report the results for stronger baseline like the Ultron-Atomic.", "questions_to_authors": "- For multiple documents share the same doc id, have the authors manually check the similarity of those documents?", "paper_topic_and_main_contributions": "This paper proposes an interesting idea to generate document ids based on document semantics for generative retrieval. With this method, generative retrieval model can be easily applied to new documents. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O1842R2qlp", "reviewer": "Reviewer_agxP", "summary": "", "questions": "1.\tHow does the decoder ensure the validity of the output doc id?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below, we respond to each of your concerns in detail. \n\n$ $\n\n1. *Regarding the related work [1] .*  \n   - Thank you for providing this concurrent study. Considering it is shared in arXiv as a **preprint** within three months, we did not discuss this **concurrent work** as per the [EMNLP policy](https://2023.emnlp.org/calls/main_conference_papers/#citation-and-comparison). Although this GenRet is an end-to-end framework for learning to assign the doc-id together with retrieval stage, its structure and training processes are extremely complicated. While ours leverages two semantic-oriented losses to automatically learn the docid assignments together with retrieval in an end-to-end manner based on the Query-Doc pairs. Moreover, the performance of our ASI is significantly better than GenRet on MS MARCO (see Response 2). \n\n   - Nevertheless, we will still add the following discussion in the revision: \n\n   - Concurrently with our work, Sun et al. also investigated a totally different end-to-end framework, GenRet, which uses a codebook and discrete auto-encoder with progressive training to learn the doc-id assignments within retrieval stage. However, the decoding semantic space of GenRet (i.e., the codebook) is initialized by an isolated clustering module, while ours is learned end-to-end by ASI itself just using Query-Doc pairs, thus resulting in no semantic gap. Additionally, GenRet relies on a progressive training scheme and unique docid assignment, leading to limited training and inference efficiency for large-scale corpora. \n\n$ $\n\n2. *Regarding the suggested baselines [2] [3].* \n\n   - Thank you for providing these SOTA baselines. We will add the comparisons as follows in the revision:\n\n     | Model                      |      Params      |         R@1          |         R@5          |         R@10         |       MRR@ 10        |\n     | :------------------------- | :--------------: | :------------------: | :------------------: | :------------------: | :------------------: |\n     | BM25 $\\dagger$             |        -         |        0.1894        |        0.4282        |        0.5507        |        0.2924        |\n     | DocT5Query $\\dagger$       |        -         |        0.2327        |        0.4938        |        0.6361        |        0.3481        |\n     | RepBERT $\\dagger$          | $220 \\mathrm{M}$ |        0.2525        |        0.5841        |        0.6918        |        0.3848        |\n     | Sentence-T5 $\\dagger$      | $220 \\mathrm{M}$ |        0.2727        |        0.5891        |        0.7215        |        0.4069        |\n     | DPR $\\dagger$              | $220 \\mathrm{M}$ |        0.2908        |        0.6275        |        0.7313        |        0.4341        |\n     | SimCSE $\\ddagger$          | $110 \\mathrm{M}$ |        0.2867        |        0.6470        |        0.7322        |        0.4390        |\n     | GTR-Base[3]                | $110 \\mathrm{M}$ |        0.4620        |          -           |        0.7930        |        0.5760        |\n     | DSI-Semantic $\\dagger$     | $250 \\mathrm{M}$ |        0.2574        |        0.4358        |        0.5384        |        0.3392        |\n     | DSI-Atomic $\\dagger$       | $495 \\mathrm{M}$ |        0.3247        |        0.6301        |        0.6992        |        0.4429        |\n     | DSI-QG [2]                 | $200 \\mathrm{M}$ |        0.2381        |          -           |        0.2805        |          -           |\n     | NCI $\\ddagger$             | $376 \\mathrm{M}$ |        0.2574        |        0.4358        |        0.5384        |        0.3392        |\n     | SEAL $\\ddagger$            | $139 \\mathrm{M}$ |        0.2884        |        0.5683        |        0.6829        |        0.4066        |\n     | DynamicRetriever $\\dagger$ | $495 \\mathrm{M}$ |        0.2904        |        0.6422        |        0.7315        |        0.4253        |\n     | Ultron-URL $\\dagger$       | $248 \\mathrm{M}$ |        0.2957        |        0.5643        |        0.6782        |        0.4002        |\n     | Ultron-PQ $\\dagger$        | $257 \\mathrm{M}$ |        0.3155        |        0.6398        |        0.7314        |        0.4535        |\n     | Ultron-Atomic $\\dagger$    | $495 \\mathrm{M}$ |        0.3281        | $\\underline{0.6490}$ |        0.7413        |        0.4686        |\n     | GenRet[1]                  | $215 \\mathrm{M}$ | $\\underline{0.4790}$ |          -           | $\\underline{0.7980}$ | $\\underline{0.5810}$ |\n     | ASI                        | $125 \\mathrm{M}$ |  $\\mathbf{0.6121}$   |  $\\mathbf{0.7831}$   |  $\\mathbf{0.8207}$   |  $\\mathbf{0.6857}$   |\n     | ASI (Expectation)          | $125 \\mathrm{M}$ |  $\\mathbf{0.5497}$   |  $\\mathbf{0.7072}$   |        0.7414        |  $\\mathbf{0.6175}$   |\n\n   - The performance of GTR-Base[3] and GenRet[1] are referred from [1] and DSI-QG[2]'s is referred from [4]. \n\n$ $\n\n3. *Regarding the performance of baselines on new documents.* \n\n   - Thanks for your valuable suggestions. We will add the following comparisons on new documents in the revision:\n\n     | Metrics    | Full Valid | Existing | New Content | New Semantic |\n     | :--------- | :--------: | :------: | :---------: | :----------: |\n     | # Sample   |   10000    |   1661   |    8146     |     193      |\n     |            |            |          |             |              |\n     | **ASI**    |            |          |             |              |\n     | R@1        |   0.3952   |  0.4218  |   0.4088    |    0.0570    |\n     | R@5        |   0.6542   |  0.6865  |   0.6629    |    0.2073    |\n     | R@10       |   0.7259   |  0.7583  |   0.7333    |    0.2487    |\n     | Mi-QS      |   0.5053   |  0.5174  |   0.5041    |    0.4640    |\n     | Ma-QS      |   0.4857   |  0.5006  |   0.4872    |    0.4638    |\n     | D/Q        |    1344    |   1565   |    1178     |     925      |\n     |            |            |          |             |              |\n     | **SimCSE** |            |          |             |              |\n     | R@1        |   0.0934   |  0.1013  |   0.0901    |    0.0043    |\n     | R@5        |   0.2853   |  0.2888  |   0.2877    |    0.0130    |\n     | R@10       |   0.3891   |  0.3925  |   0.3884    |    0.0179    |\n     | Mi-QS      |   0.3122   |  0.3244  |   0.3100    |    0.2987    |\n     | Ma-QS      |   0.3122   |  0.3244  |   0.3100    |    0.2987    |\n     | D/Q        |     10     |    10    |     10      |      10      |\n     |            |            |          |             |              |\n     | **SEAL**   |            |          |             |              |\n     | R@1        |   0.0444   |  0.1295  |   0.0459    |    0.0271    |\n     | R@5        |   0.1463   |  0.2073  |   0.1652    |    0.0464    |\n     | R@10       |   0.1998   |  0.2487  |   0.2260    |    0.0656    |\n     | Mi-QS      |   0.5291   |  0.5366  |   0.5290    |    0.5005    |\n     | Ma-QS      |   0.5291   |  0.5366  |   0.5290    |    0.5005    |\n     | D/Q        |     10     |    10    |     10      |      10      |\n\n   - As shown in the above table, ASI outperforms the baselines in the four groups of validation sets on the metric Recall@K and shows competitive performance on the metric Quality Score. More importantly, we highlight that ASI can simultaneously retrieve far more documents than the baselines with the same computational costs. \n\n$ $\n\n4. Regarding how to ensure the validity of the output docid?\n\n   - Actually, there is no need to adopt methods such as constraint beam search to enforce that the generated docid must have corresponding documents. On the one hand, as studied in Section 4.4., new documents might occupy new docids. Consequently, the \"invalid\" docids have certain guiding significance for us to expand the coverage of documents. On the other hand, we have also counted the proportion of generated invalid docids, which is only about **0.58%**. \n\n$ $\n\n[1] Sun, Weiwei, et al. \"Learning to Tokenize for Generative Retrieval.\" arXiv preprint arXiv:2304.04171 (2023). \n\n[2] Zhuang, Shengyao, et al. \"Bridging the gap between indexing and retrieval for differentiable search index with query generation.\" In The First Workshop on Generative Information Retrieval at SIGIR2023. 2023. \n\n[3] Ni, Jianmo, et al. \"Large Dual Encoders Are Generalizable Retrievers.\" In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n[4] Tang, Yubao, et al. “Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies.” In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe end-to-end pipeline proposed in this paper is effective for the combination of doc-id assigning and retrieval, and resolves the new document problem. \n2.\tThe three losses are technically sound and are shown useful for the performance.\n3.\tExtensive experiments have been conducted to evaluate the model on suitable datasets and metrics. And the results clearly show significant improvements over competitive baselines.\n", "reasons_to_reject": "1.\tSun et al. April 2023 also proposes an end-to-end framework for learning to assigning the doc-id together with retrieval stage.\n2.\tMore baselines on improving generative retrieval e.g., Zhuang et al., and dense retrieval e.g., Ni et al, in recent years could be included.\n3.\tThe performance on new documents is not compared with the baseline methods. \n\nSun, Weiwei, et al. \"Learning to Tokenize for Generative Retrieval.\" arXiv preprint arXiv:2304.04171 (2023).\nZhuang, Shengyao, et al. \"Bridging the gap between indexing and retrieval for differentiable search index with query generation.\" arXiv preprint arXiv:2206.10128 (2022).\nNi, Jianmo, et al. \"Large Dual Encoders Are Generalizable Retrievers.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n", "questions_to_authors": "1.\tHow does the decoder ensure the validity of the output doc id?", "paper_topic_and_main_contributions": "This paper proposes a new end-to-end generative retrieval framework called Auto Search Indexer (ASI) for document retrieval via an encoder-decoder-based model. The main contributions are:\n1.\tFully end-to-end pipeline that supports assigning and retrieval within a joint framework.\n2.\tTwo novel semantic-oriented losses for training the indexing module.\n3.\tDemonstrating superior performance over existing baselines.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1.\tSun, Weiwei, et al. \"Learning to Tokenize for Generative Retrieval.\" arXiv preprint arXiv:2304.04171 (2023).", "Typos_Grammar_Style_and_Presentation_Improvements": "1.\tThe paper is reasonably well written overall.\n2.\tThe captions in the tables could be more detailed.\n"}, "extra_text": {}}, {"review_id": "RZimfDz2Gv", "reviewer": "Reviewer_WTWX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you a lot for your positive comments!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors first clearly highlight the prevailing issues in generative retrieval. To alleviate these, they proposed a novel seq2seq model for generative retrieval. Their motivation makes sense and is convincing. Furthermore, the authors validate their method on two datasets, demonstrating significant improvements compared to previous works. Overall, this paper is well-structured and may make a valuable contribution to the field of generative retrieval research.", "reasons_to_reject": "None", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a groundbreaking approach called Auto Search Indexer (ASI), which revolutionizes the retrieval process. ASI is a novel and fully end-to-end generative retrieval paradigm that seamlessly integrates two critical components: end-to-end learning of docids for both existing and new documents, and end-to-end document retrieval. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission studies the generative retrieval and focuses on the problem of document indexing to handle new documents properly. In particular, this submission proposes a new framework with a reparameterization mechanism to combine two modules of semantic indexing and generation. Extensive experimental results on both public and industrial datasets validate the effectiveness of the proposed method. Active discussion and additional experiments addressed most concerns from reviewers and further demonstrated the rationality of the proposed method.
Decision: Accept-Findings