Kati8/ICLR_2023_2025 huggingface dataset:
authors: Wuchao Li
authors: Kai Zheng
authors: Defu Lian
authors: Qi Liu
authors: Wentian Bao
authors: Enyun Yu
authors: Yang Song
authors: Han Li
authors: Kun Gai
reviews: 
Comments: This paper proposes an end-to-end retriever and index, It simultaneously clusters query and relevant items and ensure that they both follow similar distribution over cluster nodes at each level. The URI (proposed approach) ensures that clusters are balanced and accurate.
Strengths: This paper proposes an end-to-end retriever and indexer.
Results are state of the art
Weaknesses: - This paper looks a lot like EHI: End-to-end Learning of Hierarchical Index for Efficient Dense Retrieval with similar loss functions, TMLR 2024
- Results on standard bier benchmark datasets are missing
- In real world scenario new items are always popping in and out of existence, how does URI deals with zero shot items
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper proposes URI (Unified framework for Retrieval and Indexing), a novel approach that unifies index construction and retrieval model training for generative retrieval systems. The key innovation is using the same Transformer decoder both as the retriever and the indexer, constructing the index simultaneously with model training rather than as a separate pre-processing step. The paper introduces Average Partition Entropy (APE) as a new metric for evaluating generative indices and provides theoretical analysis using EM algorithm to justify their approach. The authors demonstrate URI's effectiveness through experiments on three real-world datasets.
Strengths: - The paper identifies clear limitations in existing two-stage approaches where index construction is separated from retrieval model training. 
- The approach of using the decoder itself as an indexer is creative and original and introduction of APE as an evaluation metric is well-justified and useful
- Addresses fundamental limitations in generative retrieval systems leads to potentially applicable across various retrieval tasks
- Could influence future work in information retrieval and recommender systems
Weaknesses: Limited Empirical Analysis
- While experiments are conducted on three datasets, more details about these datasets would be helpful
- Ablation studies could better isolate the impact of different components
- Comparison with more baseline methods would strengthen the evaluation

Scalability Concerns
- The paper doesn't thoroughly discuss computational complexity
- Practical considerations for large-scale deployment are not addressed
- Memory requirements for the unified approach could be significant
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces URI (Unified Retrieval and Indexing), a framework for generative retrieval that integrates index construction with Transformer Decoder training. URI enhances consistency between the index and retrieval model. The authors also introduce Average Partition Entropy (APE), a model-independent metric for evaluating generative indices after index construction. Targeting this metric, they also propose an optimization algorithm with a theoretical explanation. Experiments confirm the effectiveness of the proposed method.
Strengths: - The motivation is clear. The authors argue that recent methods separate index construction from retrieval  in traditional and generative models, which limits the overall performance. Thus, this paper attempts to unify index construction and retrieval.
- This paper provides a theoretical guarantee of the proposed greedy algorithm.
- The experiments on multiple benchmark datasets validate the effectiveness of the proposed method compared with the baseline models.
Weaknesses: - The authors overlook recent retrieval work such as [1,2]. The authors are encouraged to discuss them in the related work.
- Since the final goal of the proposed method has many hyper-parameters in (11), more sensitivity analysis also is crucial. 
- Theorem 1 lacks the corresponding empirical results, so more evidence is needed.
- Including both index construction and retrieval in the unified framework may increase computation and storage overhead, affecting efficiency. Therefore, an efficacy analysis (e.g., memory usage, training/inference time, and FLOPs) is necessary.

[1] Improved Diversity-Promoting Collaborative Metric Learning for Recommendation. IEEE TPAMI, 2024.

[2]Hierarchical latent relation modeling for collaborative metric learning,”inRecSys,2021
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper proposes URI, a variant of generative retrieval methods which learns the retrieval model and index construction jointly. Besides, this paper also proposes averaged partition entropy (APE) as a measure for index construction quality. In URI, the same decoder is used to generate tokens for both query and items and is trained in a using an EM algorithm. In E-step, the model is optimized through minimizing the cross entropy loss.  In M-step, To avoid model collapse while minimizing APE, two loss functions apart from the cross entropy loss are introduced to encourage that the token distribution w.r.t. each item follows a peak distribution while the expected token distribution w.r.t. the whole item space follows a uniform distribution. Finally, an additional dual encoder model is trained to identify the topk items as the final retrieved result. Experiments on KuaiSAR and Amazon Beauty/Toys/Games demonstrates the effectiveness of URI compared to other generative retrieval methods.
Strengths: 1. Learning to construct index jointly with retrieval models is an important problem for generative retrieval. The proposed method is well motivated.
2. Performance is impressive, especially on the KuaiSAR dataset.
Weaknesses: 1. Writing can be further improved, especially the experiment section. The difference between URI and GR methods w/ URI index is unclear in Table 1. The analysis of token consistency is hard to understand and it will be better to demonstrate their differences to original URI through notations.
2. Insufficient analysis and comparison on other generative methods (e.g., ASI [1] and GenRet [2] ) that learns both retrieval models and indexing jointly. 
3. The assumption of Theorem 1 seems very strong and there lacks empirical analysis on real-world dataset to verify the rationality. 
4. The comparison in experiments is not unfair. The candidate size of URI is larger than other GR methods which requires the mapping between the token representation and the item is bijective while URI allows each leaf node contains more than 1 items. Besides, URI is equipped with an additional ranker.

[1] Yang T, Song M, Zhang Z, et al. Auto Search Indexer for End-to-End Document Retrieval[C]//Findings of the Association for Computational Linguistics: EMNLP 2023. 2023: 6955-6970.
[2] Sun W, Yan L, Chen Z, et al. Learning to tokenize for generative retrieval[J]. Advances in Neural Information Processing Systems, 2024, 36.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 1