Title: FORGE: Forming Semantic Identifiers for Generative Retrieval in Industrial Datasets
Primary area: datasets and benchmarks
Keywords: semantic identifiers, industry dataset, generative retrieval

Abstract: Semantic identifiers (SIDs) have gained increasing attention in generative 
retrieval (GR) due to their meaningful semantic discriminability. However, current research on SIDs faces three main challenges: (1) the absence of large-scale public datasets with multimodal features, (2) limited investigation into optimization strategies for SID generation, which typically rely on costly GR training for evaluation, and (3) slow online convergence in industrial deployment. To address these challenges, we propose **FORGE**, a comprehensive benchmark for **FO**rming semantic identifie**R** in **G**enerative r**E**trieval with industrial datasets. Specifically, FORGE is equipped with a dataset comprising **14 billion** user interactions and multimodal features of **250 million** items sampled from one of the biggest e-commerce platforms in China, which serves over 300 million users each day. Leveraging this dataset, FORGE explores several optimizations to enhance the SID construction and validates their effectiveness via offline experiments across different settings and tasks. Further online study deployed in our platform for the homepage recommendation shows a 0.35\% increase in transaction count, highlighting the practical impact of our method. Regarding the expensive SID validation accompanied by the full training of GRs, we propose two novel metrics of SID that correlate positively with the recommendation performance, enabling convenient evaluations without any GR training. For real-world applications, FORGE introduces an offline pretraining schema that reduces online convergence by half of the original. The code and data are available at https://anonymous.4open.science/r/forge.

TL;DR: We propose FORGE, a comprehensive benchmark for forming semantic identifiers in generative retrieval with industrial datasets.