Less is More: Learn to Select Histories for Personalized Search
Authors: Jiongnan Liu, Zhicheng Dou, Shuting Wang, Xiaochi Wei, Yu Lu, Shuaiqiang Wang, Dawei Yin


Reviewer paper summaries: This paper presents a personalized search paradigm that incorporates a history selector. This history selector serves to filter noise from users' history and improve the quality of personalized ranking.
Summary of strengths: The experimental results demonstrate that the Filtered Personalized Search (FPS) framework outperforms other personalized models, showcasing the effectiveness of the proposed model.
Summary of weaknesses: 1. The model lacks novelty, as its modules closely resemble those in several other motivation papers. However, the results of the ablation study suggest that removing these modules may not significantly affect the performance of the FPS model. As a result, it is important to determine the actual impact of these modules on the overall performance of the model.
2. Insufficiently comprehensive research is evident as the majority of references, related works, and baselines are attributed to the same team. However, it would be valuable to consider including works related to personalized search from other teams as references.
3. There is a lack of clarity regarding the approach to feature construction, experimentation, dataset usage, and evaluation in this work:
- Feature construction: Equation 10 indicates that the authors utilize not only personalization scores as input features for ranking, but also concatenate ad-hoc relevance scores by the K-NRM model, as well as some handcrafted relevance features. It is unclear why these additional features are used in Filtered Personalized Search, and how the authors developed the handcrafted relevance features.
- Experimentation: It appears that this model is a re-ranking model, but there is no clear explanation of how the candidates are retrieved from the corpus, and how many candidates are used in the training and testing processes.
- Dataset: The two datasets used do not appear to be open-source. Additional experimental results in open-source datasets would be helpful.
- Evaluation: The authors have chosen MAP, MRR, and Precision@1 as metrics, but it is not specified whether MAP is calculated at 100, 1000, or any other value, and similarly for MRR.
Comments, suggestions and typos: 1. Are the user histories h_i input to the text encoder sorted by time, or are they randomly sorted when input to the text encoder?
2. How many masks in the history selector belong to the same session as the query? Could all masks be from other sessions, allowing us to consider session as a filter?
3. Can you provide an ablation study with personalization scores, ad-hoc relevance scores by the K-NRM model, and handcrafted relevance features? It would be interesting to understand which kind of features have the most significant impact on this ranking model.
4. The performance of the Top K selector is comparable to that of the self-adaptive dynamic threshold. From Table 2, the self-adaptive dynamic threshold has helped to improve 0.0002 of MAP@N in the AOL dataset. Additionally, from Table 4, the average clicked document for each query in the AOL dataset is 1.38. If N=100, a change in the final ranking list in just 0.05% of queries may not be considered a significant improvement. What makes the self-adaptive dynamic threshold method effective and different from other methods?
Overall assessment scores: 1