NEXTLOCLLM: NEXT LOCATION PREDICTION USING LLMS

Authors: SHUAI LIU, Ning Cao, Yile Chen, YUE JIANG, Gao Cong
Primary area: foundation or frontier models, including LLMs
Keywords: next location prediction, large language model, zero-shot

Abstract:
Next location prediction is a critical task in human mobility analysis and serves as a foundation for various downstream applications. Existing methods typically rely on discrete IDs to represent locations, which inherently overlook spatial relationships and cannot generalize across cities. In this paper, we propose NextLocLLM, which leverages the advantages of large language models (LLMs) in processing natural language descriptions and their strong generalization capabilities for next location prediction. Specifically, instead of using IDs, NextLocLLM encodes locations based on continuous spatial coordinates to better model spatial relationships. These coordinates are further normalized to enable robust cross-city generalization. Another highlight of NextlocLLM is its LLM-enhanced POI embeddings. It utilizes LLMs’ ability to encode each POI category’s natural language description into embeddings. These embeddings are then integrated via nonlinear projections to form this LLM-enhanced POI embeddings, effectively capturing locations’ functional attributes. Furthermore, task and data prompt prefix, together with trajectory embeddings, are incorporated as input for partly-frozen LLM backbone. NextLocLLM further introduces prediction retrieval module to ensure structural consistency in prediction. Experiments show that NextLocLLM outperforms existing models in next location prediction, excelling in both supervised and zero-shot settings.

Decision: desk reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730215441978

thread_type:
official_review

<thread_object>
creator: reviewer_arn2

<content>
summary:
This paper addresses a significant research question concerning model transferability across different cities and introduces NextLocLLM, which integrates large language models (LLMs) with next location prediction models. The method incorporates multi-dimensional trajectory content embeddings, LLM-enhanced POI embedding, an LLM backbone, and a prediction retrieval module, purportedly achieving state-of-the-art (SOTA) results in both fully supervised and zero-shot settings. Despite these claims, the experimental design, framework rationale, and methodological clarity are not convincing enough, leading me to lean towards rejection.

strengths:
1. The paper tackles the crucial issue of model transferability across urban settings, a topic of growing importance in location-based services.
2. This paper demonstrates improvements in performance, achieving state-of-the-art results in both fully supervised and zero-shot next location predictions.
3. This paper innovatively integrate multiple data sources, including points of interest (POI), textual descriptions, and trajectory data, enriching the model’s contextual understanding.

weaknesses:
1. The experiments are not solid. For example, 1) why using LLaMA2 and LLaMA3 does not outperform GPT2. 2) It seems that the proposed KD-Tree-based prediction contributes, but you did not involve it in the experiments. 3) In the ablation study, only using LLM-enhanced POI achieves the second-best result. It challenges the significance of the proposed methods. In addition, it is weird that using both POI and LORA will lead to worse performance than pure POI.

2. This framework seems not reasonable. This work aims to achieve generalization ability by LLM. However, I am wondering whether this ability can be achieved without road graph. It seems that this paper aim to realize zero-shot generalization with purely POI even without trajectory. 

3. The formalization is a little bit confusing. TI=\{(d,t)\} and t is time-of-day (0 ≤ t ≤ 23 in hours). Since Taxi data seem to be less than an hour. This formalization seems not complete.

questions:
Please explain the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533853268

thread_type:
official_review

<thread_object>
creator: reviewer_za3u

<content>
summary:
The paper proposes a novel LLM-based framework, NextLocLLM, for next-location prediction. By integrating trajectory and POI data with a fine-tuned LLM, the framework aims to enhance prediction accuracy. The primary components include spatial coordinate encoding for better spatial representation, LLM-enhanced POI embeddings capturing functional location attributes, and a prediction retrieval module to provide top-k predictions.

strengths:
• Innovative Framework: The use of LLM for next-location prediction is a key improvement, as it integrates semantic POI information, which is not fully leveraged by traditional methods.
•  Extensive Experiments: The authors present comprehensive experiments validating the model's performance across various datasets, demonstrating its robustness.
•  Clarity: The paper’s structure is well-organized, making the methodology clear and accessible.

weaknesses:
• Innovation Concerns: The core contribution, POI information embedding, appears somewhat incremental since it aligns with previous works' embedding logic but integrates this with LLM-based representation.
•  Contribution of KD-tree: The KD-tree application is commonplace in traffic scenarios, particularly with GNN-based models, and thus may not qualify as a novel contribution.
•  Model Architecture Innovation: There appears to be limited innovation in the model structure. I would appreciate a discussion with the authors to ensure I am not overlooking any novel aspects.
•  Case Studies in Experiments: The inclusion of case studies could bolster the practical applicability of the framework by demonstrating real-world scenarios.
•  Minor Errors: Minor errors, such as in Equation 9, should be corrected for accuracy.

questions:
Please refer to weakness. In addition, I hope the author can give an intuitive analysis of why the LLM-based model performs better. And what aspects of the design the author thinks are most effective, for example, the prompt prefix.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557471191

thread_type:
official_review

<thread_object>
creator: reviewer_xqud

<content>
summary:
This article proposes a novel method for next location prediction based on fine-tuning large language models.

strengths:
1. Well written, clear structure, easy to understand.

2. Proposed a series of key insights to enhance the adaptation of LLM in the next location prediction task.
   - Applying *Spatial Coordinate Normalization* to unleash LLM's cross-city generalization ability
   - Proposing *LLM-enhanced POI Embedding* to integrate diverse functional attributes of regions.
   - Predicting spatial coordinates as intermediate results, then using KD-tree retrieval for top-k locations, which is a **rather smart idea** that not only retain the generalization ability brought by coordinate prediction itself, but also is compatible with classic problem definition (Next **Location ID** prediction).

3. Conducted sufficient ablation study to demonstrate the importance of each module.

weaknesses:
1. The **model parameters** and **configurations** are **completely lacking** in introduction.
     - **Basic training parameters**: learning rate, batch size, optimizer, etc.
     - **Key hyperparameters**: the dimension of $d_{llm}$, etc.
     - **Model Input**: Length of historical and current trajectories.
     - **LLM Tokenizer and LLM Token Embedding Layer**: Which models/methods are used for?
  
2. **Lack of sensitivity analysis on hyperparameters**.
   Although the authors conducted a complete ablation study demonstrating the key roles of each module, it is still necessary to conduct corresponding sensitivity analysis on key parameters to demonstrate the robustness of the model.

3. The overlooked issue of **computational efficiency**. 
Inference efficiency of LLMs has always been a critical issue. The proposed model not only requires using *pre-trained LLM as the backbone network* but also needs to use *LLM encoding POI information and prompt prefix*. Therefore, I am very concerned about the computational efficiency issues of the proposed method, including training and inference time as well as resource utilization. I hope that the authors can report on this point and compare it with baselines. It is acceptable that the computational efficiency is not good for an ICLR paper, but it is better to discuss the drawback (if have)

questions:
Q1. Why is the NextlocLLM model labeled as a **white box model** in Figure 1(c)? I did not find any evidence in this paper to support this claim.

Q2. In the conclusion section, the authors mentioned that the *geographical distance error remains a challenge*, and maybe further reducing the grid size can mitigate this? The current study used a grid size of 500m x 500m for location prediction. I would like to know how grid size affects accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730603595410

thread_type:
official_review

<thread_object>
creator: reviewer_7tux

<content>
summary:
The paper presents NextLocLLM, a model designed to predict the next location by enhancing the extraction of spatial relationships and improving generalizability across different cities using a large language model (LLM) approach. First, NextLocLLM employs normalized spatial coordinates to represent discrete locations, accurately modeling spatial relationships and bypassing location ID inconsistencies across cities. Second, the model integrates LLM-enhanced POI category information, which captures functional attributes of locations more effectively. Finally, NextLocLLM leverages a KD-tree to convert output coordinates into the top-k most likely predicted locations, thereby incorporating neighborhood spatial relationships into the prediction process.

strengths:
S1. Unlike other methods that rely solely on location IDs for prompt design, NextLocLLM focuses on spatial relationships and semantic embeddings derived from natural language descriptions of POI categories. By using spatial coordinates, the model gains a deeper understanding of spatial relationships between locations, enhancing transferability and generalization across diverse urban environments.
S2. The authors conduct extensive experiments with various models and baselines. These experiments are comprehensive, covering four datasets and demonstrating strong performance in both supervised and zero-shot settings.
S3. The paper is well-written and easy to follow.

weaknesses:
W1.  Ablation studies to assess whether using spatial coordinates provides a clear advantage over location IDs is missing.
2. The paper does not include a detailed description of raw trajectory processing. For instance, there is no clarification on determining the length of historical and current trajectories or whether noisy points in the raw trajectory data are filtered out.
3. The paper could benefit from further analysis on which specific design modules contribute to the zero-shot capability of NextLocLLM and why it outperforms other LLM-based methods.

questions:
Q1. How is the frequency of each POI category within a location calculated?
Q2. In fully-supervised scenarios, such as for training methods like DeepMove, is the same data used as with NextLocLLM? Additionally, are the POI category and stay duration included as training data for DeepMove?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732113192263

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response To All Reviewers

comment:
We sincerely thank all reviewers for their insightful and constructive feedback, which significantly contributed to improving the quality of our paper.  We are pleased to see the reviewers recognize our work's clarity and readability (Reviewer 7tUX, Reviewer xQuD, Reviewer za3U). We also appreciate their acknowledgement of our model's strong transferability and generalization capabilities (Reviewer 7tUX, Reviewer arN2). Furthermore, we are delighted that the reviewers find our experimental evaluations extensive and sufficient (Reviewer 7tUX, Reviewer xQuD, Reviewer za3U) and view our proposed idea as smart and impactful (Reviewer xQuD, Reviewer za3U, Reviewer arN2).

The major revisions and new experiments conducted in response to the reviewers' comments are summarized below:
+ We include new experiments comparing the fully-supervised and zero-shot performance of our model when using spatial coordinates versus location IDs as inputs. 
+ We add a case study to highlight the practical advantages of NextLocLLM in cross-city tasks, demonstrating its real-world applicability.
+ We append a comprehensive list of model parameters and configurations to ensure improved reproducibility.
+ We conduct a detailed sensitivity analysis of the key hyperparameters to evaluate the robustness of the model's performance.
+ We conduct additional experiments to analyze training and inference times.
+ We conduct experiments where the LLM in our framework was replaced with a randomly initialized Transformer model, showcasing the critical role of the LLM in our approach.
+ We conduct experiments to analyze how varying grid resolutions affect the model's performance
+ We included new ablation studies on various components of the framework, focusing on their contributions to zero-shot performance.
+ We answer the concerns raised by reviewers.
+ We revised our paper to account for all changes, and the revisions are highlighted in blue color in the revised manuscript for your convenience.

We hope that our answers and revision of our paper will address your concerns.
</content>
</thread_object>

</thread>


Metareview: