Mobility-LLM: Learning Visiting Intentions and Travel Preference from Human Mobility Data with Large Language Models
Authors: Letian Gong
Yan Lin
Xinyue Zhang
Yiwen Lu
Xuedi Han
Yichen Liu
Shengnan Guo
Youfang Lin
Huaiyu Wan
Venue: NeurIPS 2024 poster (2024)
Keywords: spatial-temporal data mining
location-based service
check-in sequence
large language model
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: We propose a novel unified framework that reprograms the check-in sequence to let LLMs comprehensively understand human visiting intentions and their travel preferences

Abstract: Location-based services (LBS) have accumulated extensive human mobility data on diverse behaviors through check-in sequences. These sequences offer valuable insights into users’ intentions and preferences. Yet, existing models analyzing check-in sequences fail to consider the semantics contained in these sequences, which closely reflect human visiting intentions and travel preferences, leading to an incomplete comprehension. Drawing inspiration from the exceptional semantic understanding and contextual information processing capabilities of large language models (LLMs) across various domains, we present Mobility-LLM, a novel framework that leverages LLMs to analyze check-in sequences for multiple tasks. Since LLMs cannot directly interpret check-ins, we reprogram these sequences to help LLMs comprehensively understand the semantics of human visiting intentions and travel preferences. 
Specifically, we introduce a visiting intention memory network (VIMN) to capture the visiting intentions at each record, along with a shared pool of human travel preference prompts (HTPP) to guide the LLM in understanding users’ travel preferences. These components enhance the model’s ability to extract and leverage semantic information from human mobility data effectively. Extensive experiments on four benchmark datasets and three downstream tasks demonstrate that our approach significantly outperforms existing models, underscoring the effectiveness of Mobility-LLM in advancing our understanding of human mobility data within LBS contexts.

Mobility-LLM: Learning Visiting Intentions and Travel Preferences from Human Mobility Data with Large Language Models
Letian Gong! 12+ Yan Lin'; Xinyue Zhang! Yiwen Lu', Xuedi Han' Yichen Liu 1.2 Shengnan Guo 12f Youfang Lin Huaiyu Wan 1.2
School of Computer Science and Technology, Beijing Jiaotong University, China Beijing Key Laboratory of Traffic Data Analysis and Mining, Beijing, China "Department of Computer Science, Aalborg University, Denmark {gonglt, zhangxinyue, luyiwen, hanxuedi, liuyichen, guoshn, yflin, hywan) edu. cn, lyan@cs aau. dk
Abstract
Location-based services (LBS) have accumulated extensive human mobility data on diverse behaviors through check-in sequences. These sequences offer valuable insights into users' intentions and preferences. Yet, existing models analyzing check-in sequences fail to consider the semantics contained in these sequences, which closely reflect human visiting intentions and travel preferences, leading incomplete comprehension. Drawing inspiration from the exceptional semantic understanding and contextual information processing capabilities of large language models (LLMs) across various domains, we present Mobility-LIIMM novel frame- work that leverages LLMS analyze check-in sequences for multiple tasks. Since LLMs cannot directly interpret check-ins, we reprogram these sequences to help LLMs comprehensively understand the semantics of human visiting intentions and travel preferences. Specifically, we introduce a visiting intention memory network (VIMN) to capture the visiting intentions at each record, along with shared pool of human travel preference prompts (HTPP) to guide the LLM in understanding users' travel preferences. These components enhance the model's ability to extract and leverage semantic information from human mobility data effectively. Extensive experiments on four benchmark datasets and three downstream tasks demonstrate that approach significantly outperforms existing models, underscoring the effectiveness Mobility-LLLM advancing our understanding of human mobility data within LBS contexts.
1 Introduction
Location-based services (LBS) such as Gowalla, Weeplace, and Foursquare enable users to share and discover location information and nearby services. This results in the collection of extensive human mobility data, often presented in the form of check-in sequences. These sequences record users' visits to different points interest (POIs) like restaurants and hospitals at various times, reflecting significant semantics about their intentions and preferences. Analyzing these check-in sequences crucial as offers valuable information on human mobility data, which can positively impact individuals, businesses, and urban management.
Both authors contributed equally this rescarch. i Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

The key to effectively mining check-in sequences lies in understanding their rich semantics. Ex- isting methods primarily focus on specific tasks, such as location prediction [O] 59 52 26), time
prediction 40 45 and trajectory user linking 34 13 62 rather than delving into the semantics of human behaviors. This narrow focus often results limited optimization goals and a shallow understanding of the semantics contained in check-in sequences. Recently, large language models (LLMs) have demonstrated impressive capabilities in semantic understanding and contextual infor- mation processing, demonstrating successful adaptability across different domains. LLMs trained on extensive corpora surpass task-specific models in their potential to understand semantic information. Inspired by this, we aim utilize pre- mre-trained LLMs powerful check-in sequence learners.
Nevertheless, LLMs encounter a significant obstacle in their inability directly interpret check-in sequences. As typical sequential data, check-in sequences contain a wealth of semantic information that reflects various near-term regularities and inherent characteristics. The future intention of an individual is prone to be dictated by near-term regularities that are close to recent visits, termed visiting intentions. Furthermore, an individual's inherent characteristics tend to persist over time and determine their travel preferences, which is necessary to analyze them across multiple domains for comprehensive understanding. Hence, our main challenge is to enable LLMs to effectively extract semantics from check-in sequences and comprehensively understand human visiting intentions and travel preferences.
To address this challenge, we present a novel unified framework called Mobility-LLMN for various check-in sequence analysis tasks. It leverages pre-trained LLMs for general check-in sequence analysis. Our contributions can be summarized as follows:
We propose a unified framework called Mobility- that a pre-trained LLM achieve a SOTA or comparable performance across various check-in analysis tasks including location prediction, trajectory user link, and time prediction. We extract the semantics of check-in sequences to enable LLMs to gain a comprchensive understanding of human visiting intentions and travel preferences.
A visiting intention memory network (VIMN) is proposed for capturing users' visiting intentions users at each check-in record by prioritizing relevant check-in in records.
A shared pool of human travel preference prompts (HTPP) in different domains is introduced, which enables comprehensive understanding of human travel preferences and matches appropri- ate prompts from multiple domains.
Our model's exceptional performance is validated through extensive experiments four bench- mark datasets involving three tasks. Our robust outcomes in cross-domain pre-training exhibit an average enhancement of 17.8% and an average of 23.6% to 38.3% on the few-shot scenario.
2 Related Works
In this section, we provide short reviews literature in the areas of mobility data mining and cross- domain applications of LLMs. We postpone the detailed discussion of works to the Appendix A due to limited space.
Mobility Data Mining has emerged as promising research area due to the proliferation of location- based services. This has led to the development of three significant tasks that enhance service quality: next location prediction (LP), next time prediction (TP), and trajectory user link (TUL). The LP task aims to anticipate a user's future location based on their historical movement. Several notable models have emerged as leading approaches in IP [10 [5 [I on establishing connections between different trajectories and users, facilitating the analysis of user movement patterns and uncovering valuable insights into their behavior [3 7 60 16]. The TP task focuses on estimating the time at which a user is likely to visit their next location. Various models have been developed to model the intensity function representing the rate or density of event
48
5 23 58) The TUL task focuses
occurrences, effectively making accurate time predictions [57 17 53
Cross-domain Application of LLMs has gained attention in recent studies, which adopt large language models to address the challenge limited training data. the field of time series analysis, LLM4TS 4 is pioneering method that aligns pre-trained large language models with temporal characteristics, introducing a two-level aggregation method to effectively incorporate multi-scale temporal data into pre-trained LLMs. One-Fits-All [63] is unified framework that leverages a frozen

---

pre-trained language model to attain state-of-the or comparable performance across various major types of time series analysis tasks. AutoTimes [30] facilitates the tokenization of time series into the embedding space of L.MSs and intelligently utilizes the inherent token transitions to effectively predict time series in an autoregressive manner. the field of computer vision, L.4VVE [35] incorporates frozen transformer block from LLM as general-purpose visual encoder layer. To tackle graph-related tasks, TAPE [19] utilizes semantic knowledge generated by LLMs to enhance the quality of initial node embeddings in GNNs. MoleculeSTM [27] aligns GNNs and LLMs within a shared vector space, integrating textual knowledge into graphs to enhance reasoning capabilities.
3 Preliminaries
POI Visiting Record In the check-in datasets, a user's visit to a certain place is represented by a POI visiting record R = (T.p: 1) generated by the user Lp indicates the visited POI at time 1. I,p is represented by (Tea, Ltoo; Liat; Iocategory); comprising Lud as POI index, and accurate longitude Lion and latitude Liat. Leategory denotes the category of the visited POI (e.g., hospital, restaurant).
Check-in Sequence A user's movement during specific period can be represented by sequential POI visiting records, which we refer to as check-in sequence. We denote a check-in sequence as = R1, Ry, Rn 2, where the POI visit records are ordered by their visited time, and n the length of the sequence.
Problem Statement Given a check-in sequence C, our objective is encode this sequence into meaningful representation. This representation can be used for various tasks. In this paper, we choose three typical check-in prediction tasks: Identifying the user á who generated the check-in sequence (TUL task). 2) Predicting the next location 4n+1 the user will arrive at (I.P task). 3) Predicting the arrival time +1+1 at this location (TP task).
4 Methodology
We introduce a novel unified framework, Mobility-LLM, designed to address variety of check-in sequence tasks. The overall structure of this framework is illustrated in Fig. [ 1) Initially, to embed POIs by incorporating category semantics, we introduce the POI Point-wise Embedding Layer (PPEL) to generate the embedding of POIs in the current check-in record (referred to as PPE). 2) Subsequently, we feed the PPEs and timestamps of check-in sequence into the Visiting Intention Memory Network (VIMN) capture the visiting intentions of users at each check-in record. 3) Human Travel Preference Prompt (HTPP) pool is introduced to extract users' preferences from check-in sequences, which more effectively. 4) Finally, we use the different parts outputs of LLM (corresponding to VIMN, HTPP), each with its own projection head, forecast the user's location, the estimated arrival time, and the user who generates the check-in sequence.
cues assist the LLM comprehending users' travel preferences
POI Point-wise Embedding Layer (PPEL) is designed to generate the semantic information embedding for each POI a check-in record. This is especially important since POI categories contain wealth of semantic information. It has been observed that the category descriptions of POIs in the original check-in datasets are often vague or unidentified. When we refer to "vague," we indicating that the deseriptions are too broad or contain abbreviations, making it challenging to determine the specific POI categories accurately. To address this issue, we have developed a category word pool (see Appendix for the whole categories list) that allows each POI match with the most appropriate categories automatically. We reindex the POI IDs and word IDs in each dataset, making it convenient for subsequent encoding of these IDs. As shown in Fig. [T, we the learnable embedding EL., of each POI ID as the query, the learnable embedding Ec,a of the category word ID as the key, and the corresponding category word token Ec from the LLM tokenizer the value. A Point-wise attention mechanism is used to calculate the i-th PPE 8; of POIs:
8; Softmax Que ELL Koy (ey.))..) Vall(E + GcoHash(Lion. Ltat),
Vd
where Que(-), Key()) (+), and Val(.) denote linear projections, d the dimension of FEG.ser GeoHash(..)) encodes geographic coordinates (latitude and longitude) into embedding vec-
3

---

Tasks
User Link B
Location Prediction
Su1z
Predaction
Time
Proves
Emoalding
Vieting
(6) LLM Partially-rmeent LORA
Addrice
Cencat
Traning
Mean Pooling Head
Multiclass Head
TPP Head
Ponpa
FDI Parz
(6g, R2 dg... By
18
1
Trandienaa Blads
LoRA
(d) LLM (Partially-Free Frozen
+ LoRA
(b)
Food Forward
Traasfomet Blacs
Sva
VINN
Add Norn
Food Furward Layer
(b)
VIMN
(c) HTPP
Midittions
Attention
Activo
Commininn
(a)
PPEL
White
Smeetool
Canegoes hompital
Infommation
POL Informstiinn
Stabeet
W
para
Check-in
Sequence
Occupation
Lifestyle
Activity
Picy
codlee
User 41
User 42
PDT ID
Teletion:
Tiepuc [H.U
Figure 1: The overall of Mobility- framework. POI Point-wise Embedding Layer (PPEL). b) Visiting Intention Memory Network (VIMN), c) Human Travel Preference Prompt (HTPP). d) a denotes the output of the LLM corresponding VIMN (i.e. first n output of the LLM), while the remaining outputs are denoted as 3.
tor (see Appendix H for details). We omit the MLP Layer and Layer Normalizations in Transformer layers simplify the representation in Eq. [
Visiting Intention Memory Network (VIMN) is proposed to capture the visiting intentions of users each check-in record by prioritizing relevant check-in records. As shown in Fig. Dp, feed the timestamp of each check-in record and the time interval between the adjacent records into the Imminent GRU layer. We adopt dual encoding approach for time representation: 1) Periodic encoding for timestamps t as (t)) = [cos(w) t), sin(wit),.................... cos(w,tt), sin(w:t)) where {wie) ) are frequencies determined to capture periodicity across various temporal scales. 2) Logarithmic encoding for time intervals At represented as AT = log(1 + At), which adjusts the GRU's ' forget gate based on the time interval At. This adjusted factor, denoted as AT, affects the forget gate of the GRU unit through Grorger ( AT) = a oW (W,AT AT + by). The other components like Gupdate remain unchanged from the original GRU configuration. The Imminent GRU layer be depicted as:
2j = (W.., T(1.) + Gupdate (1;-1) Grorget (AT)),
where z; represents the output of the Imminent GRU at time step 4, and H4-1 denotes the hidden state at time step 1. This setup allows for filtering out less relevant, temporally distant data. Subsequently, the outputs [z--++1,-- 2i-1, 2i] from the most recent r cycles of the Imminent GRU, along with the latest PPEs, are forwarded the feed-forward layer [44] (further elaborated in Appendix G to refine the representation of the user's visiting intentions h.
Travel preference Prompt Pool
Significant Score
Commuting, Mealtime, Exercise, Studying. Working, Shopping
Activity
Student, Doctor, Teacher, Engineer, Lawyer, Nurse, Programmer
Occupation
malchinaliiin Function
Top- XC
Active, Healthy, Unhealthy, Luxurious, Urhan, Rural, Leisure
Lifestyle
h h
Figure 2: The architecture of HTPP.
Human Travel Preference Prompt (HTPP). This paragraph introduces a method for extracting
users' travel preferences help the LLM enable a comprehensive understanding of human travel preferences and match appropriate prompts from multiple domains. Prompting strategies have demonstrated encouraging outcomes in various applications that aid model predictions [3]. Previous works primarily focus utilizing a fixed prompt to boost the pre-trained models" performance
through fine-tuning (30 or learnable prompts lacking reality semantic meaning
41 3]. User behavior
is diverse and cannot be accurately summarized single fixed prompt to describe the user's travel

---

preferences. Adopting prompt- based approach with meaningless learnable prompt vectors can only handle sequences with simple semantics, such as time series 4.3. However, it lacks the capability to adequately extract human travel preferences. To address this limitation, we introduce shared pool of travel preference prompt words across D domains (e.g., occupation, activity type, lifestyle) as illustrated in Fig. 2 Each domain includes a selection of m = 16 prompt words. For every prompt word, we create a key-value pair where the value represents the word token obtained from the LLM tokenizer, and the key is derived from the word token through a trainable linear transformation. Specifically, these key-value pairs are defined as follows:
P [(Kk, VP). (R2 V.P),... (Kip V2) D E (1,2,3),
where km, VD m E TR'E is key-value pair, and we maintained it at the same embedding size L as visiting intentions h;- We then employ a scoring function o(h,, km)) = h; k /m:l| || to determine the significance score of each h,, where each h, indicates the relevance of the current visiting intention to each word in the travel preference prompts pool. Subsequently, we aggregate the significant scores of each h; identify the top-1C most significant pairs a domain. By defining { as subset of indices for the selected top-IC prompt words in each domain, obtain the output [V,,;; :... V.Rk of HTPP prompts (illustrated in Fig. (T)) to more accurately recognize and comprehensively understand user's travel preferences at the level of individual check-in sequences.
RLE x RLEE - RR. The significant score of
Training Task In Fig. [d. the input to the LLMs consists of the output of VIMN H [h, h2 ... kn). HTPP, and the user embedding U, denoted as Timput
Timput U, VD, Vine D (1,
We concatenate all the tokens along the check-in length dimension. As shown in Fig. De. o denotes the output of the LLM corresponding VIMN (i.e., the first output of the LLM), while the remaining outputs are denoted as 3, where n is the length of this check-in sequence. Various projection heads are used to predict the next location 8 the user will arrive at and its arrival time f with the B. We use the B and with a mean pooling projection head to predict the user à who generates this check-in sequence.
5 Experiments
To evaluate the performance of our proposed model, we carry out extensive experiments on four real-world check-in sequence datasets, targeting three different types of downstream tasks: Next Location Prediction (LPP), Trajectory User Link (TUL), and Time Prediction (TP). We use TinyLlama- IB (56) as the default backbone unless stated otherwise. The code of Mobility-LLM released at wwt.e/oo//eennnnn////aa
Baselines: For the LP task, cover five state-of of- art LP models to demonstrate the superiority
of our model: DeepMove [00], LightMove (22). PLSPL [47], HMT-LSTM (2). LSTPM (59
For the TUL task, select four end-to-end models for comparison: TULER [3], TULVAE 62
MoveSim [T] S2TUL 77.. For time prediction methods, we select four SOTA models for comparison:
IFLTTP 40 THP 64 NSTPP 117. DSTPP (53) NSTPP and DSTPP can also be applied to the LP
task. For sequence representation methods ReMVC (55) VaSCL [54] SML 61. CACSR [13, we
apply them to learn the representation of the check-in sequence and serve different downstream tasks.
More details are in Appendix D
Datasets: In our experiments, we use four real-world datasets derived from Gowalla WeePlace 28
B1I, Brightkite [ and FourSquare 49 50]. To ensure data consistency, we set a maximum historical time limit of 120 days and filter out users with fewer than 10 records and places visited fewer than 10 times. Appendix provides statistical information for each processed dataset. We shuffle all the samples and then split the datasets into training, validation, and test sets in 6:2:2 ratio based on the number of samples.
thtpss:/snap.ssantioii http:///sunssaattooo....ddddaees wttp:///aaaaaoooooooaaaaao
5

---

Implement Details: During the training phase, we adopt a partially frozen strategy to fine-tune the pre-trained LLM. We apply different parameter freezing strategies to the Lp layers and the IF LF+U layers. In order to enhance the model's performance on trajectory data, we employ the Low-Rank Adaptation (LoRA) algorithm [21] to incorporate additional parameters into the L.LMs.
Details of the partially frozen strategy and other settings can be found in Appendix B
We run each
set of experiments times and reported their mean values
5.1
Next Location Prediction
Table I: Next location prediction (LP) performance results. A higher value indicates better perfor- mance. Red: the best, Blue; the second best. The units of all metics are expressed as c-2.
Datasess
Gowalla
WeePlace
Brightkite
SourSquare
Metric (6-2)
Mothod
Acc I Acc Acco MRR Acc@1
Acc Acc@
MRR
Acc @
Acc@5 Acc 20
Acc @ Acc Acc MRR
DeepMove
10.51
23.21
31.9
16.65
19.23
37.79
52.61
27.03
49.82
66.
71.19
56.94
16.3
35.06
48.39
25.02
LightMove
9.88
20.93
29.95
15.01
18:33
36.37
52.75
27.08
63.11
68.94
55.46
13:27
29.33
20.71
PLSPI.
11.26
24.12
33.82
17
18.77
37.31
53.31
27.86
51.42
65.34
71.46
57.59
13:73
30.65
43.18
21.42
HMI-LSTM
10.73
22.41
32.77
16.47
17.29
34.23
49.82
25.69
49.22
63.36
67.9
55.41
13:67
29.9
42.6
21.17
LSTPM
9.83
20.88
15.12
31.15
45.98
42.58
54.65
60.21
15.46
34.17
48,49
24.27
VaSCL
11:47
22.17
32.92
16.56
18.11
37.54
$3.04
28. 46
49.95
66.
71.17
57.23
14.99
32.84
47.06
23.39
SinCSE
7.15
16.12
23.8
11.53
14.12
31.06
46.89
22.31
46.65
64.48
70.43
54.89
14.5
33,13
47,89
23.29
NSTPP
10.81
23.23
32.94
16.87
16.58
32.37
47,82
24.2
45.83
58.44
64.56
52.16
14.89
33.18
47,03
23.36
DSTPP
10.85
23.11
33.19
16.74
18.85
37.68
$3.44
27.52
48.71
62.82
67.71
55.26
13.3
29.11
41,53
20.66
RcMVC
11.03
22.94
35.92
$1.93
49,57
63.58
16.92
36.05
26.02
SML
9.92
20.91
30.36
15:25
17.42
35,23
$1.07
25. 96
46.26
58.93
65.35
51.97
14.72
32,54
46,87
23.27
CACSR
10.94
18.22
26.56
12.83
19.66
36.46
51.25
28.15
44.56
62.01
65.91
51.91
14.73
31,54
46.47
22.78
Mobility-LLMM
11.87
25.14
36.6
18.29
20.47
39.22
56.69
29.21
53.18
68.31
74.11
59.89
17.29
37.17
53.16
26.47
Table 2: Trajectory user link (TUL) performance results. A higher value indicates better performance. Red: the best, Blue: the second best. The units of all metics are expressed as c-2.
Datasets
Gowalla
WeePlace
Brightkite
Foure
Metric (e-2)
Method
Acc@1 Acc Acc 20 MRR
Acc
Acc@5 Acc0 20
MRR
Accis Ace@ 20RR
Acc
Acc
Acc MRR
TULER
55.85
64.55
72.19
60.09
63.69
79.24
86.4
72.37
58.37
73.48
83.36
65.26
44.43
58.41
69.14
51.04
TULVAE
41.33
43
42.22
39.02
44.87
47.42
54.99
21.7
26.52
24.44
MoveSim
46.5
59.42
68.21
52.66
57.07
70.49
79.13
63.42
60.19
70.2
78.52
65.04
37.48
50.65
61.57
44.02
S2TUL
59.33
67.78
67.07
61.2
52.82
55.09
57.43
51.77
39.54
44.46
44.73
48.7
37.97
45.08
43.42
44.55
VaSCL
59.9
68.34
63.99
83.25
88.64
78.42
63.88
67.67
68.67
51.44
61.22
SimCSE
26.78
43.63
57.76
35.02
55.58
71.74
82.56
63.34
59.96
71.11
79.9
65.39
40.1
53.15
65.54
46.78
ReMVC
68.75
74.4
73.19
70.02
65.78
73.09
71.64
66.15
73.85
82.55
87.93
77.93
58.18
66.84
72.67
65.14
SML
57.42
62.44
63.07
61.97
59.44
69.93
69.52
62.43
63.91
72.28
70.52
66.38
55.69
58.42
63.7
57.7
CACSR
52.62
63.5
71.29
57.84
70.01
81.3
86.98
75.24
58.6
72.54
79.72
65.18
51.89
64.59
72.91
57.98
Mobilit-LLLM
80.43
86.29
83.18
88.04
91.48
83.21
88.52
90.35
85.73
72.08
79.67
75.71
Setups: We consider the LP task as a multi-classification problem. Given a check-in sequence C. from specific user Uù, feed it to framework GLLM to obtain the check-in sequence representation GLLM((
As shown in Fig. De. a multi-class projection head fo (3)
softmax (W B + b)) is used to predict the next location 8 with the corresponding output B of VIMN. We maximize the conditional log-likelihood for a given N observations as follows: LMLE () = Σm log frrulti
(3), where N is the number of POIs. We maintain all baseline user embeddings at 256 dimensions and POI embeddings 128 dimensions. The evaluation metrics include Acc @k and mean reciprocal rank (MRR). The details of the implementation and metric can be found in Appendix [..2
Results: Our brief results are shown in Tab. I and consistently surpass all baselines. The comparison with CACSR is particularly noteworthy since it is the latest check-in sequence learning model. We note average performance gains of 17.19% and 7.49% over CACSR and ReMVC, respectively. Compared with the SOTA task-specific models, PLSPL and LSTPM realized an average MRR improvement of 9.32 % and 19.88%. Relative to the time point process models, e.g., NSTPP and DSTPP, our i improvements are also pronounced, exceeding 14.86% % and 13.22%.
5.2 Trajectory User Link
Setups: Unlike the LP task, the TUL. task requires predicting which user generated a given check-in sequence. Therefore, the input information cannot contain any details about the user. We use the

---

Table 3: Time Prediction (TP) preference results. A lower value indicates better performance. Red:
the best, Blue: the second best. The units of all metrics are minutes.
Method
Mobiltty-LLMM
IFLTIP
NSTPP
DSTPP
ReMVC
CACSR
Darasets
MAE
RMSE
MAE
RMSE
MAE
RMSE]
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
Gowalla
353,89
509.55
369.82
522.83
154.6
534.52
362.39
532.99
368.76
524.33
360.62
531 46
362.03
515.16]
356.37
518.22
WeePlace
28.28
35.54
29.31
36.96
29.47
36.11
29.41
37.17
28.82
37.28
29.61
36.11
28.34
36.04
28.42
35.97
Brightkite
246.44
423.26
362.72
46
354.41
433.84
345.74
435.11
358.91
427 49
348
86
440.61
348.86
457.66
358.91
440.61
FourSquare
309.78
505.03
314.74
503.01
314.52
513.11
319.39
521.69
317.84
524.22
319.39
515.13
318.46
523.71
315.67
513.11
Table 4: Few-shot learning on 5% training data. Red: the best, Blue: the second best. We keep the
same protocol with the other settings. The results on all training datasets are in Tab. 1 and Tab. The results of few-shot on 1% and 20% can be found in Appendix
Datasets
Gowalla
WeePlace
Brightkite
FourSquare
Metric
Acc A
Acci MRRR
Acce! Acc Accil 20 MRR
Acc@I 1 Acc AccRR0MRR Acc Acces Acc
Tash
Method
DeepMove
7.58
18.01
26.29
12.58
15.47
32,89
46.85
23.35
42.25
60:56
61.72
50.62
14.44
32.26
44.34
22.73
LighMoove
7.12
16.24
23.23
11:34
15.17
52.8
47.59
23.78
41.55
57,69
59.76
49.3
12.9
28.3
39.15
20.1
8.27
13.17
43.61
61.95
46.56
25.32
HMT-LSTM
7.74
17.39
25,41
12:44
14.02
30,66
45.48
22.43
41.74
57.92
58.92
49.26
13.26
29.07
40.16
20.6
LSTPM
7.09
16.2
23,46
11.42
15.82
31.73
45.64
24.32
36.11
49 96
52.2
42.81
14.99
33, 22
45.71
23.61
VaSCL
24.59
44.37
LP
SimCSE
5.16
12.51
18.46
8.71
11.36
27.03
41.75
19.27
39.56
58.94
61.06
48.0 68.62
14.06
32.21
45.15
22.56
NSTPP
7.79
18:02
25.55
12.74
13.34
28.17
42.58
20.91
38,86
53.42
55.97
46
15.81
34.09
45.62
24.34
DSTPP
23.35
58.7
49.
39.08
ReMVC
7.95
17.8
25.89
12. 18.58
15.1
32.47
47.47
24.07
42,04
58.12
60:06
50.06
13:32
29. 8
40.71
20184
SML
7.15
16.22
23.54
11.52
13.91
29.79
44.36
22.19
39.23
53.87
56.65
46.2
14.28
31.64
44.19
22.64
7,89
9.69
27.11
40.94
20.
37.79
46.
43.81
Mobility-LILM
9.98
21.82
32.02
1.5.74
18:277
346.99
53.23
27. 17.15
48.44
65.33
68.66
55.64
16.86
36.47
51.36
25.98
TULER
26.32
40.62
52,43
31.6 62
44.62
59.14
70.72
52.95
49.67
65.38
76.97
56.71
22.29
36.6
50.7
29.4
TULVAE
27.24
43.36
55.17
52.58
63.22
56.58
47.35
29
9.94
Movesim
12.28
27.73
41.55
19.17
40.61
53.33
63.86
46.16
41.
54.05
67.93
47.21
29.9
41.88
$2.98
24.69
S2TUL
32.13
46.84
53,43
17:93
35.88
41.55
46.56
37,16
26.01
34.32
38.15
35.96
27.61
36.3
46.74
35.5
TUL
SimCSE
VaSCL
18.95
21.2
26.76
36.02
31,46
19.89
27.15
21.50 59
37.89
22.19
$3.88
28.26
66.82
46.28
26.42
41,45
40,75
56.68
57.2
73.66
69.7
47.58
46.6
26.29
19.7
32.78
40.4
48.66
24.89
31
ReMVC
27.76
42.29
48.14
33.63
46.08
55.73
58.49
47.89
43.38
52.84
58.61
47.55
18.85
31.63
45.11
24.8
SML
26.78
39.45
45.19
34
40.53
53,91
57,55
45.63
43,56
56.31
62.53
49.68
18.42
27.95
31,59
33.25
CACSR
24.67
39.52
50.7
31.36
48.85
61.46
72.04
55.73
39.83
57.24
69.76
47.51
24.95
38.3
50.28
32.37
Mobility
55.75
69.25
76.45
61.97
66.82
77.65
83,57
71.94
69.65
79.28
85.12
74.21
51,75
64.01
73.04
$7.66
corresponding output 3 of HTPP and a with a mean pooling projection head to predict the user â who generates this check-in sequence (TUL task). Other settings and evaluation metrics are the same as those used in the IP task.
Mobility-LLMM consistently surpasses all baselines in Tab. 2 outperforming the all baselines by an average of 47.3%. Mobility-LLLM remains competitive even when compared with the SOTA model, ReMVC, by 17.17%. Our model performs exceptionally well in the TUL task thanks to its effective extraction of users' travel preferences, allowing for precise identification of users.
Results:
5.3 Time Prediction
Setups: For the time prediction task, we follow the method of IFLTPP 40 method to model the interaction time as mixture distribution. We first obtain the mixture weights w, means μ and standard deviations from the corresponding output 3 of LLM with linear layer. Then we use the TPP projection head built by a mixed distribution function and sample to get the prediction time int1 as follows:
using an intensity-free
p (T W. 8)
Σ
1
exp
(log 22)2
Σ wk exp (apa b+
a²sk 2
(5)
V 2T
2sh² 2
where l represents the number of independent Gaussian distributions in the mixed distribution, a denotes the mean of the whole set and b denotes the standard deviation of the whole set. We sample from the mixture model in the parsing solution. The evaluation metrics include root mean square error (RMSE) and mean absolute error (MAE).
Results: Our brief results are shown in Tab. 3 Due to the diversity of user behaviors and the unpredictability of activity timing, even the ttate-of-the-art (SOTA) models for Temporal Point
7

---

Table 5: Ablations Gowalla dataset in all tasks. Red: the best, Blue: the second best. Our full results can found in Appendix E.3.1 The setting of A.1-8 can be found Appendix E. and the settings of B.1-4 can be found in Appendix E.2
Tasks
LP
TUL
TP
Variant
Metic
Acc
Acc
Acc
MRR
Azc
Acc
Acc
MRR
MAE
RMSE
TinyLian Liaya (Default)
11.87
353.89
509:55
A.2 Tinyl TTlyLLLaaa-Caaa Lama-Chat
11.59
24.33
35.76
17.88
71.83
79.37
83.38
75.38
356.34
513.76
A.3 LiteLlama
11.56
25.39
35.78
18.41
80.53
85.11
86.12
81.11
354.71
496.36
phi-2
512.28
pythis-700 70M
11.03
21.86
35.74
17.91
79:47
85.51
86.18
356.71
513.15
A.6 pythin-
11.19
24.84
35.82
18.01
78.
84.91
58:46
80.92
354.21
$11.04
pythia 2:8B
11.76
25.14
36.11
18.02
79.63
64.32
$7.24
80.95
354,57
512.59
GPT-2
353.85
B.1 wio HTPP
11.35
24.29
33.62
17.38
72.02
79.64
80.45
75.17
361.08
$17.35
B.2 wio VIMN
11.01
23.27
34.03
16.83
15.68
81.79
83.72
78.06
355.65
517.26
wil
513.12
wlo
366.84
Epoch 0
Epoch
Epoch 5
Epoch 20
Oonain
Behavioral
significance particular sample varies 16 different behavioral prompts within three domains the HTPP module.
Sigvificant Bahavioral Prompe Less Signfficam
Behuvioral Prompt
30 miin
Mr. 115
Mt 350
Mr 15B
Time
pm
11,00 pm
am
an
(b) wio HTPP
w/ HTPP
(d) w/o VIMN
(e) w/o Deltà time
w/ VIMN
Delta time
Figure 3: Showcases of the HTPP and VIMN.
Processes (TPP) struggle achieve excellent results across all datasets. However, model outperforms all baselines in most cases and does so significantly for most of them.
5.4 Few-shot Forecasting
Setups: LLMs have recently demonstrated remarkable few-shot learning capabilities [29]. This section assesses whether our reprogrammed LLM retains this ability in different tasks. In our experiments, maintain consistent splits for training, validation, and test in both standard learning (where the full training set is used) and few-shot learning. For few-shot scenarios, we intentionally limit the training data percentage of sample number (i.c., using first 20%, 5%, 1% samples of the training dataset).
Our 5% few-shot learning results in Tab. remarkably excel over all baseline methods, and we attribute this to the successful knowledge activation our reprogrammed LLM. Interestingly, approach both LP and TUL tasks consistently surpasses other competitive baselines, further underscoring the potential prowess language models as proficient human behavior analysis ma- chines. Concerning recent SOTA models such NSTPP, DSTPP, S2TUL, ReMVC, and CACSR, our average enhancements surpass 21.4% %. 21.7% % 86.6% %. 46.2 %. and 45.2% w.r.t. average on all the metrics. Even with only 5% of the training dataset,
Results:
model achieves results comparable other models using 100% of the training dataset. This is particularly significant for srivacy-protectee and typically smaller Check-in datasets, as our model can effectively understand the distribution patterns of human behaviors with minimal data.
5.5 Model Analysis
Language Model Variants. We compare eight representative backbones with varying capacities (A.1-8 in Tab. 5). We find that the TinyLlama (A.1 in Tab. backbone model performed the best overall for our task, while Chat version (A.2) is relatively suitable reprogramming. Our results indicate that the scaling law is not strictly retained after the LLM reprogramming. Even

---

(%6
Gowalla- TUL
(15) (%)
WeePlace- TUL
(%)
(%)
Brightkite- TUL
(%5 (%)
FourSquare TUL
(%)
88
99 95
95
92
92 85
85
Accop5
Accog5
Accep5
Apcgg5
80
80 80
83
74
(a) Effect of the selection of Top-k in HTPP.
Gowalla-LP
WeePlace-LP
Brightkite-LP
FourSquare-LP
12
Acojts
18
22
30
54-
AccsES
61
18
Acoo85
18
20
20
51
58
17
48
17
18
Aco85
45
52 15-
10
16 16
24
42
49 14
(b) Effect of the recent eycles VIMN.
Figure 4: Effects of hyper-parameters validated on different datasets.
the Pythia-70M (A.5 in Tab. 5) backbone model, which has fewer parameters, can demonstrate advantages certain datasets and tasks. Therefore, can be inferred that the suitability of backbone models for different tasks is significantly related to the training strategy of the backbone model, and not just to the number of parameters. For example, time prediction tasks, GPT-2 (A.8) performs better than the 2.7B+ parameter backbone model (A.4,7).
Ablation Study. As shown in Tab. 5 we find that the removal of (B.1) stands as a pivotal element in harnessing the efficiency for the TUL. task. We observe notable average performance degradation of 5.16% on the TUL task. The inherent habits of human activities can effectively reflect a user's behavioral characteristics, which strongly validates the importance of (B.1) in capturing the inherent habits of people's s activities. Ablation of this component (B.2) results in over 4.09% degradation on LP task, as people's actions are largely influenced by visiting intentions. Additionally, (B.3) allows for more accurate embedding modules and tasks. In the w/o LLM setting (B.4), we replace the large language model with standard transformer. (B.4) also shows that even without using an LLM as the base model, our reprogramming still achieves excellent results, even surpassing many baselines.
POI types and semantics, providing a foundation for subsequent
Reprogramming Interpretation. We provide three case studies visualize the improvements brought by HTPP and VIMN. In Fig. 33, we visualize the significance of different prompt words in three domain pools within the HTPP model. Orange indicates high significance, and Purple indicates low significance. It can be observed that as training progresses, the prompt words for this sample each domain concentrate on a few words. This also indicates that user behavior is diverse and personalized. To more intuitively see the effect of HTPP, we select 10 different users, each with several samples. We visualize the output of L.M with t-SNE 4 corresponding to the samples generated by these 10 users in Fig. 3p and Fig. 3.. It can be seen that with the addition of HTPP, the model can more comprehensively capture user behavior patterns and better recognize the behavioral characteristics of each user. Similarly, to better visualize the effect of VIMN, we select a continuous sequence of five user behavior points. Fig. 3-f show the Pearson correlation coefficients between the five outputs h without using the VIMN, without considering Delta Time input, and when using VIMN, respectively. Fig.Blg shows the time intervals among these five points. It can be observed that with the use of the VIMN, the closer the time, the greater the influence the current state, which aligns with human behavior patterns. Since plans cannot keep up with changes, people's decisions mostly influenced by recent behaviors. Therefore, our model uses Delta Time as constraint for correlation, effectively mimicking human activity patterns and achieving better prediction results.
show
Hyperparameter Analysis. There is significant difference in user behavior patterns across different datasets, As shown in Fig.4h in larger datasets like Foursquare, user behavior is more diverse, and larger K values yield better results. In Fig. 4. it can also be seen that the number of nearby cycles in VIMN is highly correlated with the distribution of time intervals in the dataset.
Reprogramming Efficiency. Tab. 6 provides an overall efficiency analysis with and without the backbone LLM. Our proposed reprogramming network itself is lightweight in activating the LLM's ability for different tasks (i.e., fewer than 10 million trainable parameters; only around 3.4 % of the total parameters in TanyL. and the leveraged backbones actually cap the overall efficiency. This is favorable even compared to the parfmeter-efffcient fine-tuning methods (c.g., QLoRA (8)

---

Table 6: Efficiency analysis of Mobility-LLM on WeePlace dataset on all tasks. Param. represents the total parameters of the model. Mem. denotes the GPU Memory. The ratio represents the ratio of trainable parameters (including the trainable parameters in QLoRA and the reprogramming parameters). The Time column denotes the total training time.
Tasks
LP
TUL
Variant
Metric
Param.
Mem.
Ratio
Time
Param.
Mem.
Ratio
Time
Param.
Mem.
Ratio
Time
Tiny (Default)
1,07B
11.3GB
3,72%
5.45h
1,03B
10.6GB
3.34%
13.24h
1.03B
10,6GB
3.42%
2.76h
TinyLlama-Che
1.07B
1133BB
3.72%
5.45h
1.03B
10.6GB
3.34%
13.24h
1.03B
10.6GB
3.41%
2.76h
LiteLlama
434M
8.7GB
5.42%
4.58h
412M
8.2GB
5.02%
5.65h
414M
8.2GB
4.60%
2.36h
phi-2
2.82B
21,7GB
1.12%
8.95h
2.81B
20.76GB
1.03%
17.59h
2.81B
20.76GB
1.21%
6.42h
Pythia-70M
63M
2.74GB
27.23%
2.76h
2,56GB
20.21%
4.92h
51M
2.56GB
Pythia-IB
1.08B
12.03GB
3.89%
3.89h
1.01B
11.87GB
3.21%
6.5h
1.01B
11.87GB
3.43%
3.98h
py.h
2.73B
22.4GBB
1.19%
11.7h
2.71B
21,76GB
1.23%
11.7h
2.718
21.76GB
1.15%
6.53h
GPT-2
0.138B
3.67GB
19.32%
3.06h
0.135B
3.23GB
19.11%
4.5th
0.135B
3.23GB
19.04%
1.27h
w/o L.L
[0.0128
3.23GB
92.23%
2.42h
0.011B
3.12GB
94.32%
4.95h
00.0118
3.12GB
97.53%
1.2h
in balancing task performance and efficiency. terms of runtime, the total training time with the advanced fine-tuning framework is acceptable compared to not using LLMs.
6 Conclusion
In conclusion, our work presents Mobility-LLMM, unified framework leveraging large language models (LLMs) to analyze check-in sequences and understand human mobility behaviors. By incorporating the visiting intention memory network (VIMN) and the human travel preference prompts (HTPP), our model excels in various tasks. Moreover, our model exhibits robust few- shot learning capabilities, outperforming existing methods by an average of 23.6% to 38.3%. Our work paves the way for a more comprehensive and accurate analysis of human mobility, benefiting individuals, businesses, and urban management.
Limitations The sets of POIs in different datasets (which usually cover different regions) are unique. Therefore, our proposed model trained on one dataset, its learned information about the set of POIs is not easily transferable to another dataset. Different sets of POIs have different functionalities and usually have a different number of POIs, making many modules (such as embedding and predictor) technically untransferable in a zero-shot setting. Future work will focus on developing universal user and POI embeddings enhance pross-datase migration and improve model versatility.
Acknowledgment. This work was supported by the National Natural Science Foundation of China (No. 62372031) and the Beijing Natural Science Foundation (Grant No. 4242029).
References
[1] Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXhv:2404.14219, 2024.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning. pages 2397-2430. PMLR, 2023.
[3] Defu Cao, Furong Jia, Sercan O Arik, Tomas Pfister, Yixiang Zheng, Wen Ye, and Yan Liu. Tempo: Promdt-based generative pre-trained transformer for time series forecasting. arXiv preprint 1eXii:2310000948 2023.
[4] Ching Chang, Wen-Chih Peng, and Tien-Fu Chen. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained lmms. arXiv preprint arXiv:2308. 08466, 2023.
[5] Quanjun Chen, Renhe Jiang, Chuang Yang, Zekun Cai, Zipei Fan, Kota Tsubouchi, Ryosuke Shibasaki, and Xuan Song. Dualsin: Dual sequential interaction network for human intentional
10

---

mobility prediction. In Proceedings of the 28th International Conference on Advances in Geographic Information Systems, pages 283-292, 2020.
[6] Tri Dao al. Tinyllama: An open-source small language model. arXiv:2401.022 023885, 2024.
arXiv preprint
[7] Liwei Deng, Hao Sun, Yan Zhao, Shuncheng Liu, and Kai Zheng. S2tul: A semi-supervised framework for trajectory linking. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM '23, page 375-383 Association for Computing Machinery, 2023.
[8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 2023.
[9] Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Komez-Rodrigure and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16, page 1555-1564. Association for Computing Machinery, 2016.
[10] Jie Feng, Yong Li, Chao Zhang, Funing Sun, Fanchao Meng, Ang Guo, and Depeng Jin. Deepmove: Predicting human mobility with attentional recurrent networks. In Proceedings of the 2018 world wide web conference, pages 1459-1468, 2018.
[11] Jie Feng, Zeyu Yang. Fengli Xu, Haisu Yu, Mudan Wang, and Yong Li. Learning to simulate human mobility. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 3426-3433, 2020.
[12] Kue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Bhushan TN. Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization? arXiv preprint 240:::2002000441 2024.
[13] Qiang Gao, Fan Zhou, Kumpeng Zhang, Goce Trajcevski. Xucheng Luo, and Fengli Zhang. Identifying human mobility via trajectory embeddings. In IJCAI, volume 17, pages 1689-1699 2017.
[14] Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning sentence embeddings. arXiv preprint arXiv:2104.0888) 2021.
[15] Letian Gong, Youfang Lin, Shengnan Guo, Yan Lin, Tianyi Wang, Erwen Zheng, Zeyu Zhou, and Huaiyu Wan. Contrastive pre-training with adversarial perturbations for check-in se- quence representation learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):4276-4283, Jun. 2023.
[16] Letian Gong, Huaiyu Wan, Shengnan Guo, Xiucheng Li, Yan Lin, Erwen Zheng, Tianyi Wang, Zeyu Zhou, and Youfang Lin. Spatial-temporal cross-vies view contrastive pre-training for check-in sequence representation learning. IEEE Transactions Knowledge and Data Engineering, 2024.
[17] Vinayak Gupta, Srikanta Bedathur, and Abir De. Learning temporal point processes for efficient retrieval of continuous time event sequences. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 4005-4013, 2022.
[18] Vinayak Gupta, Abir De, Sourangshu Bhattacharya, and Srikanta Bedathur. Learning temporal point processes with intermittent observations. In Proc. of the 24th International Conference on Artificial Intelligence and Statistics (AciicsssASSt (AISTATS), 2021.
[19] Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: LIm-based features for text-attributed graphs. arXiv preprint arXiv:2305. 195223, 2023.
[20] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 100-1735-17800 1997.
[21] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
11

---

[22] Jinsung Jeon, Soyoung Kang, Minju Jo, Seunghyeon Cho, Noseong Park, Seonghoon Kim, and Chiyoung Song. Lightmove: A lightweight next-poi recommendation fortaxicab rooftop advertising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, CIKM '21, page 3857-3866. Association for Computing Machinery, 2021.
[23] Renhe Jiang, Quanjun Chen, Zekun Cai, Zipei Fan, Xuan Song. Kota Tsubouchi, and Ryosuke Shibasaki. Will you go where you search? deep learning framework for estimating user search-and-ge behavior. Neurocomputing. 472:338-34 348, 2022.
[24] Ming Jin, Shiyu Wang, Lintao Ma. Zhixuan Chu, James Y Zhang, Xiaoming Shi. Pin- Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728. 2023.
[25] Nicholas Lim, Bryan Hooi, See-Kiong Ng. Yong Liang Goh, Renrong Weng, and Rui Tan. Hierarchical multi-task graph recurrent network for next poi recommendation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 1133-11433 2022.
[26] Yan Lin, Huaiyu Wan, Shengnan Guo, and Youfang Lin. Pre-training context and time aware location embeddings from spatial-temporal trajectories for user next location prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4241 -2448, 2021.
[27] Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. Multi modal molecule structure- text model for text-based retrieval and editing. Nature Machine Intelligence. 5(12):1447-144 2023.
[28] Xin Liu, Yong Liu, Karl Aberer, and Chunyan Miao. Personalized point-of-interest recommen- dation by mining users' preference transition. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 733-738 2013.
[29] Xin Liu, Daniel McDuff, Geza Kovacs, Isaac Galatzer-Levy, Jacob Sunshine, Jiening Zhan, Ming-Zher Poh, Shun Liao, Paolo Di Achille, and Shwetak Patel. Large language models are few-shot health learners. arXiv preprint arXiv:2305. 15525, 2023.
[30] Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, and Mingsheng Long, Autotimes: Au- toregressive time series forecasters via large language models. arXiv preprint arX1v:2402.02370. 2024.
[31] Yong Liu, Wei Wei, Aixin Sun, and Chunyan Miao. Exploiting geographical neighborhood characteristics for location recommendation. In Proceedings of the 23rd ACM international conference on conference on information and knowledge management, pages 739-748, 2014.
[32] Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Frozen pretrained transformers as universal computation engines. Proceedings of the AAAI Conference Artificial Intelligence, (667) Jun. 2022.
[33] Yingtao Luo, Qiang Liu, and Zhaocheng Liu. Stan: Spatio-temporal attention network for next location recommendation. In Proceedings of the Web Conference 2021, pages 2177-2185, 2021.
[34] Congcong Miao, Jilong Wang, Heng Yu, Weichen Zhang, and Yinyao Qi. Trajectory-user linking with attentive recurrent network. In Proceedings of the 19th international conference on autonomous agents and multiagent systems, pages 878-886, 2020.
[35] Ziqi Pang, Ziyang Xie, Yunze Man, and Yu-Xiong Wang. Frozen transformers in language models are effective visual encoder layers. arXiv preprint arXiv:2310. /2973, 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages PMLR, 2021.
12

---

[37] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. hayate-lab.com, 2018.
[38] Xuan Rao, Lisi Chen, Yong Liu, Shuo Shang, Bin Yao, and Peng Han. Graph-flashbace network for next location recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 1463-1471, New York, NY, USA, Association for Computing Machinery.
2022.
[39] Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. Representation learning with large language models for recommendation. CoRR, 0bs/2310.11950, 2023.
[40] Oleksandr Shchur, Marin Biloš, and Stephan Günneemnn. Intensity-free learning of temporal point processes. International Conference on Learning Representations (ICI.R) 2020.
[41] Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. Test: Text prototype aligned embedding to activate Ilm's 's ability for time series. arXiv preprint arXiv:2308.0822 2023.
[42] Zhou Tian, Niu Peisong, Wang Xue, Liang Sun, Rong Jin, and Patchtst Timesnet. One fits all: Power general time series analysis by pretrained Im. CoRR, abs/2302.1 1939, 2023.
[43] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advances in Neural Informa- tion Processing Systems, pages 5998-6008, 2017.
[45] Huaiyu Wan, Yan Lin, Shengnan Guo, and Youfang Lin. Pre-training time-aware location embeddings from spatial-temporal trajectories. IEEE Transactions on Knowledge and Data Engineering, 2021.
[46] Jiawei Wang, Renhe Jiang, Chuang Yang, Zengqing Wu, Makoto Onizuka, Ryosuke Shibasaki, and Chuan Xiao. Large language models as urban residents: An I1m agent framework for personal mobility generation. arXiv preprint 240. 2024.
[47] Yuxia Wu, Ke Li, Guoshuai Zhao, and QIAN Xueming. Personalized long-and short-term preference learning for next poi recommendation. IEEE Transactions on Knowledge and Data Engineering, 2020.
[48] Xiaohang Xu, Toyotaro Suzumura, Jiawei Yong, Masatoshi Hanai, Chuang Yang. Hiroki Kanezashi, Renhe Jiang, and Shintaro Fukushima. Revisiting mobility modeling with graph: A graph transformer model for next point-of-intedt recommendation. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems, pages 1-10, 2023.
[49] Dingqi Yang, Daqing Zhang, Zhiyong Yu, and Zhu Wang. A sentiment-enhancee personalized location recommendation system. In Proceedings of the 24th ACM Conference on Hypertext and Social Media, pages 1 119-128. ACM, 2013.
[50] Dingqi Yang, Daqing Zhang, Zhiyong Yu, and Zhiwen Yu. Fine Fine-grained preference-awarn location search leveraging crowdsourced digital footprints from lbsns. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 479 488. ACM, 2013.
[51] Song Yang, Jiamou Liu, and Kaiqi Zhao. Getnext: trajectory flow map enhanced transformer for next poi recommendation. In Proceedings of the 45th International ACM SIGIR Conference on research and development in information retrieval, pages 1144-1153, 2022.
[52] Di Yao, Chao Zhang. Jianhui Huang, and Jingping Bi. Serm: A recurrent model for next location prediction in semantic trajectories. In Proceedings the 2017 ACM on Conference on Information and Knowledge Management, pages 2411-2414, 2017.
13

---

[53] Yuan Yuan, Jingtao Ding, Chenyang Shao, Depeng Jin, and Yong Li. Spatio-temporal diffusion point processes. arXiv preprint arXiv:2305. 12403, 2023.
[54] Dejiao Zhang, Wei Xiao, Henghui Zhu, Xiaofei Ma, and Andrew Amold. Virtual augmentation supported contrastive learning of sentence representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 884-876,40 2022.
[55] Liang Zhang, Cheng Long, and Gao Cong. Region embedding with intra and inter-view contrastive learning. IEEE Transactions Knowledge and Data Engineering. 2022.
[56] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
[57] Qiang Zhang, Aldo Lipani, Omer Kirmp, and Emine Yilmaz. Self-attentive Hawkes process. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11183-11193, PMLR, 13-18 Jul 2020.
[58] Qianru Zhang, Haixin Wang. Cheng Long, Liangcai Su, Xingwei He, Jianlong Chang, Tailin Wu, Hongzhi Yin, Siu-Ming Yiu, Qi Tian, and Christian S. Jensen. A survey of generative techniques for spatial-tenm namiaal-emmpral data mining, 2024.
[59] Pengpeng Zhao, Anjing Luo, Yanchi Liu, Fuzhen Zhuang, Jiajie Xu, Zhixu Li, Victor S Sheng, and Xiaofang Zhou. Where go next: A spatio-temporal gated network for next poi recommendation. IEEE Transactions on Knowledge and Data Engineering, 2020.
[60] Fan Zhou, Shupei Chen, Jin Wu, Chengtai Cao, and Shengming Zhang, Trajectory-usee linking via graph neural network. In ICC 2021 IEEE International Conference on Communications, pages 1-6, 2021.
[61] Fan Zhou, Yurou Dai, Qiang Gao, Pengyu Wang, and Ting Zhong, Self-supervised human mobility learning for next location prediction and trajectory classification. Knowledge-Based Systems, 228: 107214, 2021.
[62] Fan Zhou, Qiang Gao, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Fengli Zhang. Trajectory-user linking via variational autoencoder. In IJCAI, pages 3212-3218, 2018.
[63] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36, 2024.
[64] Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In Proceedings of the 37th International Conference Machine Learning, ICMI.'20. Rorg 2020.
14

---

Appendix
Here we introduce the related work of this paper in Sec A The implement details including experiment settings and evaluation metries can be found in Sec B The datasets and Partially Frozen Attention (PFA) LLM settings are shown in Sec. [ An introduction to Baselines is presented in See We display the results of ablation and few shot on all datasets in See|Eaane and Sec|F respeeesseyy.illl. Moreover, We introduce the calculation process of data in feed-forward layer and geohash embedding layer in and Sec in in detail respectively. We also list POI categories in Sec
A Related Works
A.1 Mobility Data Mining
Location-based services have given rise new and promising research topic known mobility data mining, which has led to the emergence of three significant tasks that contribute to enhancing the quality of services: next location prediction (LP), next time prediction (TP), and trajectory user link (TUL). Recent studies have confirmed that deep learning techniques, specifically recurrent neural networks (RNNs) and attention mechanisms, are highly effective in capturing sequential and periodic patterns of human mobility. By combining deep learning techniques, researchers have made significant advancements in capturing both the sequential and periodic patterns of human mobility. The core of these models is the modeling of check-in sequences, which leads to improved accuracy in location prediction and trajectory analysis.
LP aims to anticipate a user's future location based on their historical movement. Several notable models have emerged as prominent approaches in LP. DeepMove [10] leverages RNNs and attention mechanisms to capture the spatial-temporal intentions in users' location data and predict their next destination. STAN K3 introduces a spatial-temporal attention network that incorporates spatial and [59] focuses on long and short-term patterns in 20 model. SERM (52) utilizes encoder-decoder
temporal contexts for accurate prediction. LSTPM
user trajectory using an attention-based LSTM
architecture with a spatial- -tattiall residual network to capture user preferences and predict future locations. PLSPL [47] trains two LSTM models for location- and category-based sequences to capture the user's preference. LightMove [22] designs neural ordinary differential equations to enhance robustness against sparse or incorrect inputs. HMT-GRN (25) alleviates the data sparsity problem by learning different User-Region matrices of lower sparsities in multitask setting. Graph- Flashback (38 constructs a spatial-temporal knowledge graph to enhance the representation of POIs, having a great advantage when dealing with the historical sequence input of the same length. GETNext (51) introduces a user-agnostic global trajectory flow map as means to leverage the abundant collaborative signals.
TUL is a significant task that focuses on establishing connections between different trajectories, facilitating the analysis of user movement patterns, and uncovering valuable insights about their behavior. Notable models have been specifically developed address the challenge of predicting trajectory links. TULER [33 takes advantage of advanced algorithms to establish links between trajectories, allowing for comprehensive understanding of user movement patterns. TULVAE 62 uses latent variables to model the variability in trajectories. capturing hierarchical and structural semantics and improving the identification and linking performance. MoveSim []] simulates human mobility using a generative adversarial framework that incorporates attention mechanisms to capture complex spatial-temporal transitions in human mobility. DeepTUL. 34 utilizes deep learning techniques to extract representations from trajectory data and facilitate the prediction of trajectory links. S2TUL [7] utilizes graph convolutional networks and sequential neural networks to capture trajectory relationships and trtja-trajectory information. GNNTUL (60) employs graph neural networks for human mobility and associates the traces with users on social networks.
TP focuses on estimating the time at which a user is likely to visit their next location. To accomplish this, is common practice to use intensity functions to represent the rate or density of event occurrences, various models have been developed to model the intensity function and make accurate time predictions effectively. Modeling the intensity function using RNNs or attention mechanisms is common approach for predicting the occurrence of events. IFLTPP 40 approximates any distribution of inter-event times using normalizing flows and mixture distributions. RMTPP [9] utilizes RNNs to model the intensity function. SAHP 57] combines the Hawkes process with self-
15

---

attention mechanisms to capture the temporal dependencies and spatial influences in event sequences. THP 64 combines the Hawkes process with transformer- architectures to capture temporal sequence sequences. NSTPP [7 utilizes neural ODEs to model discrete events in continuous time and space, enabling the learning of complex distributions spatial and temporal domains. IMTPP [88] models the generative processes of observed and missing events and utilizes unsupervised modeling and inference methods for time prediction. DSTPP [53] proposes a novel parameterization framework that uses diffusion models to learn complex joint distributions.
dependencies in event
In recent years, many sequence representation models have become a hot topic, including those for natural language sequences and spatial-temporal trajectory sequences. For natural language sequences, SimCSE [14] is a classical sequence representation language model that utilizes a straight- forward contrastive learning framework. It employs standard dropout as noise, making it easy to implement. However, it cannot effectively distinguish between hard negative samples or semanti- cally similar sequences. Thus, VaSCI. (54) leverages neighborhood information to generate virtual augmentations, improving the quality of data transformations compared to traditional methods. In- spired by models for natural language sequence representation, researchers in the netpaaltemppoal domain have also conducted similar studies on spatial-temporal sequences. SML [6]] first employs self-supervised contrastive learning to effectively manage sparse and noisy trajectory data, enhancing trajectory representation through spatial-temporia temporal data augmentation. However, the spatial-temporal augmentation in SML needs to be manually specified, which is relatively cumbersome. Therefore, ReMVC (55) effectively addresses this issue by adopting a cross-view contrastive approach. uses contrastive learning to extract meaningful region representations, improving intra-view and inter-view learning. And CACSR 1.5] effectively adds adversarial perturbations and automated data augmentation, enhancing contrastive training processing. While it lacks interpretability and does not understand the semantic information of time and space from the actual semantics of human activities.
In summary, it can be seen that whether it is end-to-end models designed for specific tasks or spatiotemporal sequence representation learning models, their understanding of human activities is not yet deep enough, and they lack a profound understanding of the essence and spatial-temporal patterns behind human activities.
A.2 Cross-domain Application of LLMS
We have witnessed the great success of Large Language Models (LLMs) in natural language process- ing [77] and some cross-domain (such as computer vision [36], time series, and graph-related tasks. In the field of graph theory, To tackle graph-related tasks, Graph Neural Networks (GNNs) have emerged as one of the most popular choices for processing and analyzing graph data. While GNNs excel at capturing structural information, their reliance on semantically constrained embeddings as node features limits their ability to fully express the complexities of the nodes. By incorporating LLMs, GNNs can be augmented with stronger node features that effectively capture both structural and contextual aspects. TAPE utilizes semantic knowledge that is pertinent to the nodes (c.g., papers) generated by LLMs enhance the quality of initial node embeddings in GNNs. Furthermore, InstructGL.M replaces the predictor in GNNs with LLMs, harnessing the expressive capabilities of natural language through techniques like graph flattening and prompt-based instruction design. MoleculeSTM aligns GNNs and LLMs within a shared vector space, integrating textual knowledge into graphs (e.g., molecules) to enhance reasoning capabilities.
In the field of Computer Vision (CV), ViT is an image classification model based on the Transformer architecture, which has achieved excellent performance on multiple benchmark image classification datasets. CLIP is a large-scale pre-trained model developed by OpenAl that aligns images and text, enabling simultancous learning of representations for both modalities, which allows for interactive understanding between images and text. DALL-E utilizes large-scale pre-trained language model to transform text into images through a generative approach, showcasing remarkable capabilities in image generation. TEST [11. as a time series (TS) embedding method tailored for Large Language Models (LLMs), generates embeddings that capture similarity. instance -wiee, feature-wise and text- eex--prtootype alignment for TS tokens. Time-LLM (24] introduces a reprogramming framework. Given the challenge of limited training data, recent studies turn to adopting large language models (LLMs) to address cross-domain applications. In the field of time series analysis, LLM4TS 4 is the pioneering method that aligns pre-trained Large Language Models with temporal characteristics, introducing two-level aggregation method to effectively incorporate multi-scale temporal data
16

---

into pre-trained LLMs. One-Fits-All [63 is unified framework that leverages frozen pre- prettraied language model to attain state-of-the-ar comparable performance across various major types of time series analysis tasks. TutoTimmss 30) facilitates the tokenization of time series into the embedding space of LLMs and intelligently utilizes the inherent token transitions to effectively predict time series in an autoregressive manner. In the field of computer vision, LM4VE [35] incorporates a frozen transformer block from an LLM as a general- To tackle graph-related tasks, TAPE [99] utilizes semantic knowledge that is pertinent to the nodes generated by LLMs to enhance the quality of initial node embeddings GNNs. MoleculeSTM [27] aligns GNNs and LLMs within shared vector space, integrating textual knowledge into graphs to enhance reasoning capabilities.
visual encoder layer.
GPT4TS 42 utilizes PLMs by freezing the self-attention feed-forward layers. For visual encoding tasks, L.4 VisualFnnoding [35] incorporates a frozen transformer block from PL.M as a general- purpose visual encoder layer. RLMMec (39] integrates the semantic space of PLMs and collaborative relational signals using an alignment framework. An L.M agent framework is proposed for flexible and efficient personal mobility generation [46].
While these studies provide valuable insights, it is important recognize that their methods cannot be directly applied to the domain of trajectory learning. Trajectory data exhibits distinct spatial- temporal characteristics and contains unique information that requires customized approaches and considerations.
B
Implement Details
B.1
Settings
The Mobility-LLLM model was constructed using the Py PyTorch" The loss function is a cross-entropp loss for the LP and TUL tasks and an MAE loss for the TP task. The performance the validation sets determines the prraa-pprameters and the best models. All experiments are performed five times, and the means and standard deviations embedding dimension el is 256, while the hidden state h has a size of 256. The learning rate is 0.001. The model is pre-trained for 100 epochs the training with the early-stopping mechanism of 10 patients. All trials have been conducted on Intel Xcon E5-2620 CPUs and NVIDIA RTX A40 GPUs.
calculated. To make fair comparison, for all methods, the
B.2 Evaluation Metrics
Four metrics are used for evaluating the models: mean absolute error (MAE), root mean squared error (RMSE), Accuracy at (ACC@K), and Mean Reciprocal Rank (MRR). MAE and RMSE quantify absolute errors. ACC@ @ K measures the accuracy of the predictions within the top K ranks. MRR calculates the average of the reciprocal ranks, where a higher rank indicates a better performance.
MAE
Y, RMSE
Σ
Y,
m
where the number of predicted values, Y; is the predicted value, Y, is the target value.
K ACC@K = Σ Y = Y.). MRR = m rank;
where m is the total number of queries, rank; the rank position of the first relevant result for
i - th query.
C Check-in Datasets
C.1 Datasets Introduction
Gowalla was a location-based social networking service where users shared their locations by checking-in. The dataset includes 6,442,890 check-ins made by 196,591 users from February 2009 to
SHIPPP://pytorch.org
17

---

October 2010. Additionally, it contains the undirected friendship network, which consists of 950,327 edges. This dataset provides valuable information for studying mobility patterns and social network analysis.
Nodes: 196,591
Edges: 950,327
Check-ins: 6,442,890
Time Period: Feb 2009 Oct 2010
Data Fields: User ID, Check-in Time, Latitude, Longitude, Location ID
Weeplaces dataset was collected from service that visualizes users' check-in activities across multiple location-bies based social networks, including Facebook Places, Foursquare, and Gowalla. It consists of 7,658,366 336 check-ins generated by 15,799 users over 971 ,3009 locations. The data primarily captures the check-in history and friend connections of users.
Check-ins: 7,658,368
Users: 15,799
Locations: 971,309
Data Fields: User ID, Check-in Time, Latitude, Longitude, Location Name, Category
Brightkite was necationbbssed social networking service where users could check in at various locations and share their locations with friends. The dataset includes 4,491,145 check-ins made by 58,228 users from April 2008 to October 2010. This dataset is useful for researching user mobility and social interaction patterns.
Check-ins: 4,491,143
Users: 58,228
Time Period: Apr 2008 Oct 2010
Data Fields: User ID, Check-in Time, Latitude, Longitude, Location ID
Foursquare is popular location-based social networking service where users check in at various venues. The dataset includes check-in data from multiple cities, such as New York City and Tokyo, over period from April 2012 February 2013. It contains 227,428 check-ins in New York City and 573,703 check ins Tokyo. Each entry provides a user ID, check-in in time, latitude, longitude, and venue category.
Check-ins in NYC: 227,428
Check-ins in Tokyo: 573,703
Time Period: Apr 2012 Feb 2013
Data Fields: User ID, Check-in Time, Latitude, Longitude, Venue Category
C.2
The Staties of Processed Datasets
We can see that the dataset we selected includes the Gowalla dataset, which has a large number of samples, users, and Points of Interest (POI), as well as the Foursquare dataset, which has small number of users but large number of POIs. Additionally. we have the Brightkite and WeePlace datasets, which have a small number of both users and POIs. These datasets cover different scenarios and can fully validate the model's comprehensive capabilities.
C.3 Partially Frozen Attention (PFA) LLM
The frozen pre-trained transformer (FPT) has demonstrated effectiveness in variety of downstream tasks across non-language modalities [2) We adopt a partially frozen attention (PFA) LLM, specifically designed enhance prediction accuracy in check-in sequences prediction.
The difference between the FPT and our PFA primarily lies the configuration of frozen attention layers. In the FPT framework, both the multi-heal attention and feed-forward layers are frozen during
18

---

Table 7: The statics of Processed Datasets
Datasets
#Samples
#PP1s #POIs
Gowalla
413,563
5,853
52.032
WeePlace
104,762
1,028
9,295
Brightkite
44,716
431
3,554
FourSquare
60,734
703
11,117
training, as these layers contain the most significant portion of the learned knowledge within the LLM. In the PFA, maintain the first F layers identical the FPT, but crucially, we unfreeze the last U multi-head attention layers since the attentions offer effective handling of spatial-temporal de- pendencies in data. Consequently, Mobility-LLM can adapt to check-in datasets while preserving the Foundational knowledge acquired during pre-training.
D Overview of Baselines
DeepMove [10] is an attentional recurrent network designed for predicting human mobility from lengthy and sparse trajectories. It introduces multi-modal embedding recurrent neural network that captures complex sequential transitions by embedding multiple factors influencing human mobility. DeepMove further incorporates a historical attention model with dual mechanisms to effectively capture multi-level periodicity. This historical attention model enhances the recurrent neural network's ability to predict mobility patterns by leveraging the inherent periodic nature of human movement.
LightMove [22 is lightweight and accurate deep learning-based method developed for predicting next locations of taxicabs in order enhance targeted advertising on taxicab rooftop devices. The paper focuses on Motov, leading company in South Korea's 's taxicab rooftop advertising market, and aims to leverage demographic information of locations to improve the preparation of targeted advertising campaigns.
PLSPL 47 addresses the task of recommending the next Point of Interest (POI) for users based on their historical check-in data. The main objective is to capture both the users' general taste and their recent sequential behaviors, as these factors are crucial making accurate recommendations. However, existing methods often assume the same dependencies for all users, disregarding the fact that different users may have varying preferences and dependencies on these two aspects.
addresses the challenging task of predicting the next Point-of-Interee (POI) that a user likely visit in personalized recommender systems. One of the main difficulties in this task is the large search space of possible POIs in the region, which leads to data sparsity issues in cxisting works and hampers performance.
HMT-LSTE (55
LSTPM (59 proposes a method for POI recommendation by modeling long-term and short-term preferences. It uses nonlocal network for capturing long-term dependencies and geo-dilated RNN for considering geographical relationships. This approach improves the reliability of recommendation results compared to existing methods.
TULER [33 addresses the task of understanding human trajectory patterns Location- Social Networks (I (LBSNss applications. This task is crucial for various applications like personalized recommendation and preference-based route planning. Existing methods often classify trajectories or their segments into predefined categories based on spatial-temporal values and activities, such as walking or jogging. However, the paper focuses on novel problem called Trajectory-Usse Linking (TUL), which aims to identify and link trajectories to the users who generated them in LBSNs.
TULVAE [62] addresses the important task of Trajectory- -user Linking (TUL) in Geo-tagged social media (GTSM) applications. It utilizes neural generative architecture with stochastic latent variables that span hidden states in an RNN. By incorporating variational autoencoder techniques, TULVAE can capture the hierarchical and structural semantics of trajectories using high-dimensional latent variables. Moreover, TULVAE addresses the data sparsity challenge by leveraging large-scale unlabeled data, providing more comprchensive representations of user mobility patterns.
19

---

MoveSim [1]
is a framework for the realistic simulation of massive human mobility data. Existing solutions for mobility simulation have limitations in generating transitions and intricate regularities in human mobility patterns.
data due to complex
S2TUL 7 is flexible Seni-Supervised framework for Trajectory- User Linking (TUL) with five main components.
IFLTPP 440 is novel approach that models event sequences with irregular intervals using temporal point processes. Unlike the standard intensity-based methods, IFLTPP directly models the conditional distribution of inter-event times. Drawing inspiration from normalizing flows, IFLTPP captures complex dependencies in the data while maintaining practicality and interpretability through simple mixture model.
THP model designed for handling massive event sequence data with complex temporal dependencies. To address the limitations of existing recurrent neural network-basee models, THP utilizes the self-attention mechanism of Transformers, which effectively captures short-term and long-term dependencies in the data.
an approach that addresses the retrieval problem of continuous-time event sequences (CTESs) using marked temporal point processes (MTPP). It improves the retrieval performance by applying trainable unwarping function the query sequence, making it comparable to the corpus sequences.
NSTPP 17
DSTPP [53] is framework designed for modeling spatial-temporal point processes (STPPs) by leveraging diffusion models. addresses the limitations existing solutions that assume conditional independence between time and space.
ReMVC (55] is a model designed for unsupervised region representation learning from unlabeled urban data. It addresses the limitations of previous methods by leveraging contrastive learning for multi-view region representation learning.
VaSCL [54] is a framework designed for data augmentation in contrastive representation learning, with focus on the challenging context of natural language processing (NLP). Unlike other domains, NLP lacks general rules for effective data augmentation due to the discrete nature of language.
SML [61 is framework that tackles the challenges of extracting meaningful supervised signals from sparse and noisy human mobility data obtained from sesation-based services (LBS). The framework aims to leverage this data for various applications, including location recommendation, anomaly trajectory detection, crime discrimination, and epidemic tracing.
CASCR [5] is model that addresses the accurate representation learning of user- user-generated check-in sequences in human mobility data. introduces a contrastive pre- pre-training approach specifically designed for check-in sequence representation learning, eliminating the need for manual adjustments to data augmentation strategies.
E Variants
E.1 Variants of the Used LLMs
TimyLlama 6). TinyLLama-I.IB a lightweight language model developed by team of researchers at the Singapore University of Technology and Design (SUTD). It has 1. billion parameters and was pre-trained on about 3 trillion tokens. The training process for TinyLlama was done in 90 days using 16 A100-40G GPUs. The researchers explored the model's 's performance when the number of tokens suggested by the scaling law was exceeded by training the mini-model using a large amount of data. The model also used various optimizations such flash attention 2, FSDP (Fully Sharded Data Parallel), and xFormers to improve the efficiency and throughput of training. The application of these techniques gives TinyLlama significant advantage in terms of training speed and memory usage.
Laaa adopts exactly the same architecture and tokenizer as Llama 2. This means TinyLlama can be seamlessly integrated into many open-source projects built upon Llama. Additionally, TinyLlama compact with only 1.1B parameters, allowing cater to multitude of applications demanding a restricted computation and memory footprint.
20

---

SinyLlam--Chal [6]. The TinyLLama-Chie Chat model is fine-tuned on top of Man.aaaaaaaaiaaaaa J.B-intermeddite-stepy -twpedditttsse-1411111111 1431k-31, Following HF's Zephyr's training recipe, the model was "initially fine-tuned on a variant of the UltraChat dataset, which contains a diverse range of synthetic dia- logues generated by ChatGPT. We then further aligned the model with TRL's DPOTrainer on the Openbmb/UltraFeedhace dataset, which contains 64k prompts and model completions that are ranked by GPT-4.".
LiteLlama [2]. LiteLlama Series Models Published by Xiaotian Han, Texas A&M University. The LiteLlama series of models have wide range of applications on edge devices, such as smartphones, IoT devices, and embedded systems, which usually have limited computational power and storage space, and they are unable to run large language models efficiently. Therefore, it is particularly important to explore small models in depth.
The main models in the I Lite family are Lite-.lam--40MM. Late presents open-source reproduction of Meta AI's LLLMa 2. It has 460M parameters with 1T tokens. Litel.lmma-460-111 was trained on the RedPajama dataset and the text was tokenized using GPT2Tokenizer. The authors evaluated the model on the MMLU task, comparing the parameter sizes to those of a large model of equal capacity, and the results demonstrate that with a significantly reduced number of parameters, LiteLlama-460M-11 IT still achieves results comparable to or better than the other models.
The Phi family of language models are small language models introduced by Microsoft. The goal of the Phi family of language models is to demonstrate that by constructing high-quality pre-training data, small models can achieve significantly better performance than larger models at lower training costs. The Phi family of language models includes Phi-1, Phi-1.5, and Phi-2.
Phi-2 M.
Phi-2 is a Transformer with 2.7 billion parameters. Trained using the same data sources as Phi-1.5, Phi-2 incorporates a new data source consisting of various NLP synthetic texts and filtered websites (for safety and educational value). It showcases nearly state-of-the-arr performance among models with less than 13 billion parameters, especially when assessed against benchmarks testing common sense, language understanding, and logical reasoning.
Pyth 1B, 2.8B (2). The Pythia Scaling Suite is a collection of models developed to facilitate interpretabiliity research. It contains two sets of eight models of sizes 70M, 160M. 410M, IB, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same order. Additionally, 154 intermediate checkpoints per model are hosted on Hugging Face as branches.
The Pythia model suite was deliberately designed to promote scientific research on large language models, especially interpretability research. Despite not centring downstream performance as a design goal, the models match or exceed the performance of similar and same-sized models, such as those in the OPT and GPT-Neo suites.
GPT-2 (37]. GPT-2 is transformers model pre-trained on very large corpus of English data in self-supervised fashion. This means it was pre-trained on the raw texts only, with no humans labeling them in any way. More precisely, it was trained guess the next word in sentences.
Inputs are sequences of continuous text certain length, and the targets are the same sequence, shifted one token to the right. The model uses internally a masking mechanism to make sure the predictions for the i-th token only use the inputs from to but not the future tokens.
E.2 Variants of Ablation
To further evaluate the effects of different components in Mobility-LLM, we conduct ablation experiments and analyze experimental results on all datasets. We compare these four variants on three downstream tasks.
We remove the HTPP Module. The rest of the settings are the same Mobility-LLM. We use this setting to evaluate the function of the HTPP Module.
w/o HTPP:
w/o VIMN: We use full-connection to replace the VIMN module. The rest of the settings are the
same as Mobility-LLM. We use this setting to evaluate the function of the VIMN module.
w/o PPEL: We use a learnable parameter to represent a POI. We use this setting to evaluate the function of the PPEL.
21

---

w/o LLM: We use Transformer layer to replace the LLM. The rest of the settings are the same as MLL. We use this setting evaluate the function of the LLM.
E.3 Rusults of Variants on Different Datasets
E.3.1
Rusults of Variants WeePlace
Table 8: Ablations on WeePlace dataset in all tasks. Red: the best, Blue: the second-best.
Tasks
TUL
Variant
Metic
Acc
Acc
Acc@220
MRR
Acc
Acc
Acc
MRR
MAE
RMSE
TloyLlama (default)
20.47
39.22
56.69
29.21
79.03
88.04
91.48
83.21
28.25
35.54
TinyLlama-Chen
80.90
Litel.luma
20:06
40.20
53.28
29.67
76.20
86.49
87.01
81.30
28.12
34,45
phi-2
19.10
38.23
55.97
28.51
70.72
79.03
83.38
75.87
28
34,87
py pythia-70M
Ry pythia-1B
19.11
37.78
28.45
90.74
28
35.15
p.ttha-2.88B
19.77
39.77
54.33
27.77
75.28
82.42
$9.57
$0.49
28. 28.70
36.36
GPT-2
19.11
36.03
54.47
27.59
73.93
83.15
86.87
75.94
27.88
34,97
HTPP
19.32
38.31
51.42
26.89
76.73
$4.94
78.25
28.85
35.43
w/o VIMN
19.72
36.61
$3.69
27.05
74.77
86.08
86.28
79,56
28.22
35.43
PPE
37.67
58.00
77.21
84.96
88.29
81.85
28.82
34.76
LLM
17.66
32.84
49.
24.80
68.15
81.34
83.52
74.59
28.53
Language Model Variants. We compared eight representative backbones on WeePlace dataset with varying capacities in Tab. 8 We found that the TinyLlama backbone model performed the best overall for our task. particular, TinyL.ama achieved the highest scores in I.P (Acc@) 1, Ace 20) and TUI. (Acc@1 @cc@5, 5, Acc @2, MRR) tasks. The Chat version of TinyL was relatively less suitable, showing lower performance across most metrics. Interestingly. LiteLlama exhibited competitive performance, achieving the best Acc 5 and MRR in the LP task. Despite having fewer parameters, Pythia-70 70M demonstrated advantages in some tasks, achieving the second-best Acc@5 in the TUL task and the best MAE in the TP task. This suggests that the suitability of backbone models for different tasks significantly influenced by the training strategy rather than just the number of parameters. For example, in trajectory prediction tasks, GPT-2 performed better than some models with larger parameter sizes such as Pythia-2.8B.
Ablation Study. As shown in Tab. we conducted an ablation study to understand the impact of different components on performance. Removing HTPP (w/o HTPP) led to significant decrease in performance, particularly in the I.P task where Acc @20 decreased of 9.29 % and MRR of 7.94 %. This highlights the importance of HTPP the model's ability to make accurate predictions. Similarly, removing VIMN (w/o VIMN) resulted in noticeable performance drops in both LP and TUL tasks, indicating its role in maintaining robust prediction capabilities. Without PPE (w/o PPE), the model showed a moderate decrease in performance, with MRR in the LP task decreased of 5.75%. The removal LLM (w/o LLM) had the most pronounced negative effect across all tasks, with LP Acc@1 decreased of 13.7 % and MRR of 15.1 %, and TUL metrics significantly reduced. This underscores the crucial role of the large language model in the overall framework, as even replacing it with standard transformer could not achieve comparable performance.
Table 9: Ablations on Brightkite dataset in all tasks. Red: the best, Blue: the sccond-best.
Tasks
Variant
Metic
Acc81 1
Acc
Acc 20
MRR
Acct1
Acc
Acct9 20
MRR
MAE
RMSE
TinyL (default)
53.18
68.31
74.11
59.89
83.05
88.52
90.35
85.73
346.44
423.26
TlayLllam--hha
72.81
77.54
347.79
LiteIInma
$1.74
69.13
72.93
59.98
83.08
87.40
87.51
83.18
345.51
411.89
phi-2
50.17
65.87
73.02
57.52
74.40
81.59
85.24
77.86
351.12
423,40
py Pythia-70M
87.65
427.10
py thiii-BB
50.23
67.70
73.23
59.03
81.65
86.84
89.89
83.10
316.06
424.50
pythia-2.8B
52.85
68.04
73.58
58.83
81.90
86.59
88.65
83.68
346.06
424.93
GPT-2
50.86
61.19
70.30
56.41
79.21
85.31
85.88
81.24
346.75
423.22
428
w/o VIMN
51.12
63.07
71.47
$7.46
80.67
85.01
86.52
81.24
348.51
428.38
w/o PPE
50.61
64.44
71.83
57.49
80.42
86.10
86.68
82.83
348.52
424.52
51.10
22

---

E.3.2 Rusults of Variants on BrightKite
Language Model Variants. We compared various language model variants on the Brightkite dataset across different tasks, as shown in Tab. 9 The TinyL (default) variant achieved the best performance in LP tasks, with the highest Acc@1 1 (53.18%) and Acc@20 (74.11 %),, and the second-best MRR (59.89%) %). also excelled in TUL tasks, achieving the highest scores across all metrics, indicating its robust performance. LiteLlama performed exceptionally well, securing the highest A (69.13%) and MRR (59.98% %) LP tasks, and the best MAE TP tasks.
TinyLlama- Chat showed slightly lower performance compared to TinyLlama, particularly in TUL tasks, where fell behind the other variants. PYthia-70M, despite having fewer parameters, performed competitively, achieving the second-best Acc@ 5 (87.63% ) in TUL tasks and the best RMSE in TP tasks. Pythia-2.8B demonstrated strong performance as well, securing the second-best -oooddbesttt Acc@20 (73.38% %) in LP tasks and showing consistency in TUL tasks.
Overall, the results suggest that the TinyLlama variants, especially the default one, are highly effective across different tasks, while LiteLlama and Pythia variants show specific strengths, indicating that the choice of backbone model can be tailored to the specific requirements of the task.
Ablation Study. The ablation study results in Tab. Thighlight the importance of different components in the model. Removing HTPP (w/o HTPP) led to a significant drop in performance across all tasks. In TUL tasks, MRR decreased of 5.95%. while in LP tasks, there was a notable decrease in performance metrics, with Acc @ 20 decreased of 7.91 % and MRR of 5.26% indicating the critical role of HTPP in maintaining prediction accuracy.
Removing VIMN (w/o VIMN) also resulted in decreased performance, particularly in LP and TUL tasks, where the metrics showed notable average performance degradation 4.07% %.
The absence of PPE (w/o PPE) led to moderate performance degradation, with MRR in LP tasks decreased of 4.01 % and in TUL tasks of 3.38% %. The removal of LLM (w/o LLM) had the most pronounced negative effect, especially in LP tasks where Acc@5 5 decreased of 17.03 % and MRR of 14.68%. % This underscores the importance of the LLM component, as its absence significantly hinders the model's overall performance across all tasks.
Table 10: Ablations on FourSquare dataset in all tasks. Red: the best, Blue: the second-best.
Tasks
TUL
TP
Variant
Metic
Acc
Acc
Acc 6:20
MRR
Acc G
Acc
6:20:
MRR
MAE
RMSE
Llama (default)
17.29
37.17
53.16
26.47
72.08
79.67
84.35
75.71
309.78
505.03
TinyLLama-
25.88
310.68
507.17
LiteLlami
16.50
37.50
$2.10
26.59
72.10
78.42
81 $1.83
73.46
311.43
489.99
phi-2
16.36
35.81
52.48
25.37
64.76
73.36
79.55
68.69
313.34
506.72
Pythla-70M
16.07
36.83
$2.04
25.97
71.01
78.55 18.79
82.30
74.43
507.00
py 11
308.51
507.01
psthia-2.88
17:13
37.28
$2.95
26. 16
71 5%
77.71
82.65
73.31
310.66
S05 50
GPT-2
16.47
34.96
50.38
24.91
69.22
76.55
80.39
71.89
310.67
506.50
w/o VIMN
16.22
35.51
51.06
25.27
69.79
76.21
80.26
71.60
311.63
510.11
w/o PPE
16.51
35.14
51.73
36
70.00
77.42
80.81
72.88
310.70
506.53
LLM
14.55
30.78
45.95
22.58
64.28
73.46
76.52
68.62
321.44
$19.45
E.3.3 Rusults of Variants on Foursquare
Language Model Variants. We evaluated multiple language model variants on the FourSquare dataset across different tasks, with results presented in Tab. 1O, The TinyL.ama (default) variant demonstrated the best performance in IP tasks, achieving the highest Acc@) 1 (17.29 ))). Acc @ 20 (53.16%) and the second-best MRR. In TUL tasks, it also performed exceptionally well, securing the second-best Acc@ (72.08%) %) and the best Acc Acc@20 (84.32%) and MRR. This indicates its overall robustness in handling various tasks.
The LiteLlama variant excelled with the highest Acc @5 (37.50 %) and MRR in LP tasks and matched TinyLlama in TUL tasks with the best Acc@1 (72.10% although it slightly underperformed in other TUL metrics. TinyLlama-Chee Chat, while showing strong performance, did not outperform the default LamyLlama variant, particularly in TUL tasks where it scored lower across all metrics.
Pythia-70M and Pythia-2.88 showed competitive results, with Pythia-70M achieving the second-best Acc @5 (78.79 %) in TUL tasks and Pythia-2. -y securing the second-best Acc @ (17.13%) in LP
23

---

tasks. Pythia-IB IB also performed well, partieularly in TUL tasks with the highest Acc@ 20 (8.39%) GPT-2, while generally performing well, did not achieve top ranks in any specific metric, indicating it may not be the best choice for these tasks compared to other variants.
Ablation Study. The ablation study in Tab. T0 highhlgghs the importance of various components in the model. Removing HTPP (w/o HTPP) led to a noticeable drop in performance across all tasks. For instance, in LP tasks, Acc@ 20 decreased of 7.35% and MRR of 4.69%, % while in TUL tasks, MRR decreased of 6.51 %, underscoring the critical role of HTPP in maintaining high performance.
Removing VIMN (w/o VIMN also resulted in decreased performance, particularly in LP and TUL tasks, where metrics showed a notable average performance degradation of 4.32%.
The absence of PPE (w/o PPE) led to moderate performance degradation, with MRR in LP tasks decreased of 4.19% and in TUL. tasks of 3.76% 7.. The removal of L.MM (w/o L.M) had the most significant negative effect, especially in I.P tasks where A decreased of 17.19% and MRR of 14.69 %. This emphasizes the importance of the LLM component, as its absence significantly hinders the model's overall performance across all tasks.
F
Few Shot Study
The results of 1 % few-shot learning are presented in Tab. 11 and they significantly outperform all baseline methods. We attribute this to the efficient knowledge activation in our reprogrammed LLM. Remarkably, our approach consistently surpasses other competitive baselines across both LP and TUL tasks, further highlighting the potential of language models in proficient human behavior analysis. When compared to recent state-of-the-an models such NSTPP, DSTPP, S2TUL, ReMVC, and CACSR, our average improvements exceed 20.3%, 22.9%, 84.5%, 48.2%, and 41 41.2% respectively, across all metrics. Even with only 1% of the training dataset, our model achieves results comparable to other models using 100% of the training dataset. This is particularly significant for model can effectively understand the distribution
and typically smaller Check-in datasets, patterns of human behaviors with minimal data.
Table 11: Few-shot learning on 1% training data. Red: the best, Blue: the second-best.
Dainsets
Gowalla
WeePlace
Brightkite
FourSquare
Metric
Accil I Acc Acc 20 MRK êcc Acc@ 5 Acc 20 MRR
Acc@1
Acc Accor 20 MRR
Acc Acc 5 Acc MRR
Task
Method
DeepMove
5.03
16.72
33.55
47.07
52.57
40.67
14.65
LightMove
4.68
10.33
14.84
7.40
13:04
28.98
42.41
20.81
33.13
44.57
50.90
39.45
10:05
22.95
32.30
12.92
PLSPL.
5.49
11.98
16.75
8.57
12.38
27.64
41.58
20.03
34.91
46.42
52.71
41.05
12:77
28.48
38.46
16.32
HMTT-LTMM
5.12
11.08
16.20
8.07
11.93
27.03
40.57
19.53
33.08
35.02
50.54
39.65
10.31
23.58
33.07
13.32
LSTPM
4.68
10.32
14.94
7.46
13:60
28.20
40.67
21.15
28.85
38.64
44.55
34.46
11.68
26.78
37.87
15.19
VaSCL
5.35
8.20
12.49
42.
33.87
46.77 76
25.79
36.43
14.66
LP
SinCSE
3.39
8.00
11.73
5.70
9.67
23,86
37.20 37.20
16.81
31.60
45.86
52.22
38.79
11.00
25,97
37.37
14.58
NSTPP
5.12
11.49
16.30
8.30
11.46
25.04
38. 13
18.30
31.04
41.40
47.58
37,21
12.33
27.70
37.79
15.74
DSTPP
12.62
32.74
50.15
39.19
32.09
12.96
ReMVC
5.25
11.32
16.44
8.22
12.96
74
42.72
20.55 20.96
33.58
44.95
51.41
48.18
10.44
24.24
33.52
13.42
SML
4,70
10:33
14,90
7.52
11.92
26.24
39.89
19.36
31.21
41.79
48
36.93
11.12
25.69
36.39
14.68
CACSR
5.21
13.10
6.34
10.70
36.66
17.55
30.13
44.01
36.96
11.13
24.87
36.29
Mobilty-LLM
6.59
13.95
20.37
10.26
15.63
32,39
47.67
23.64
38.62
50.71
58.60
44.61
13.16
29.52
42.38
16.78
TULER
18.54
29.77
36.15
22.95
34:62
$2.05
65.
43.70 70
33.31
59
63.44
40,41
19.
29.78
20
26.89
TULVAE
19.21
31.59
38.26
24.56
40.79
56.07
66.97
46.55 84
16.72
26.06
39.11
21.19
8.60
12.81
18.26
12.56
Movesim
8.70
20.24
28.56
13.93
31.48
46 61
59.52
38.02
27.43
39.39
55.82
33.51
25.72
33.78
42 80
22.71
S2TUL
22.66
34:06
27.61
36.42
43.22
30.79
17.29
25.03
25.57
23.75
29.42
38.06
32.53
TUL
SimCSE
VaSCL
14.98
13.32
26.27
19.51
21.62
34,50
15.75
19.61
29.43
17.27
47.19
24.77
62.34
32.82
21.76
38.01
27.11
21.77
41.51
41.18
$7.68
60.90
34,02
33.01
22.68
17.05
26.60
32.65
43.39
39:51
22,94
28.90
ReMVC
19.50
30.90
24.39
35.72
19.10
51.24
39.41
28,83
33.75
36.37
22.70
SML
18.91
28.
28.68
31.37
24.80
51.54
47.45
53.75
37.44
29.01
40.91
51.54
35.15
13.99
22.74
25.54
30.41
CACSR
17.29
28.73
35.13
22.83
38.01
53.99
66.95
45.77
26.45
41.88
57.79
33.65
21.59
30.92
40.62
29.81
MobilityLLLM
39.28
50.50
44.98
77.66
59.32
46.48
57.77
70.33
52.67
44.69
59.18
The results of 20% few-shot learning are presented in Tab. 12 and they significantly outperform all baseline methods. We attribute this the efficient knowledge activation in our reprogrammed LLM. Remarkably, our approach consistently surpasses other competitive baselines across both LP and TUL tasks, further highlighting the potential of language models in proficient human behavior analysis. When compared to recent state-of-the-an models such as NSTPP, DSTPP, S2TUL, ReMVC, and CACSR, our average improvements exceed 21.4%, 21.8% 86.6%, 46.3%, 3%, and 45. 1 respectively, across all metrics.
24

---

Table 12: Few-shot learning on 20% training data. Red: the best, Blue: the second-best.
Datasets
Gowalla
WeePlace
Brightkite
FourSquare
Acc@ I Acce5 A0c@220MM Accel Acc@ Acc MRR
Accdl
Acc@
Acc
Acc@ Acc MRR
Task
Method
DeepMove
8.19
19.09
27.59
13.10
16.66
33,50
47.76
24.33
44.48
61.15
64.51
52.28
14.67
32.69
44.12
23.06
LightMove
7.67
17.11
24.30
11.89
16.24
33.71
48.66
24.63
43.79
58.49
62.28
51.07
13.12
28.62
39.15
20.31
PLSPI.
8.98
19.82
27.41
11.82
15.65
31.94
47.42
23.85
46.10
60.25
64.50
53.19
16.62
35.66
46.32
25.64
HMT-LSTM
841
12.97
46.50
44.17
13.51
20.78
LSTPM
7,70
17.15
24.50
11.96
16.97
32.58
46.85
25.16
38.33
50.60
34,51
44,66
15.21
33.63
45.85
23.86
VaSCI.
8.75
18.19
26.68
13.11
15.66
33.48
48.
25.64
44.96
61.44 422
63,85
52. 70
14,80
32.42
44.33
22.93
SinCSE
NSTPP
8.45
18.96
26.73
13.29
14.31
28.87
43.
21.72
41.00
53.83
58.39
48.37
16.06
34,65
45.62
24.62
DSTPP
8.42
18.85
27.01
13.17
15.86
32.51
48.
24.16
43.50
57.81
61.05
50.99
13:05
28.73
39.04
20.48
ReMVC
8.55
10
58.57
52.17
40.59
21.16
SML
7,70
17.12
24.75
11.98
15.01
30.47
45. 08
23.05
41.43
$4.45
59.21
47.76
14.49
32,09
44.15
23.02
CACSR
8.50
14.96
21.62
10.13
13.49
27.67
41.61
21.02
39.83
57.58
59.73
48.04
14.54
31,08
43.63
22.33
Mobllity-LI.M
23.01
28.
51,26
66.08
51.36
TULER
30.98
44.60
55.36
35,
45.96
60.55
71
54.71
49.63
6-5 .999
26.
$7.09
23.
37.81
51.72
30.56
31.87
47.57
73.49
Movesim
14.41
30.36
43.83
21.09
41.71
$4.61
64.79
47.60
41.35
54.17
67.45
47.62
31 11.30
43,70
53.66
25.89
SETUL
37.93
51.02
$6.59
42.83
37.18
42.55
47.10
38.32
26.14
34.50
37,84
36.13
29.05
37.54
47.77
37.12
VaSCL
29.33
24.57
57.56
27.58
TUL
SimCSE
24.98
39.28
52.73
30.65
38.99
55.56
68. 14
47.62
41.83
56.92
69.55
47.91
20.77
33.93
49.59
26.03
ReMVC
32.74
46.30
51.24
37.93
47.56
57.12
59.40
49.33
43.78
53.12
58.14
47.68
19.73
32.71
45.78
25.96
31.61
42.97
47.96
38.35
41
43.83
56.67
49.86
29.11
32.13
34.70
CACSR
28.98
43.44
53.54
35.62
50.37
62.81
73 60
57.41
39.92
$7.77
69.33
47,83
26.28
39.81
51.34
33.88
MobilityLLLM
65.55
75.74
70.25
79.75
84.96
69.94
79.86
84.60
74.56
54.39
74.28
60.17
G Feed Forward Layer
We aim to effectively fuse the sequences [z--++1,"" 11 [g] and 18 8, using a gated feed-forward layer inspired by transformers. The fusion mechanism designed to be sophisticated while maintaining the core principles of the feed-forward structure.
Gating Mechanism and Gated Inputs: To enhance the fusion process, introduce gating mech- anisms gx and g, to control the flow of information from input vectors x and z. Then, apply the gating mechanisms the inputs:
= σ W 4= bg.),g. o p. z bb.))
(8)
Xg g.
① x, Zg = g: Oz. z,
where a is the sigmoid activation function, Wg. e xd, b Rd, W.. 9. E Rd xd, and
bg. E Rdd and @ denotes clement-wise multiplication.
Multi-Layer Perceptron (MLP), Layer Normalization, and Residual Connection:
Concatenate the gated inputs and process them through a multi-layer perceptron, apply layer normal- ization to stabilize the training process, and incorporate a residual connection to preserve the original information:
h = ReLReW [x,; ng! + b1), h2 ReLU(W2h, + b2).
(9)
h h LayerNorm (2),, has hporm + [xgi zg],
where W, Rdi((d+dd.) b E Rd,,, W₂ Redad, by Rd Perform final linear transformation to produce the fused output:
h Wyhes + bg.
(10)
where W, E (d.) by Rdd
This design leverages the powerful feed-forward layer structure with enhancements like gating mechanisms, MIPs, layer normalization, and residual connections to achieve a sophisticated and effective fusion of the input sequences [2-r+11, 2.111,2]] 2:] and [s-r++1," ,5--1. s.).
25

---

H
GeoHash Embedding Layer
The Geohash embedding function converts geographic coordinates (latitude and longitude) into Geohash-encoded embedding vector. This involves two primary steps: first, encoding the latitude and longitude into Geohash string, and second, converting this Geohash string into an embedding vector.
Geohash Encoding Given latitude Liat and longitude Lions the Geohash encoding function G maps the coordinates to a Geohash string g:
g = G(Tiat; Iston
(11)
The function G performs the following operations: Interleave the binary representations of latitude and longitude. Encode the resulting binary string into base32 characters to form the Geohash string
Geohash to Embedding Vector Let & be the embedding function that converts Geohash string g into an embedding vector v in Rd:
(g)
(12)
The embedding function & can be implemented using various methods such as lookup table or neural network that maps the discrete Geohash values continuous vector spaces.
Combined Function Combining the two steps, we define the overall Geohash embedding function F:
F(Liot Lion) E(G(Liat: Lion)
(13)
This can be expanded as:
F(Ltat, Iston;
(14)
where first encodes the latitude Laa and longitude Loon into a Geohash string using G, and then maps the Geohash string to embedding vector using E.
I List of POI Categories
Airport, Bank, Bakery, Beach, Bridge, Cafe, Cinema, Clinic, College, Church, Courthouse, Embassy, Firestation, Gym, Harbor, Hospital, Hotel, Library, Market, Mall, Museum, Office, Park. Pharmacy, Pub, Restaurant, School, Stadium, Station, Subway, Supermarket, Theater, University, Zoo, Alley, Aquarium, Arch, Art, Bakery, Bar, Basin, Bay, Bench, Bicycle. Boat, Border, Bowling, Brewery, Buffet, Bungalow, Butcher, Cabaret, Cabin, Canal, Candy, Casino, Castle, Cemetery, Circus, Cliff, Club, Coffeehouse, College, Court, Creek, Cruise, Dam, Dance, Deck, Diner, Dive, Dock, Dorm, Drive, Embassy, Factory, Farm, Fastfood, Ferry, Field, Fishing, Fitness, Fountain, Gallery, Garage, Garden, Gate, Gazebo, Grill, Guesthouse, Hike, Hostel, Ice, Inn, Island, Jail, Kiosk, Lake, Lane, Library, Lighthouse, Mansion, Marina, Meadow, Motel, Monument, Mountain, Nursery, Observatory, Opera, Orchard, Outpost, Palace, Pantry, Pier, Planetarium, Plaza, Pool, Post, Promenade, Pub, Ranch, Recreation, Refuge, Resort, Restaurant, Retreat, Roadhouse, Ruin, RV. Salon, Sanctuary, Sauna, Shel- ter. Shrine, Silo. Ski, Snack, Spa. Speedway, Spring, Square, Statue, Studio, Subway, Swim. Tavern, Temple, Terminal, Track, Trail, Tram, Tunnel, Vineyard, Warchouse, Wharf, Wildlife, Windmill, Winery, Yard, Yoga, Zoo, Abattoir, Aireraft, Amphitheater, Apartment, Arena, Auction, Auditorium, Avenue, Baggage, Barbecue, Bazaar, Boathouse, Buffet, Cafeteria, Campsite, Carpark, Carwash, Carousel, Chapel, Chemistry, Circus, Clinic, Clubhouse, Compound, Conservatory, Convent, Corner, Crematorium, Croft. Deli. Den, Dockyard, Driveway, Enclosure, Estate, Facility, Farmhouse, Festival, Fieldhouse, Firehouse, Florist, Forge, Fountain, Foundry, Gallery, Gasworks, Grotto, Gymnasium, Hall, Hangar, Harbormaster, Heritage, Homestead, Hospice, Hostel, House, Hut, Inn, Jamboree, Jetty, Junction, Kiosk, Laboratory, Lagoon, Lavatory, Library, Lighthouse, Lodge, Lookout, Mill, Mis- sion, Motel, Office, Oratory, Pavilion, Plaza, Platform, Postbox. Range, Refinery, Reserve, Retreat,
26

---

Schoolhouse, Shop, Skatepark, Slope, Stand, Station, Synagogue, Tavern, Teahouse, Terrace, Tower,
Treasury, Villa, Waterslide, Wharf, Workshop.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Abstract and Section Introduction..
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes)
Justification:
See Section 0 Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Section 5 and Appendix E.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification: We have submitted our data and code on
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ccm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.comm http:/pbllcgggudee///ppp Stt.teeeeeoooeeSeeiiiiiiiie
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
See Section 5 and Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
See Subsection 5.5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the Tab. Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppw....ao...ooo/////////a
Answer: [Yes]
Justification: respects.
We ensured that the study complied with the NeurlPS Code of Ethics in all
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not suffer from this risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Ans
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprssiiiitt. thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
32

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "QT7NWS9rvt", "reviewer": "Reviewer_NycQ", "summary": "This paper aims to leverage large language models (LLMs) to predict human mobility behavior recorded in social media check-ins. The authors argue existing models fail short to model the visiting intentions and travel preferences embedded in check-in sequences, which could be addressed by the semantic understanding capabilities of LLMs. The authors propose to use visiting intention memory network (VIMN) to encode the visit intention in each check-in record, which is feed into the LLMs along with the shared pool of human travel preference prompts (HTPP). The employed LLMs are trained with Low-Rank Adaptation (LoRA) algorithm. The proposed method is evaluated on four benchmark datasets with the tasks of next location prediction, trajectory user link and time prediction.", "questions": "1. Please provide the statistical significance of the performance gains.\n2. Please provide a cost analysis of the proposed method.\n3. Please explain the design choice of visiting intention memory network. \n4. Please discuss the limitation of different data sizes in more detail.", "limitations": "The limitations are discussed in the paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed Mobility-LLM framework is well motivated and clearly illustrated.\n2. The proposed method is evaluated on open datasets, and the source code is made publicly available.\n3. The authors provide comprehensive experiment results of their model, including the analysis of few-shot learning, language model variants, ablation study and hyperparameter sensitivity.", "weaknesses": "1. The performance gain is not substantially enough. The main results in Table 1 show the performance improvement for next location prediction is often less than 1%. Moreover, it is unclear if the performance gain is statistically significant.\n2. The proposed Mobility-LLM framework leverages large language models for mobility prediction, which have substantially larger model size compared to other deep learning baselines. The authors should provide a cost analysis (e.g. complexity of training and inference) to discuss whether the performance gain justifies additional cost.\n3. The model design of visiting intention memory network (VIMN) is not adequately explained. The authors propose to use LLM to capture the visiting intention and travel preferences in check-in sequences, but then they claim these semantics information cannot be directly interpreted by LLM, which needed to be reprogramed by VIMN. This design seems a bit self-contradictory and VIMN needs better justification.\n4. In the Limitation section, the authors mention their Mobility-LLM is hindered by the differing user and POI counts in check-in data. This is an important observation and it should be discussed with experiment results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive comments.\n\n### **[W1&Q1]**\n\nIn the paper, we ran each set of experiments 5 times and reported their mean values. Therefore, the reported results are not single accidental one. Below, we also report the variances of metrics for the proposed model and the best baseline on the Location Prediction task, along with the corresponding t-value measuring statistical significance.\n\n|Datasets|Gowalla||||WeePlace||||Brightkite||||FourSquare||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Metric (e-2)|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|\n|The Best Baseline|11.47±0.03|24.12±0.17|33.82±0.15|17.44±0.05|19.66±0.04|37.68±0.18|53.44±0.13|28.46±0.06|51.42±0.12|66.25±0.08|71.46±0.11|57.59±0.14|16.92±0.04|36.05±0.04|49.39±0.16|26.02±0.04|\n|Mobility-LLM|11.87±0.04|25.14±0.04|36.36±0.09|18.29±0.11|20.47±0.13|39.22±0.19|56.69±0.10|29.21±0.15|53.18±0.17|68.31±0.14|74.11±0.18|59.89±0.03|17.29±0.03|37.17±0.02|53.16±0.20|26.47±0.05|\n|t-value|29.85|13.40|37.84|37.95|45.25|19.13|55.95|27.99|32.77|57.54|53.97|36.75|20.67|62.57|52.63|10.07|\n\nThe formula for calculating t-value is:\n\n$t = \\frac{\\mu_{\\text{Mobility-LLM}} - \\mu_{\\text{Best Baseline}}}{\\sqrt{\\frac{\\delta_{\\text{Best Baseline}}^2}{n}}}$\n\nThe standard deviation is that of the best baseline, with (n = 5). We use a significance level of $\\alpha = 0.05$ and a two-tailed t-test with degrees of freedom $d_f = 4$. For this t-distribution, the critical t-value is approximately 2.776. If the calculated t-value is greater than 2.776, the difference is considered significant. As shown in the table, all the calculated t-values are well above the critical value of 2.776.\n\n### **[W2&Q2]**\n\nBelow, we provide an empirical performance and efficiency comparison between the proposed model and the optimal baseline for LP and TUL tasks.\n\n**Performance comparison for LP task:**\n\n|Datasets|Gowalla||||WeePlace||||Brightkite||||FourSquare||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Metric (e-2)|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|\n|PLSPL|11.26|24.12|33.82|17.44|18.77|37.31|53.31|27.86|51.42|65.34|71.46|57.59|13.73|30.65|43.18|21.42|\n|Mobility-LLM|11.87|25.14|36.36|18.29|20.47|39.22|56.69|29.21|53.18|68.31|74.11|59.89|17.29|37.17|53.16|26.47|\n|Improvement (%)|5.42%|4.23%|7.52%|4.87%|9.06%|5.12%|6.34%|4.84%|3.42%|4.54%|3.71%|3.99%|26.00%|21.25%|23.13%|23.60%|\n\n**Efficiency comparison for LP task:**\n\n|Method|Memory|Training Time|Inference Time|Training Epoch|\n|-|-|-|-|-|\n|PLSPL|3.4GB|4.73h|2.77m|79|\n|Mobility-LLM (TinyLlama)|11.3GB|5.45h|6.69m|17|\n\n**Performance comparison for TUL task:**\n\n|Datasets|Gowalla||||WeePlace||||Brightkite||||FourSquare||||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Metric (e-2)|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|Acc@1|Acc@5|Acc@20|MRR|\n|ReMVC|68.75|74.4|73.19|70.02|65.78|73.09|71.64|66.15|73.85|82.55|87.93|77.93|58.18|66.84|72.67|65.14|\n|Mobility-LLM|80.43|86.29|88.56|83.18|79.03|88.04|91.48|83.21|83.06|88.52|90.35|85.73|72.08|79.67|84.32|75.71|\n|Improvement (%)|16.93%|15.99%|20.99%|18.81%|20.18%|20.47%|27.69%|25.76%|12.48%|7.24%|2.75%|9.99%|23.87%|19.15%|16.03%|16.19%|\n\n**Efficiency comparison for TUL task:**\n\n|Method|Memory|Training Time|Inference Time|Training Epoch|\n|-|-|-|-|-|\n|ReMVC|4.6GB|2.77h|1.49m|45|\n|Mobility-LLM (TinyLlama)|11.7GB|3.65h|6.98m|13|\n\nWe observe that our model increases memory usage due to the inclusion of LLM. However, since our model uses a pre-trained language model, its convergence speed is much faster than other baseline models. \n\nOn the high-quality Foursquare dataset, our model shows an improvement of over 21% compared to PLSPL, with the Acc@1 metric reaching up to 26%. Additionally, our model's advantage is more pronounced in the few-shot scenario. Therefore, the total training time is justified compared to traditional models.\n\nIn summary, we believe the increased cost is worthwhile for the performance improvement.\n\n### **[W3&Q3]**\n\nLLM is designed to understand semantic information in natural language but not to interpret information in check-in sequences. Therefore, VIMN is proposed to reprogram the check-in sequence. This way, the output of VIMN provides semantic information aligned with natural language, allowing the LLM to process it.\n\n### **[W4&Q4]**\n\nSorry for causing misunderstandings. What we are trying to discuss in the Limitation section is that the sets of POIs in different datasets (which usually cover different regions) are unique. Therefore, if the proposed model is trained on one dataset, its learned information about the set of POIs is not easily transferable to another dataset. Different sets of POIs have different functionalities and usually have a different number of POIs, making many modules (such as embedding and predictor) technically untransferable in a zero-shot setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jB6CPdK60c", "reviewer": "Reviewer_wrzL", "summary": "This paper combines large language models (LLMs) to better analyze check-in sequences and understand human mobility behaviors, and proposes  a visiting intention memory network(VIMN) and a shared pool of human travel preference prompts (HTPP) to capture the semantics of human visiting intentions . The experiments show promising improvement. It is a good work in both theoretical and experimental perspectives.", "questions": "1. In the HTPP module, if there is possible that the k prompt words selected in each domain have semantic contradictions?\n2. When doing TUL tasks, are user embeddings also passed into LLM as part of the input? User information should not be included in the input.\n3. In section 5.5, why not experiment with the GPT-3.5 as a language model variant?As a large language model that performs well in various fields, is it possible to achieve better results by using GPT-3.5 as the backbone network?", "limitations": "When doing downstream tasks in this article, different parts of the LLM output are used for different tasks. What are the design considerations for this part? I concerned what should I do if I want to apply this model to other downstream tasks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is interesting to apply large language models and neural networks to capture the latent semantic information of check-in sequences.\n2. This paper is well-organized, which clearly explains the research purpose, methods and conclusions.\n3. The paper has done a lot of experiments, and the experimental results show that the method proposed in this article outperforms state of-the-art methods on three downstream tasks.", "weaknesses": "1. In the figure1, it’s not clear how the user embedding U_i\\ is obtained?It is best to explain it in the article.\n2. There are some inconsistencies in the writing of this article. For example, in Section 5.1, the LP task uses the MLE loss, but in Appendix B, the LP task uses the cross-entropy loss.\n3. The model training part in the article is not detailed enough. For example, what is the parameter L_F when fine-tuning LLM? Are the three downstream tasks trained separately or uniformly?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive comments.\n\n### **[W1]**\n\nThe user embedding $U_i$ is an index-fetch embedding module implemented using the nn.Embedding module in PyTorch. It finds the corresponding embedding vector for a given user index.\n\n### **[W2]**\n\nSorry for the confusion. We want to clarify that these two expressions are actually equivalent in our implementation, and we will ensure consistency and standardize the terminology thanks to your suggestions. Below is a detailed explanation of why these two losses are equivalent in our case.\n\n> Cross-entropy loss is a method used to measure the difference between two probability distributions, commonly used in classification problems. It is often used to train neural networks, especially in classification tasks.\n> \n> \n> Assume we have a classification task where the true label distribution is $y$ and the predicted probability distribution is $\\hat{y}$. The cross-entropy loss is defined as:\n> \n> $H(y, \\hat{y}) = - \\sum_{i=1}^n y_i \\log \\hat{y}_i$\n> \n> For a binary classification problem, the cross-entropy loss can be simplified to:\n> \n> $H(y, \\hat{y}) = - (y \\log \\hat{y} + (1 - y) \\log (1 - \\hat{y}))$\n> \n> In classification problems, if we assume that the probability distribution $\\hat{y}$ predicted by the model is obtained through the softmax function, then maximizing the log-likelihood function is equivalent to minimizing the cross-entropy loss.\n> \n> Specifically, assume we have a neural network for a classification task, and the output layer uses the softmax activation function to convert the model output into category probabilities. For each sample $x_i$, its true label is $y_i$, and the probability predicted by the model is $\\hat{y}_i$. Maximizing the log-likelihood function:\n> \n> $\\ell(\\theta) = \\sum_{i=1}^n \\log P(y_i|x_i; \\theta) = \\sum_{i=1}^n \\log \\hat{y}_i$\n> \n> is equivalent to minimizing the cross-entropy loss:\n> \n> $H(y, \\hat{y}) = - \\sum_{i=1}^n \\log \\hat{y}_i$\n> \n> Therefore, in this case, minimizing the cross-entropy loss when training the model is actually performing maximum likelihood estimation.\n> \n\n### **[W3]**\n\nAs stated in line 203, we mentioned that different parameter freezing strategies are applied to the first $1 - L_F$ layers and the last $L_F - L_{F+U}$ layers. Here, $L_F$ is the number of frozen layers in the middle. The three tasks are trained separately.\n\n### **[Q1]**\n\nWe believe that in most cases, the selected $K$ words from the model will not contradict each other. This is because contradictory words are semantically different, meaning their distances in the embedding space are large. Thus, the score-matching function used in the HTPP module will rarely match contradictory words simultaneously.\n\n### **[Q2]**\n\nAs mentioned in line 225, the user embedding will not be used as input when performing the TUL task.\n\n### **[Q3]**\n\nOur method requires full access to the implementation code and pre-trained parameters of an LLM to fine-tune some of the parameters. Since GPT-3.5 is not open-sourced and only limited functionalities can be accessed through the internet interface provided by OpenAI, GPT-3.5 cannot be used as the foundation for the proposed method.\n\n### **[Limitations]**\n\nMost downstream tasks can be performed by attaching prediction modules (usually implemented with MLP) to the output latent vectors of LLM. For example, in our implementation, we use classification prediction modules for the LP and TUL tasks, and a regression prediction module for the TP task.\n\nRegarding the choice of attachment position, an intuitive design is to attach it to the position corresponding to the most relevant information for the task. Experimental evaluation is also needed to determine the optimal position and implementation details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TbIXOb1ol7", "reviewer": "Reviewer_Mti9", "summary": "The paper proposes a novel architecture, MobilityLLM, for utilizing pre-trained LLMs with various specific embedding modules for various mobility tasks. Specifically, the authors proposed to use any pretrained LLM with adding four unique components: PPEL (POI location embedding), VIMN (timestamp embedding with GRU), HTPP (user travel preference prompting), and Multi-task training strategy. MobilityLLM finds a unique place in LLM-based human mobility modeling applications by incorporating spatiotemporal aspects of mobility characteristics using learnable components (neural networks). The experiments are tested on three different tasks: location prediction, trajectory user-link, and time-prediction. The proposed model performs consistent outperformance compared to benchmarks.", "questions": "1. Can you elaborate on the statement at the end of the methodology section, \"We use the β and α with a mean pooling projection head to predict the user uˆ who generates this check-in sequence?\"\n\n2. While the metrics mainly evaluate the performance in terms of piece-wise similarities, it would be interesting to see distributional similarities as well, such as the JS similarity metric, which can be employed.", "limitations": "LLMs are powerful machines but the proposed model is right now specific to one dataset. It would be interesting to explore the combination of diverse scenarios and datasets.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Here is the set of strengths of the paper:\n1. It introduced several novel components that can be embedded in any LLM for mobility-specific applications.\n2. The three main components, PPEL, VIMN and HTTP greatly enhances the performance of LLMs on downstream mobility tasks. \n3. MobilityLLM has a lightweight architecture with LoRA trainable weights and makes it an accessible tool.\n4. The experiments supports promising results on different datasets. \n5. Lastly, appending location embedding (GeoHash) to the attention layer is a very clever way of dealing with locations.", "weaknesses": "* The paper is relatively well written; however, there are multiple typos and abbreviation errors throughout the paper.  Here the two typo examples: (reference error in table 4. Instead of Table 3 it references Table 1) and (Figures 3 and 4 have typos: HIBP -> HTTP, IIMN -> VIMN)\n* While it is designed more on check-in datasets, I wonder how applicable the model is to trajectory generation tasks. \n* Figure 1 is informative but very complicated to read without the methodology section. \n* I am aware of the page limitation, but some important details are not mentioned in the main body and are left to appendixes. For instance, few-shot learning(prediction) requires a more detailed explanation; it is not intuitively clear in the main body. \n* The Related Works section is fragile in terms of the compared baselines. They are not discussed very well in the main body and appendix. For instance, CACSR is not even discussed in both places.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive comments.\n\n### **[W1]**\n\nThank you for pointing out these errors. We have made the corrections and checked other parts of the paper for similar issues.\n\n### **[W2]**\n\nCompared to trajectory data (such as vehicle trajectories), check-in data are recorded more sparsely. Each point in a trajectory might record a passing-by, while each point in a check-in sequence corresponds to a purposeful visit to a POI. To achieve optimal performance on trajectory data, some modules of the proposed methods may need to be redesigned to fit the characteristics of trajectories.\n\n### **[W3]**\n\nThank you for your suggestions. We have adjusted the overall framework diagram to highlight key modules and provided detailed diagrams of each module as separate illustrations.\n\n### **[W4]**\n\nThank you for your suggestions. We will supplement the implementation settings for the few-shot scenario. In this scenario, we first divide the entire dataset into training, validation, and test sets in a 6:2:2 ratio based on the number of check-in sequences. Then, we only reduce the training set portion to 20%, 5%, and 1% of the original training set, while keeping the validation and test sets intact.\n\n### **[W5]**\n\nSorry for the oversight. We have added all the baseline models in the main text and appendix and included detailed discussions in the appendix.\n\n### **[Q1]**\n\nThe dimension of $\\alpha$ is $(B,n,d)$, and the dimension of $\\beta$ is $(B, 3, d)$. First, we concatenate along the second dimension, resulting in a dimension of $(B,n+3,d)$. Then, we average pool along the second dimension to obtain $(B,1,d)$. Finally, after a fully connected layer and softmax operation, we obtain the predicted user. Here, $B$ represents batch size, $n$ represents sequence length, and $d$ represents embedding dimension.\n\n### **[Q2]**\n\nWe would like to point out that JS divergence is more commonly used for tasks where the output is a sequence. Our model's prediction task only predicts a single future point, so we choose the current evaluation metrics, which are also widely used in baselines and most existing works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUxG143LsT", "reviewer": "Reviewer_6rUV", "summary": "This submission presents a framework that leverages a large language model (LLM) as the backbone architecture for human-mobility-related tasks, including user identification, next location prediction, and arrival time prediction. The authors introduce a POI (Point of Interest) Point-wise Embedding Layer and a Visiting Intention Memory Network as an encoder to process check-in data. Additionally, they establish a behavior prompt pool to incorporate behavior preference information with the embeddings from the Visiting Intention Memory Network. The framework is evaluated using four real-world datasets, and the proposed model outperforms baseline methods in terms of most metrics.", "questions": "1.\tThe reviewer suggests that the HTPP is the most significant idea in this study. However, However, several concerns need to be addressed: (i) The use of cos-similarity based scoring function to determine the significance score needs more illustration. Why is cos-similarity sufficient to evaluate the semantic similarity in urban mobility tasks? What is the impact of choice of the scoring function? (ii) What is the motivation of the selected domains as well as the prompt words? Are there any theoretical support? \n2.\tThis paper essentially treats each domain separately during top-k prompts selection. The reviewer suggests that it is the combination of prompts of each domain that presents meaningful and promising semantic information, i.e., a doctor (Domain 2) goes to exercise (Domain 1) to keep healthy (Domain 3).", "limitations": "The authors have discussed the limitations adequately.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tOverall, this paper is well-written and well organized. The authors illustrate the motivation of this study sufficiently and review the related work thoroughly.\n2.\tThe proposed framework finetunes the LLM using semantic data, which is novel in human mobility related tasks.\n3.\tThe experimental studies are thorough and the hyperparameter analysis is well presented.", "weaknesses": "1.\tWhile the framework offers a novel approach to addressing human mobility-related tasks, the robustness of the modules is questionable, particularly in the formulation of the human travel preference prompt. Additionally, the overall technical contribution is limited.\n2.\tThe paper provides no discussion of choice of scoring function.\n3.\tThere is a lack of novelty in PPEL and HTPP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive comments.\n\n### **[W1&W2&Q1]**\n\nWe design the VIMN module to reprogram the check-in sequences. After passing through the VIMN module, the check-in sequences are aligned to the natural language semantic space, producing the resulting $\\mathbf{h}_i$*.* This ensures that the two representation vectors sent to the HTPP module are in the same semantic space. We believe that in the same semantic space, vectors that are close in distance have similar semantics. Therefore, if the cosine similarity calculated by the score-function is high, it indicates that $\\mathbf{h}_i$ itself has semantic relevance in the semantic representation space. In summary, the key to evaluating the semantic similarity in urban mobility tasks lies in the design of the VIMN reprogramming module, with cos-similarity only serving to compare the similarity of vectors in the same semantic space.\n\n### **[W2]**\n\nWhile the PPEL and HTPP modules leverage existing technologies from a technical perspective, our design principles are innovative and introduced for the first time in tasks related to human activities. These modules are integral, as they employ LLM's word token embeddings for both check-in sequence embeddings and prompt tokens, achieving semantic alignment and integration with other modules. Ablation studies have shown that these modules significantly enhance performance, thereby validating the efficacy and rationality of our design.\n\n### **[Q2]**\n\nWe considered the scheme you originally mentioned. However, listing all combinations creates a very large space, reducing the model's efficiency. Manually screening reasonable combinations will cost a huge amount of human resources and may fail to cover all possibilities due to the diversity of human activity data and the many situations that can arise. Additionally, case studies show that the existing scheme is superior in efficiency and accuracy compared to the one you mentioned.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces MobilityLLM, a novel architecture designed to enhance the analysis of human mobility behaviors using large language models (LLMs). The proposed approach includes the Visiting Intention Memory Network (VIMN) to encode visit intentions in each check-in record and a shared pool of Human Travel Preference Prompts (HTPP) to capture the semantics of these intentions. The LLMs are then fine-tuned using the Low-Rank Adaptation (LoRA) algorithm. Evaluations on four benchmark datasets for tasks such as next location prediction, trajectory user linking, and time prediction demonstrate promising performance. The paper has received generally positive reviews, and the authors have addressed most reviewers' concerns. Given that this is the first application of reprogramming techniques to LLMs for understanding human mobility, I recommend its acceptance.
Decision: Accept (poster)