Uncertainty-aware Human Mobility Modeling and Anomaly Detection

## Abstract

Abstract Given the GPS coordinates of a large collection of human agents over time, how can we model their mobility behavior toward effective anomaly detection (e.g. for bad-actor or malicious behavior detection) without any labeled data?
Human mobility and trajectory modeling have been studied extensively with varying capacity to handle complex input, and performance-efficiency trade-offs. With the arrival of more expressive models in machine learning, we attempt to model GPS data as a sequence of stay-point events, each with a set of characterizing spatiotemporal features, and leverage modern sequence models such as Transformers for un/self-supervised training and inference. Notably, driven by the inherent stochasticity of certain individuals’ behavior, we equip our model with aleatoric/data uncertainty estimation. In addition, to handle data sparsity of a large variety of behaviors, we incorporate epistemic/model uncertainty into our model.
Together, aleatoric and epistemic uncertainty enable a robust loss and training dynamics, as well as uncertainty-aware decision making in anomaly scoring.
Experiments on large expert-simulated datasets with tens of thousands of agents demonstrate the effectiveness of our model against both forecasting and anomaly detection baselines. All code is available at https://anonymous.4open.science/r/mobility-ad .

## 1 Introduction

Anomaly detection of human mobility has become crucial for various applications, ranging from security and surveillance to health monitoring. Accurately detecting anomalous human behavior from GPS data can reveal critical insights, such as abnormal patterns that indicate security threats or spread of infectious diseases Meloni et al. (); Barbosa et al. (); Stanford et al. ().

However, the complexity and inherent uncertainty of human behavior make this task particularly challenging: (1) The first difficulty lies in capturing the complex spatiotemporal dependencies both within individual activities and between multiple activities across different times and locations.
The former aims to capture the correlations between different spatiotemporal features (a.k.a. markers) of a single activity; such as the time-of-day (e.g. 4am) being indicative of POI location (e.g. home).
The latter associates with the dependency patterns between different activities over time (e.g. restaurant at weekday lunchtime followed by office building). (2) Furthermore, human activity data is abound with uncertainty—arising from the unpredictable behavior of inherently stochastic individuals as well as data sparsity—making accurate anomaly detection even more difficult.

Prior work on behavior modeling often focused on temporal event forecasting, either ignoring spatial information Minor et al. (); Manzoor and Akoglu () or based only on a few markers Du et al. (); Zhou et al. (). With the emergence of the Transformer architecture Vaswani et al. (), more recent work modeled complex spatiotemporal events more expressively Xue et al. (); Corrias et al. (). However, to our knowledge, no existing work considered uncertainty-aware anomaly detection of human mobility.

To address both of the above challenges, we introduce UIFormer for uncertainty-incorporated human behavior modeling and anomaly detection, which takes advantage of a “dual” Transformer architecture Truong Jr and Bepler () as well as aleatoric and epistemic uncertainty into account.
Transformer has emerged as the de facto model for modern AI problems, due to its superior ability to model long-range dependencies. Through a dual architecture equipped with both feature- and event-level attention, we address the first challenge in capturing the intra- and inter-event dependencies.

For the second challenge, we first account for data (a.k.a. aleatoric) uncertainty, which stems from the inherent stochasticity in certain human behavior. For example,
it is typical of some populations (e.g. retired individuals) to be less predictable relative to some others (e.g. shift workers with a strict schedule).
In addition to data uncertainty, we incorporate model (a.k.a. epistemic) uncertainty, which typically stems from incomplete or sparse observations from a potentially complex underlying data generating distribution. Explicitly modeling both aleatoric (data) and epistemic (model) uncertainty benefits UIFormer in two ways: First, it factors into the learning objective, enabling a more robust training.
Second, it allows the model to reason effectively about the confidence of its predictions, thus enabling more reliable inference.
Specifically, while it is typical for forecasting-based detection methods to flag anomalies based on the deviation between the predicted and observed values alone, our uncertainty-aware UIFormer considers both the predicted values as well as total estimated uncertainties, thus providing a more nuanced and accurate approach to anomaly detection.

The following summarizes the main contributions of this work.

- •
Problem formulation: We cast the anomaly detection problem in human mobility onto sequence prediction, where we translate the regular GPS readings (of the form $\langle$x,y, $t\rangle$) into irregular stay-point events over time with spatiotemporal features. This step resembles a form of “tokenization” of the raw input data to a higher-level representation for sequence modeling.
- •
Uncertainty-aware dependency modeling:
We design UIFormer, a dual Transformer architecture that is uncertainty-aware;
which is equipped with ($i$) both feature-level and event-level attention—for capturing dependencies among features as well as across events, respectively; and ($ii$) both data and model uncertainty estimation—toward accounting for inherent stochasticity in human behavior as well as data scarcity, respectively.
- •
Uncertainty-aware anomaly scoring: Beyond prediction, we extend our uncertainty modeling to anomaly detection, which allows the model to flag abnormal behavior with greater accuracy by factoring in both prediction value and uncertainty estimates.
- •
Effectiveness: Through extensive experiments on expert-simulated benchmark datasets, we show that UIFormer significantly outperforms both forecasting as well as anomaly detection baselines. Ablation analyses demonstrate UIFormer’s key advantages: accurate uncertainty estimates and effective uncertainty-aware anomaly scores.

## 2 Problem and Preliminaries

We introduce related concepts and formulate the anomaly detection problem on human mobility.

Figure: Figure 1: Spatiotemporal event sequence with various features (a.k.a. markers) per event.
Refer to caption: x1.png

Stay-point Event. As shown in Figure 1, a stay-point event is extracted from raw GPS data that is used to describe an individual’s daily activity. Let $e_{i}$ be an event with spatiotemporal features (or markers) $\bm{x}_{i}$:

$$ ${\bm{x}_{i}}=(x_{i}^{\rm st},x_{i}^{\rm sd},x_{i}^{\rm x},x_{i}^{\rm y},x_{i}^ {\rm poi},x_{i}^{\rm dow})\;,$ (1) $$

where $\mathcal{F}_{n}=\{x_{i}^{\rm st},x_{i}^{\rm sd},x_{i}^{\rm x},x_{i}^{\rm y}\}$ is the numerical feature set: $x_{i}^{\rm st}$ and $x_{i}^{\rm sd}$ are the start time and stay duration of the event, $x_{i}^{\rm x},x_{i}^{\rm y}$ are the two-dimensional coordinates depicting
the latitude and longitude of the event’s location. $\mathcal{F}_{c}=\{x_{i}^{\rm poi},x_{i}^{\rm dow}\}$ is the categorical feature set, depicting the Point-of-Interest (POI) such as office building, store, etc. and Day-of-Week (DOW) of the event, respectively.

Spatiotemporal Event Sequence. Let ${\mathcal{E}}^{u}=[e_{1}^{u},e_{2}^{u},\dots,e_{N_{u}}^{u}]$ be the event sequence that records all historical events of individual $u$ in the dataset, where $N_{u}$ is $u$’s total number of historical events. Furthermore, let $e_{i}^{u,d}$ be the $i$-th event of individual $u$ at day $d$, and $N_{u,d}$ be the number of events of $u$ on day $d$. The event sequence with $w$-day time window can be given as

$$ ${\mathcal{E}}^{u}_{w}=[e_{1}^{u,d-w},e_{2}^{u,d-w},\dots,e_{1}^{u,d-w+1}, \ldots,e_{1}^{u,d},\dots,e_{N_{u,d}}^{u,d}]\;.$ (2) $$

Unsupervised Human Mobility Anomaly Detection. Given a individual $u$’s event sequence in a $w$-day window ${\mathcal{E}}_{u}^{w}$ and a target event $e\in{\mathcal{E}}_{u}^{w}$, we aim to learn an anomaly score function to identify whether $e$ is anomalous, i.e. not aligned with $u$’s typical behavior pattern, without any labeled data.

## 3 Proposed Model

Figure 2 illustrates the overall architecture of UIFormer,
which learns human mobility patterns in a self-supervised manner through pre-training, with the goal of accurately reconstructing masked events as well as evaluating the prediction uncertainty. In a nutshell, the feature tokenizer first projects features into high-dimensional space by representing each event’s feature as an individual token (§3.1). Then, the Dual-Transformer encoder further encodes the input by both feature-level and event-level self-attention (§3.2). Lastly, unlike previous work, we introduce the uncertainty-aware decoder to recover all features of the masked event along with the aleatoric and epistemic uncertainty associated with each feature (§3.3). These estimates are used to compare the predicted versus observed events at inference time for uncertainty-incorporated anomaly scoring (§3.4).

Figure: Figure 2: Proposed model architecture for human mobility behavior modeling. Raw GPS data is represented as a (ordered) sequence of stay-point events each with a (unordered) sequence of spatiotemporal markers. Dual-Transformer architecture models each individual’s sequence-of-sequences via both feature- and event-level attention, as well as uncertainty estimation that simultaneously enables robust training and inform anomaly scoring at inference.
Refer to caption: x2.png

### 3.1 Feature Tokenizer

The feature tokenizer transforms all features ${\bm{x}}\in{\mathbb{R}}^{F}$ of an event into embeddings ${\bm{e}}\in{\mathbb{R}}^{D}$, where $F$ is the number of input features and $D$ is the embedding dimension. Specifically, a numerical feature $x^{(num)}_{j}$ is projected by a linear transformation with weight ${\bm{W}}^{(num)}_{j}\in\mathbb{R}^{D}$ and bias ${\bm{b}}_{j}\in\mathbb{R}^{D}$, and the embedding of a categorical feature is implemented as the embedding lookup table ${\bm{W}}^{(cat)}_{j}\in\mathbb{R}^{C_{j}\times D}$, where $C_{j}$ is the total number of categories for feature $x_{j}^{(cat)}$. Overall:

$$ $\displaystyle\bm{e}^{(num)}_{j}={\bm{b}}^{(num)}_{j}+x^{(num)}_{j}\cdot{\bm{W} }^{(num)}_{j}$ $\displaystyle\in\mathbb{R}^{D}\;,$ $\displaystyle\bm{e}^{(cat)}_{j}={x}^{(cat)}_{j}{\bm{W}}^{(cat)}_{j}$ $\displaystyle\in\mathbb{R}^{D}\;,$ $\displaystyle\bm{e}=\mathtt{stack}\left[\bm{e}^{(num)}_{1},\ \ldots,\ \bm{e}^{ (num)}_{k^{(num)}},\ \bm{e}^{(cat)}_{1},\ \ldots,\ \bm{e}^{(cat)}_{k^{(cat)}}\right]$ $\displaystyle\in\mathbb{R}^{F\times D}\;.$ $$

where ${x}^{(cat)}_{j}$ is represented as a one-hot vector. The feature embedding ${\bm{e}}\in\mathbb{R}^{F\times D}$ of one event is the concatenation of all numerical embeddings and categorical embeddings. Denoting $B$ as the batch size and $L$ the max length of ${\mathcal{E}}^{u}_{w}$ in a batch, the output of the feature tokenizer is $\bm{E}_{0}\in\mathbb{R}^{B\times L\times F\times D}$.

### 3.2 Dual Transformer Encoder

Inspired by (Truong Jr and Bepler, ), we design a Dual Transformer encoder that treats the data as a sequence-of-sequences,
which takes the sequence of events each with a sequence of feature embeddings as input, and models both feature-level and event-level interactions with two types of Transformer-based (Vaswani et al., ) components. A description of the Transformer block is included in Appendix B.

Feature-level Transformer. By considering each feature as an input token to the Transformer, the Feature-level Transformer represents a feature in a way that explicitly incorporates other features’ information. In the implementation, we convert the feature tokenizer’s output $\bm{E}_{0}\in\mathbb{R}^{B\times L\times F\times D}$ into $\bm{E}_{1}\in\mathbb{R}^{(B\times L)\times F\times D}$, which is then fed into $M_{1}$ Transformer blocks to get the updated feature embeddings $\bm{E}_{2}\in\mathbb{R}^{(B\times L)\times F\times D}$. Note that we do not include any positional embedding as input for this module as there is no sequential relationship between the features of an event.

Figure: Figure 3: Within-day position encoding.
Refer to caption: x3.png

Event-level Transformer.
Based on $\bm{E}_{2}$, we calculate an event’s embedding by averaging all its corresponding feature embeddings, which results in the input of the Event-level Transformer $\bm{E}_{3}\in\mathbb{R}^{B\times L\times D}$. The Event-level Transformer then captures the dependencies (e.g., sequential patterns) between different events by considering each event as a token. Moreover, considering that the sequential information in the event sequence is important to reflect human mobility behavior, we design two types of positional encoding for each event: ($i$) Sequence positional encoding; used to describe the order of an event in the input sequence, and ($ii$) Within-day positional encoding (as shown in Figure 3); used to describe the order of the event in its specific day, explicitly demarcating day boundaries. After $M_{2}$ updates of the Event-level Transformer blocks, we get the final output of the encoder $\bar{\bm{E}}\in\mathbb{R}^{B\times L\times D}$.

### 3.3 Uncertainty-aware Decoder

Motivation.  Uncertainty modeling has shown great promise in various fields, such as computer vision (Kendall and Gal, ; Kendall et al., ) and natural language processing (Xiao and Wang, ; Gal and Ghahramani, ). In particular, human mobility sequences are inherently associated with uncertainty. For instance, certain individuals (e.g., shift workers) may exhibit highly predictable patterns (i.e., low uncertainty), while others (e.g., retirees) can be much more unpredictable in their activities over time. This type of uncertainty, known as aleatoric uncertainty (or data uncertainty) (Kendall and Gal, ), arises naturally from the data itself and cannot be reduced by simply adding more training data. In addition to aleatoric uncertainty, our setting also involves epistemic uncertainty, which refers to the uncertainty in the model itself due to limited knowledge. This type of uncertainty captures the model’s confidence in its own predictions and can be reduced by acquiring more data or improving the model specification or architecture. In human event sequence modeling, epistemic uncertainty is crucial for identifying situations where the model lacks sufficient information to make confident predictions, especially in underrepresented or complex scenarios. Furthermore, given that different features can have different predictability even for the same individual/event, we design a feature-level uncertainty learning mechanism. In the following part, we elaborate on how we model the epistemic and aleatoric uncertainty for both numerical and categorical features.

#### 3.3.1 Epistemic uncertainty Modeling

Inspired by (Kendall, ), we model the epistemic uncertainty by placing a prior distribution over the model’s parameters, and then try to capture how much these parameters vary given observed data.

To ease the presentation, we use $s$ to denote the input instance ${\mathcal{E}}_{w}^{u}$, and $\theta$ to denote all parameters of the model $\mathcal{M}$. We start by assuming $\theta$ follows a prior distribution, which reflects our initial belief about the possible values of these parameters. The goal is to update this belief based on the available data $s$ by calculating the posterior distribution of $\theta$, that is $p(\theta|s)$. This posterior captures how the parameters might vary, thereby reflecting the model’s uncertainty about its predictions. Formally, Bayes’ theorem (Bayes, ) gives us the posterior: $p(\theta|s)=\frac{p(s|\theta)p(\theta)}{p(s)}$.

However, computing the exact posterior distribution in deep learning models is often intractable due to the high dimensionality of the parameter space and the complexity of the model. As a practical alternative, we apply Monte Carlo (MC) Dropout (Gal and Ghahramani, ) to approximate the posterior. In practice, we introduce dropout layers after the embedding layer of each feature with a dropout ratio of 0.05, which randomly drops a subset of neurons during both training and inference. MC Dropout works by sampling different subsets of model parameters $\theta$ during each forward pass, thereby creating an ensemble of models. The variance in the predictions across these passes provides an estimate of the epistemic uncertainty.

Epistemic uncertainty for numerical features:  The decoding of numerical feature $f\in\mathcal{F}_{n}$ is modeled as a regression task. In this case, the variance of the predictions across $T$ (equals 50 in our model) steps of stochastic forward passes quantifies the epistemic uncertainty $\alpha_{f}^{num}$ for $f$, formulated as

$$ $\alpha_{f}^{num}:\text{Var}(y)=\frac{1}{T}\sum_{t=1}^{T}\left(\mathcal{M}_{ \theta_{t}}(s)-\bar{y}\right)^{2}\;,\quad{\rm where~{}}\bar{y}=\frac{1}{T}\sum _{t=1}^{T}\mathcal{M}_{\theta_{t}}(s)\;,$ (3) $$

Epistemic uncertainty for categorical features:  The decoding of categorical feature $f\in\mathcal{F}_{c}$ is modeled as a classification task. We apply the softmax function to the logits produced by the model, giving a probability distribution over the classes: $p(y|\mathcal{M}_{\theta}(s))=\text{softmax}(\mathcal{M}_{\theta}(s))$. Here, the epistemic uncertainty is measured by the entropy of the predicted probability distributions averaged by multiple forward processes, formulated as

$$ $\alpha_{f}^{cls}:H(p)=-\sum_{c}{p}_{c}\log(p_{c})\;,\quad{\rm where~{}}{p(y=c| s)}=\frac{1}{T}\sum_{t=1}^{T}{\rm softmax}(\mathcal{M}_{\theta_{t}}(s))\;.$ (4) $$

Overall, we approximate the epistemic uncertainty for both numerical and categorical features by MC Dropout, providing a practical and scalable way to capture the model’s uncertainty in its predictions.

#### 3.3.2 Aleatoric uncertainty modeling

We model aleatoric uncertainty by placing a distribution over the model’s output. For example, in this paper, we model the regression output as a Gaussian distribution with random noise, i.e., $p(y|\mathcal{M}_{\theta}(s))=\mathcal{N}(\mathcal{M}_{\theta}(s),\sigma^{2})$, where aleatoric uncertainty aims to learn the variance of noise as a function of the input data.

Aleatoric uncertainty for numerical features:  Let $\sigma_{f}$ denote the noise level of a feature $f\in\mathcal{F}_{n}$, which captures how much noise we have in the output of the feature. Moreover, we assume that $\sigma_{f}$ is data-dependent and can vary with the input. Then, $\sigma_{f}$ can be learned as a function of the data:

$$ $\mathcal{L}_{f}^{reg}=\frac{1}{2\sigma_{f}^{2}}\left\|y_{f}-\widehat{y}_{f} \right\|^{2}+\frac{1}{2}\log\sigma_{f}^{2}\;,$ (5) $$

where $y_{f}$ and $\widehat{y}_{f}$ are the true and predicted value of $f$. $\sigma_{f}$ is also a model’s output that serves as a learned loss attenuation. It gives less penalty to the model when the input data is associated with high aleatoric uncertainty, thus making the loss more robust to noisy data. The second term in Eq.  (5) is a regularization term that prevents the model from learning a high uncertainty score for all instances, trivially driving the first term to zero. In practice, we further adopt a more numerically stable loss given as: $\mathcal{L}_{f}^{reg}=\frac{1}{2}\exp\left(-r_{f}\right)\left\|y_{f}-\hat{y}_{
f}\right\|^{2}+\frac{1}{2}r_{f}$,
where $r_{f}=\log\sigma_{f}^{2}$. Instead of predicting the variance directly, predicting the log variance followed by an exponential function can ensure the positive value of the variance and make the training more numerically stable. In that case, the aleatoric uncertainty for numerical feature $f$ is denoted as $\beta_{f}^{num}:=\sigma_{f}^{2}$.

Aleatoric uncertainty for categorical features:  Let $C_{f}$ denote the number of unique categories (i.e. classes) of feature $f\in\mathcal{F}_{c}$. For the classification task, the model assumes that the prediction logits at sample time $t$ for each category follows a Gaussian distribution:

$$ $\widehat{\bm{m}}_{f,t}=\bm{u}_{f}+\bm{\sigma}_{f}{\bm{\epsilon}_{t}},\quad{\bm {\epsilon}}_{t}\sim\mathcal{N}(0,I)\;,$ (6) $$

where the predicted mean logits $\bm{u}_{f}\in\mathbb{R}^{C_{f}}$ and uncertainty term ${\bm{\sigma}}_{f}\in\mathbb{R}^{C_{f}}$ are the model outputs, as a function of its parameters at the $t$-th sample time. ${\bm{\epsilon}}_{t}\in\mathbb{R}^{C_{f}}$ represents a random vector drawn from a unit normal distribution $\mathcal{N}(0,I)$. Here ${\bm{\sigma}}_{f}$ accounts for the inherent uncertainty in the feature when conducting the prediction. The aleatoric uncertainty of categorical feature $f$ is then the average of variance across all classes, given as $\beta_{f}^{cls}:=\frac{1}{C_{f}}\sum_{c^{\prime}}{\bm{\sigma}}_{c^{\prime}}^{2}$.

To learn such uncertainty, the loss function $\mathcal{L}_{f}^{cls}$ calculates the cross entropy based on the average predicted logits, normalized over all possible categorical outcomes:

$$ $\mathcal{L}_{f}^{cls}=-\log\frac{1}{T}\sum_{t}\exp\left(\widehat{{m}}_{f,t,c}- \log\sum_{c^{\prime}}\exp\widehat{{m}}_{f,t,c^{\prime}}\right).$ (7) $$

The above loss function achieves a similar effect to the numerical case, which can also be considered as learning the loss attenuation (Kendall and Gal, ). When the model assigns a high logit value $m_{c}$ to the observed class $c$, and the noise value $\sigma_{c}$ is low, the loss approaches zero — which is the desired outcome. Overall, the total training loss is the sum of the regression and classification losses across all features, given as:

$$ ${\mathcal{L}}={\sum}_{f\in\mathcal{F}_{n}}{\mathcal{L}}_{f}^{reg}\;+\;\lambda{ \sum}_{f\in\mathcal{F}_{c}}{\mathcal{L}}_{f}^{cls}\;,$ (8) $$

where $\lambda$ is a hyper-parameter designated to balance the scale of regression and classification losses. In our model, we set $\lambda=1$ for simplicity.

### 3.4 Uncertainty-incorporated Anomaly Scoring

We compute event-level anomaly scores by aggregating deviations between the predicted and the observed values of all features, incorporating both epistemic and aleatoric uncertainties into the computation.
For each test event on a given day, we first create a sequence of events spanning a 3-day window, including the previous and next day as context. We mask the event of interest and use the pre-trained UIFormer to predict the masked features. For numerical features, we calculate the deviation as:
${\Delta}_{f}=\left|y_{f}-\widehat{y}_{f}\right|$ with $y_{f}$ and $\widehat{y}_{f}$ depicting the observed and predicted values, respectively. For categorical features, the deviation is computed as: ${\Delta}_{f}=1-\widehat{p}_{f}$ where $\widehat{p}_{f}$ is the predicted probability of the observed class.
UIFormer learns the inherent patterns of human behaviors using only normal data during training. Intuitively, higher deviations from these patterns suggest that an event is less consistent with learned behaviors and thus, more anomalous. Then, we can compute an anomaly score (AS) for a target event $e$ from the deviations alone as:

$$ $\text{AS}(e)=\text{agg}\left(\text{pth}\left({\Delta}_{f}\right)\,|\,f\in \mathcal{F}_{n}\cup\mathcal{F}_{c}\right)\;,$ (9) $$

where $\text{agg}(\cdot)$ denotes the aggregation function (e.g., mean or max) applied over $\Delta_{f}$’s across all features $\mathcal{F}$. The percentile function $\text{pth}(\cdot)$ is used to standardize deviations across features of different scales, effectively capturing the relative ranking of the deviation for each feature. This helps ensure that features with smaller absolute deviations are not undervalued compared to those with larger deviations.
However, using deviations alone to detect anomalies is insufficient due to the diverse nature of human mobility. Some agents may deviate from the general population’s behavior but are not necessarily anomalous. For instance, a (stochastic) agent with a tendency to explore different locations frequently (i.e., with high data uncertainty) may have higher deviations naturally, which does not imply an anomaly. To account for such cases, we propose to incorporate uncertainty estimates into the anomaly score. By scaling down deviations by the uncertainties, we can prevent inflating the anomaly scores of agents whose behaviors, though atypical, are still consistent with natural variability.
As such, we compute an uncertainty-incorporated anomaly score (UI-AS) for a target event $e$ as:

$$ $\text{UI-AS}(e)=\text{agg}\left(\text{pth}\left(\frac{\Delta_{f}}{1+\alpha_{f} +\beta_{f}}\right)\,|\,f\in\mathcal{F}_{n}\cup\mathcal{F}_{c}\right)\;,$ (10) $$

where $\alpha_{f}$ $+$ $\beta_{f}$ denotes the total of the aleatoric and epistemic uncertainties for feature $f$.

## 4 Experiments

We conduct extensive experiments to investigate the following research questions (RQs):
RQ1: What is the performance of UIFormer in the masked prediction task?
RQ2: Can our model outperform previous approaches in the anomaly detection task?
RQ3: Does UIFormer have the ability to accurately capture aleatoric and epistemic uncertainty?
RQ4: How does each uncertainty component contribute to UIFormer?

Datasets. We utilize two human activity datasets, namely SimLA and NUMOSIM. SimLA is an expert-simulated dataset that contains human mobility data in greater Los Angeles area. NUMOSIM (Stanford et al., ) is a publicly-available synthetic human mobility dataset, designed to benchmark anomaly detection techniques for mobility data. Both datasets are obtained by training deep learning models on real mobility data and simulating realistic human mobility patterns, incorporating both normal and anomalous behaviors. The statistics of the two datasets are shown in Appendix Table 5.

We split the data into train/validation/test temporally by date, respectively spanning 3/1/2 weeks. During training, we randomly mask events in each sequence using a specified mask ratio. For validation and test data, we mask only one event per sequence, aligning with how we use the pre-trained model for anomaly detection.

### 4.1 RQ1: Masked Prediction Results

Settings. For baselines, we include classical deep models MLP (Murtagh, ), LSTM (Hochreiter and Schmidhuber, ), the original Transformer (Vaswani et al., ), as well as Dual-Tr; a variant of UIFormer that removes the uncertainty estimation mechanism and only keeps the dual Transformer architecture (w/ cross-feature and cross-event attention). In addition, since UIFormer can estimate the uncertainty of its prediction, we also report its conformal prediction performance where the top-5% most uncertain test samples are not evaluated, denoted by UIFormer (5%).
We report MAE and MAPE to evaluate the prediction performance of numerical features and use accuracy (ACC) for the categorical features. Higher ACC and lower MAE and MAPE indicate better performance. The models are trained on 1 GPU of NVIDIA RTX A6000. We search the hyperparameters in a given model space for all deep models, and each model’s best performance is reported in the experiments. Detailed experiment settings and model configurations can be found in Appendix C.

**Table 1: Masked prediction results. Uncertainty-aware UIFormer outperforms the baselines, while allowing conformal prediction. Abbr: ST; start time (min), SD; stay duration (min), x, y; lat/lon (km).**
| Method | SimLA | NUMOSIM |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MAE | MAPE (%) | ACC(%) | MAE | MAPE (%) | ACC(%) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| x | y | ST | SD | x | y | ST | SD | POI | x | y | ST | SD | x | y | ST | SD | POI |  |
| MLP | 9.73 | 10.16 | 208.07 | 420.92 | 188.13 | 500.59 | 43.28 | 569.01 | 45.61 | 15.97 | 15.74 | 207.64 | 396.58 | 416.55 | 361.58 | 339.15 | 572.69 | 56.08 |
| LSTM | 4.73 | 4.63 | 19.85 | 322.62 | 204.73 | 195.76 | 18.25 | 420.76 | 63.32 | 6.92 | 6.40 | 34.41 | 343.82 | 265.52 | 139.71 | 212.30 | 547.02 | 68.29 |
| Transformer | 2.67 | 2.65 | 26.85 | 77.12 | 132.33 | 115.35 | 17.95 | 66.82 | 73.49 | 4.08 | 3.45 | 21.57 | 71.33 | 143.81 | 72.15 | 15.19 | 68.51 | 75.90 |
| Dual-Tr | 2.71 | 2.63 | 20.62 | 76.53 | 131.72 | 117.29 | 16.41 | 51.95 | 73.74 | 3.63 | 3.48 | 21.69 | 80.08 | 134.32 | 71.34 | 17.36 | 60.17 | 76.12 |
| UIFormer | 2.44 | 2.29 | 11.42 | 35.77 | 109.44 | 102.47 | 15.86 | 20.93 | 72.97 | 3.68 | 3.60 | 26.18 | 45.25 | 121.33 | 84.01 | 236.47 | 28.87 | 74.33 |
| UIFormer (5%) | 2.05 | 1.91 | 8.65 | 21.98 | 99.39 | 89.45 | 1.18 | 20.66 | 75.58 | 3.31 | 3.21 | 8.30 | 33.61 | 109.94 | 74.45 | 1.23 | 28.86 | 76.95 |

Results.
Table 1 shows the
masked prediction results (here and after, we bold the best performance and underline the second best).
UIFormer  delivers the best performance across both datasets and for all features compared with the baselines. In SimLA, UIFormer achieves the lowest MAE for start time (ST) and stay duration (SD) with values of 11.42 and 35.77 minutes, respectively, outperforming the powerful sequence model, Transformer, which yields 26.85 for ST and 77.12 for SD.
Compared to Dual-Tr, building uncertainty into UIFormer leads to significant improvements in prediction performance; in SimLA, UIFormer reduces MAE for ST from 20.62 to 11.42 minutes, and MAPE for SD from 51.95% to 21.93%, underscoring the significant impact of modeling both epistemic and aleatoric uncertainty in achieving more accurate and robust predictions. Lastly, UIFormer (5%) further improves over UIFormer, which demonstrates UIFormer’s ability to identify highly uncertain samples and that it can be quite accurate on its certain predictions.

### 4.2 RQ2: Anomaly Detection Results

Settings. There are only a few related public achievements for human mobility anomaly detection, thus it is difficult to find state-of-the-art models for direct comparison. For a comprehensive evaluation, we implement some baselines as well as extract core components from related literature in trajectory anomaly detection, including Spatial-Temporal Outlier Detector (STOD) (Cruz and Barbosa, ), RioBus (Bessa et al., ), a self-supervised (SS) MLP, LSTM, and Transformer. STOD and RioBus are designed for bus trajectories to predict the Bus ID from the sequence of GPS coordinates which we repurpose for human trajectories. SS-MLP is trained to distinguish real events from fake ones that are synthetically-constructed via random alterations.
Further details on baselines are given in Appendix D.
We report two common metrics for evaluation: AUROC (Area Under ROC curve) and AUPR (Area Under Precision-Recall curve). These are frequently used for anomaly detection as quantities of ranking quality. As the percentage of anomalous events is very small on the original datasets, we construct two additional one with around 5% anomalies by including agents with anomalous events and randomly sampling normal agents without any anomalous events. Dataset statistics for anomaly detection can be found in Appendix Table 6.

**Table 2: Anomaly detection results at the event-level (left) and the agent-level (right).**
| Method | Event-level Anomaly Detection | Agent-level Anomaly Detection |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SimLA (small) | NUMOSIM (small) | NUMOSIM (20K) | SimLA (small) | NUMOSIM (small) | NUMOSIM (20K) |  |  |  |  |  |  |  |
|  | AUROC | AUPR | AUROC | AUPR | AUROC | AUPR | AUROC | AUPR | AUROC | AUPR | AUROC | AUPR |
| STOD | 0.577 | 0.073 | 0.571 | 0.071 | 0.584 | 0.003 | 0.506 | 0.655 | 0.511 | 0.468 | 0.519 | 0.020 |
| RioBus | 0.503 | 0.044 | 0.530 | 0.047 | 0.532 | 0.002 | 0.488 | 0.632 | 0.510 | 0.453 | 0.508 | 0.019 |
| SS-MLP | 0.596 | 0.172 | 0.630 | 0.144 | 0.634 | 0.008 | 0.653 | 0.777 | 0.697 | 0.638 | 0.703 | 0.053 |
| LSTM | 0.755 | 0.182 | 0.637 | 0.122 | 0.642 | 0.020 | 0.631 | 0.706 | 0.599 | 0.526 | 0.658 | 0.062 |
| Transformer | 0.723 | 0.232 | 0.635 | 0.132 | 0.636 | 0.033 | 0.679 | 0.785 | 0.625 | 0.525 | 0.717 | 0.095 |
| UIFormer | 0.795 | 0.329 | 0.694 | 0.146 | 0.670 | 0.034 | 0.746 | 0.841 | 0.655 | 0.572 | 0.727 | 0.114 |

Results.
Table 2 presents the anomaly detection results for both event-level and agent-level anomalies. UIFormer consistently outperforms the baselines across all datasets in detecting anomalous events and agents. At the event level, particularly on SimLA, UIFormer achieves a significantly large margin against the baselines, with the highest AUROC of 0.795 and AUPR of 0.329 (8$\times$ lift over the 0.041 base rate). This highlights the effectiveness of incorporating both aleatoric and epistemic uncertainty, which substantially enhances event-level detection.
UIFormer also performs well for agent-level anomaly detection (w/ maximum anomaly scores across all events per agent). On the NUMOSIM (20K agents) dataset, it achieves an AUROC of 0.727 and an AUPR of 0.114 (57$\times$ lift over the 0.002 base rate), demonstrating its ability to effectively identify rare anomalous agents within a large population.
By incorporating uncertainty into the anomaly score computation, UIFormer prevents inflated scores for agents with exploratory but not necessarily anomalous behavior, thereby reducing false positives. As a result, it more effectively detects true anomalies that deviate from otherwise predictable behavior without incorrectly flagging exploratory behavior.

### 4.3 RQ3: Analysis of Uncertainty Estimation

Next we scrutinize the quality of the estimated uncertainty scores. To this end, we analyze the relationship between prediction accuracy as a function of total uncertainty, as well as how the estimated aleatoric and epistemic uncertainty scores vary by the amount of training data using SimLA.

Relationship between uncertainty and MAE/accuracy. We first sort the test samples by the total AU+EU uncertainty score (i.e., $\sum_{f\in\mathcal{F}}\alpha_{f}+\beta_{f}$), and then gradually remove test samples with uncertainty larger than a certain percentile threshold while recording the prediction performance of UIFormer on the rest. As shown in Figure 4, UIFormer’s performance gradually improves as we exclude test samples with large total uncertainty. This means that the uncertainty estimates are well aligned with prediction performance, as all MAE (Accuracy) curves follow a monotonically increasing (decreasing) trend for numerical (categorical) features. Notably, for stay duration (SD) prediction (middle), MAE increases sharply from around 20 minutes to 35 minutes due to the top 5% most uncertain samples. As for the POI prediction (right), around 50% of the most certain samples can yield a quite promising performance that achieves near 100% accuracy, which gradually drops with increasing uncertainty. Interestingly, AU-only UIFormer (in blue) appears to be more reflective of these expected trends for x and SD, while it is the EU-only variant (in purple) for POI, demonstrating the complementary benefit of estimating these different uncertainty types.

Figure: Figure 4: Relationship btwn. MAE (accuracy) & uncertainty follow an increasing (decreasing) trend.
Refer to caption: x4.png

Relationship between uncertainty and amount of training data. Next we investigate how aleatoric and epistemic uncertainty estimates change with varying amount of training data, under three different configurations with 1/2/3 weeks of training data and 2 weeks of test data on SimLA. Table 3 reports the average aleatoric uncertainty (AU) and epistemic uncertainty (EU) across test samples for each feature. We observe that ($i$) AU remains almost constant as training data size increases, which empirically shows that AU accounts for the inherent randomness/stochasticity in the data that is not a matter of the amount of training data, and that ($ii$) EU decreases noticably with increasing data size, suggesting that it can be reduced with sufficient training data.

**Table 3: Aleatoric (AU) and epistemic uncertainties (EU) under different training data size settings.**
| Train | Test | x | y | Start Time | Stay Duration | POI |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | AU $\times 10^{-3}$ | EU $\times 10^{-5}$ | AU $\times 10^{-3}$ | EU $\times 10^{-5}$ | AU $\times 10^{-3}$ | EU $\times 10^{-5}$ | AU $\times 10^{-3}$ | EU $\times 10^{-5}$ | AU $\times 10^{-3}$ | EU |
| 1-week | 2-week | 1.30 | 2.37 | 2.40 | 4.00 | 0.85 | 15.49 | 0.10 | 0.48 | 17.11 | 1.01 |
| 2-week | 2-week | 1.44 | 2.05 | 2.45 | 3.12 | 0.61 | 7.29 | 0.13 | 0.25 | 14.34 | 0.98 |
| 3-week | 2-week | 1.36 | 1.23 | 2.51 | 2.00 | 0.59 | 4.08 | 0.11 | 0.14 | 8.25 | 0.94 |

Figure: Figure 5: The relationship between epistemic uncertainty (EU) and accuracy of each POI class. Size of each point/POI is proportional to its frequency in the dataset. We only annotate a few POI labels for better illustration.
Refer to caption: x5.png

Relationship between EU and accuracy of each POI class. From the results in Figure 5, we have the following observations: ($i$) As before, there is an inverse trend where accuracy decreases as epistemic uncertainty (EU) increases. This suggests that as the model becomes less certain (higher EU), its performance in predicting the correct POI class deteriorates. ($ii$) A higher frequency of POI classes (such as home and office buildings) generally leads to lower EU.
Since EU aligns with amount of training data as showcased earlier,
the more frequent POI classes, for which the model is trained on more observations, associate with lower EU and higher prediction accuracy. ($iii$) A lower frequency does not always mean high EU however. For example, POI:education in Figure 5 does not have a frequency as high as POI:home and POI:store with similar accuracies, but it has a relatively low estimated EU possibly because people’s visits to such POI are highly patterned.

### 4.4 RQ4: Ablation Study

As uncertainty is a key component of our anomaly detection model, we conduct ablation studies on its role from both the training and inference perspectives.

Uncertainty from the training perspective.
To study the role of aleatoric versus epistemic uncertainty estimation, we consider three variants of UIFormer: ($i$) UIFormer-AU, which only implements the aleatoric uncertainty while keeping the model backbone the same as UIFormer, ($ii$) UIFormer-EU, which only models epistemic uncertainty; and ($iii$) UIFormer-none, which only has the bare Dual Transformer and does not model epistemic or aleatoric uncertainty.

For comparison, we plot the relationship between uncertainty and performance in Figure 4. Firstly, the inclusion of uncertainty modeling can significantly enhance the predictive accuracy of numerical features. Moreover, UIFormer-AU model significantly outperforms UIFormer-EU in regression tasks. This suggests that aleatoric uncertainty captures essential variance within the dataset that epistemic uncertainty cannot address effectively. However, when considering classification tasks, such as POI and DOW, modeling aleatorical uncertainty does not provide a significant performance boost. UIFormer slightly outperforms UIFormer-AU, which suggests that modeling both aleatoric uncertainty and epistemic uncertainty can further improve the performance.

Uncertainty from the inference/anomaly scoring perspective.
In this section, we investigate the impact of different anomaly scoring functions on detection performance.

Table 4 presents the results of UIFormer’s event-level anomaly detection using three different scores per feature: $\Delta_{f}$, $\text{pth}(\Delta_{f})$, and $\text{pth}\left(\frac{\Delta_{f}}{1+\alpha_{f}+\beta_{f}}\right)$. Here, $\Delta_{f}$ represents the feature delta error, $\text{pth}(\cdot)$ denotes the percentile function, and $\alpha_{f}$ and $\beta_{f}$ represent aleatoric and epistemic uncertainties, respectively. The feature anomaly scores were aggregated by taking the maximum across features.

**Table 4: Comparison of different anomaly scores**
| Function | SimLA (small) | NUMOSIM (small) | NUMOSIM (20K) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| AUROC | AUPR | AUROC | AUPR | AUROC | AUPR |  |
| $\Delta_{f}$ | 0.666 | 0.200 | 0.553 | 0.062 | 0.556 | 0.003 |
| $\text{pth}(\Delta_{f})$ | 0.789 | 0.295 | 0.694 | 0.143 | 0.699 | 0.031 |
| $\text{pth}\left(\frac{\Delta_{f}}{1+\alpha_{f}+\beta_{f}}\right)$ | 0.795 | 0.329 | 0.694 | 0.147 | 0.700 | 0.034 |

From the comparison between $\Delta_{f}$ and $\text{pth}(\Delta_{f})$, we observe a significant improvement in performance when using percentiles. This shows that the percentile function and thus using the relative ranking of errors is beneficial when features are variable in scale, ensuring that features with smaller error scales are not overshadowed by those with larger scales, leading to more balanced and effective anomaly detection.
Furthermore, comparing $\text{pth}(\Delta_{f})$ to $\text{pth}\left(\frac{\Delta_{f}}{1+\alpha_{f}+\beta_{f}}\right)$, we see that incorporating uncertainties into the anomaly scoring further enhances detection performance. This shows that accounting for total uncertainty improves the model’s ability to identify the true anomalies more effectively.

## 5 Related Work

Human Mobility Modeling. Methods in this field can be broadly categorized into traditional statistical models and the more recent deep learning approaches. Traditional statistical approaches typically rely on specific functional forms such as Poisson or Hawkes processes to predict event arrival times (Hawkes, ; Daley and Vere-Jones, ; Ogata, ), which struggle to capture the intricate spatiotemporal patterns present in human mobility data. Other statistical methods like Markov Chains, use transition matrices to predict future locations (Chen et al., ; Gambs et al., ; Cheng et al., ; Monreale et al., ) yet they fall short in modeling long-term dependencies. With the rise of deep learning, models such as RNNs and LSTMs have been used to learn the complex transition patterns (Gao et al., ; Song et al., ; Du et al., ), leading to better performance in the next location and trajectory prediction. Most recently, there has been a growing trend of using Transformer-based models for human mobility and trajectory data (Wan et al., ; Feng et al., ; Xue et al., ; Abideen et al., ; Wu et al., ; Wang and Osaragi, ), mainly inspired by its immense success in modeling inherently sequential data in NLP. However, most Transformer-based models overlook the underlying uncertainty in the data, especially as it pertains to human behavior over time. Specifically, most prior work focus on pretraining solely based on the masked reconstruction loss, without uncertainty estimation of the prediction. This motivates us to develop a new uncertainty-aware model to fill the gap, toward more effectively capturing human behavior that is often dependent on time and context.

Uncertainty Estimation in Deep Neural Networks.
Uncertainty estimation has been studied in the field of NLP (Gal and Ghahramani, ; Xiao and Wang, ) and CV (Kendall et al., ; Huang et al., ), which shows great promise in reflecting the underlying uncertainty in outcomes and thus improving task performance. One common approach is the Monte Carlo (MC) Dropout (Gal and Ghahramani, ), which estimates uncertainty by applying dropout during inference, thereby improving model robustness.
Others employ deep ensembles for model uncertainty estimation Lakshminarayanan et al. ().
Kendall and Gal () propose to learn aleatoric uncertainty through loss attenuation while modeling epistemic uncertainty using MC Dropout in both regression and classification tasks for CV. Mobiny et al. () drop individual weights of the network during training and inference to measure uncertainty robustly.
Motivated by the success of uncertainty estimation in these domains, we explore the potential benefit of uncertainty estimation in modeling human mobility, which forms the basis of our proposed model.

## 6 Conclusion

We introduced UIFormer for human mobility modeling and anomalous behavior detection.
UIFormer is equipped with both data and model uncertainty estimation to account for the stochasticity inherent to human nature as well as the data scarcity in sufficiently observing complex human behavior.
While its dual-attention mechanism captures dependencies among features and events, uncertainty estimation
lends itself to more robust training and more nuanced anomaly scoring.
Experiments on two benchmark datasets showed that UIFormer outperforms existing baselines, while ablation analyses underscored the benefit of uncertainty-aware forecasting and anomaly detection.

## Appendix A Data Statistics

**Table 5: Dataset statistics.**
|  | SimLA | NUMOSIM |
| --- | --- | --- |
| #Agents | 20,000 | 20,000 |
| #Anomalous Agents | 162 | 381 |
| #Events | 3,111,712 | 3,428,714 |
| #Anomalous Events | 892 | 3,468 |
| x range (km) | [-57, 57] | [-59, 59] |
| y range (km) | [-40, 40] | [-60, 60] |
| Avg of stay duration (min) | 509 | 455 |
| Avg of start time (min) | 850 | 776 |

## Appendix B Details of Transformer Block

Transformer Block, which contains two key components: a Multi-Head Self-Attention (MSA) layer and a Feed-Forward Network (FFN) layer. The MHA layer facilitates message passing between input tokens, while the FFN applies non-linear transformations to enhance feature extraction across different dimensions of the input vectors. To capture more complex interactions between tokens, multi-head attention is employed, where the attention mechanism is defined in Equation (11).

$$ ${\rm{Attention}}(\bm{Q},\bm{K},\bm{V})={\mathop{\rm softmax}\nolimits}\left({ \frac{{\bm{Q}{\bm{K}^{T}}}}{{\sqrt{d}}}}\right)\bm{V},$ (11) $$

where $\bm{Q}\in\mathbb{R}^{N\times D}$, $\bm{K}\in\mathbb{R}^{N\times D}$, and $\bm{V}\in\mathbb{R}^{N\times D}$ represent the query, key, and value matrices, respectively, all projected from the same input matrix $\bm{E}$ (which have different forms in Feature-level Transformer and Event-level Transformer) with different learnable weight matrix. The softmax function transforms the scaled dot product into attention weights for $\bm{V}$, and $d$ is the dimensionality of $\bm{K}$ used for scaling the inner product. Besides, more high-order mutual information can be captured by stacking multiple Transformer blocks.
Denote the embedding outputted by block $m\in\{1,\cdots,M\}$ as $e_{i}^{m}$, its updating process can be formulated as follows

$$ $\begin{array}[]{l}{{{{\hat{e}}}}_{i}}={{{e}}_{i}^{m-1}+{\rm{MSA}}_{i}^{m}\left ({{\rm{head}}_{1}^{m-1},\ldots,{\rm{head}}_{n_{\rm{head}}}^{m-1}}\right)}\\ {{e}}_{i}^{m}={{{{\hat{e}}}}_{i}}+{\rm{FFN(}}{{{{\hat{e}}}}_{i}}{\rm{)}}.\end{array}$ (12) $$

where $n_{\rm{head}}$ is the number of heads.

## Appendix C Detailed Settings for Masked Prediction

Baselines. The following methods are chosen as baselines: (1) MLP (Murtagh, ), which applies multiple Linear layers to encode the event sequence, and decodes the masked features according to the embedding from the last layer. (2) LSTM (Hochreiter and Schmidhuber, ), a recurrent neural network commonly used to model sequential data, it reads the event sequence and decodes the masked event based on its hidden vector. (3) Transformer(Vaswani et al., ), one of the most powerful sequence models nowadays, which models the correlation of different events by multi-head self-attention. (4) Dual-Transformer, a variant of our proposed model, which removes the uncertainty learning mechanism and only keeps the dual Transformer architecture.

Metrics. We consider the prediction of numerical features and categorical features as regression and classification tasks, respectively. To this end, we apply MAE and MAPE to evaluate the performance of regression, and Accuracy (ACC) is applied to evaluate the classification performance. The unit for x, y are kilometers while the unit for start time and stay duration are minutes.

Settings. The models are trained on 1 GPU of NVIDIA RTX A6000. For all deep models, we search the parameters in a given parameter space, and each model’s best performance is reported in the experiment. Here we report the parameter search space of our model: the mask ratio for pre-training is searched from [0.05, 0.3]. The embedding size for the transformer is searched from [32, 64, 128], and the layers for the transformer are searched from [2, 3, 5]. The batch size of each epoch is searched from 128 and the learning rate of Adam optimizer starts from 1e-3 with a weight decay 1e-05.

## Appendix D Detailed Settings for Anomaly Detection

Baselines. We selected the following baseline methods for comparison: RioBus Bessa et al. (), Spatial-Temporal Outlier Detector (STOD) Cruz and Barbosa (), a self-supervised MLP (SS-MLP), LSTM, and Transformer. RioBus Bessa et al. () is initially applied to bus trajectories and employs a Convolutional Neural Network (CNN) to predict the ID of the bus based on the sequence of GPS coordinates. For agent-level anomaly scores, it uses the negative probability of the correct prediction of the agent (bus) ID. For the stay-point (or event) level, scores are derived from the anomaly score of the 20-coordinate sub-trajectory ending at the given stay point. STOD Cruz and Barbosa () is also originally applied to bus trajectories and is GRU-based to predict the bus ID. Its anomaly score computation approaches are similar to RioBus. SS-MLP takes event features including GPS coordinates and time as input. These features are first converted into embeddings, which are then concatenated to form a unified input vector. This vector is processed by an 8-layer MLP that outputs a probability indicating whether an event is real (1) or fake (0). During training, fake events are generated by randomly altering real events, and the model learns to classify between real and fake using binary cross-entropy loss. This approach allows the model to distinguish between normal and anomalous data. Finally, the negative of the output probability is used as the anomaly score, with a higher score indicating greater anomaly likelihood.

Settings. Since the percentage of anomalous events in the entire dataset is extremely small, we construct two additional test datasets with approximately 0.05% anomalous events, named SimLA (small) and NUMOSIM (small). These datasets are constructed by including all agents with anomalous events and randomly sampling from other agents without any anomalous events. The statistics of the test datasets are in Table 6. This approach provides three test datasets with varying levels of anomaly scarcity, allowing for diverse evaluation settings.

**Table 6: Dataset statistics for anomaly detection. Ratio of anomalies depict the base rate.**
|  | SimLA (small) | NUMOSIM (small) | NUMOSIM (20K) |
| --- | --- | --- | --- |
| #Agents | 254 | 851 | 20,000 |
| #Anomalous Agents | 162 | 381 | 381 |
| Ratio of Anomalous Agents | 0.638 | 0.448 | 0.019 |
| #Events | 18,963 | 73,129 | 1,706,468 |
| #Anomalous Events | 892 | 3,468 | 3,468 |
| Ratio of Anomalous Events | 0.041 | 0.047 | 0.002 |

## Appendix E Case study

Figure: Figure 6: Case Studies of High-uncertainty and Low-uncertainty Agents
Refer to caption: x6.png

Here, we present two examples: one agent with high total uncertainty and another with low uncertainty. Their daily trajectories in the first week of training data are shown in Figure 6. Markers of different shapes and colors represent various POI types. The numerical labels next to the markers indicate the order of events occurring each day. The high-uncertainty agent visits a wide variety of POI types daily, without any fixed pattern in the locations visited or the order of visiting those locations. In contrast, the low-uncertainty agent exhibits regular patterns on weekdays, such as traveling from home to a possible workplace, and remains at home all day on Sunday during the weekend. These examples demonstrate that our model’s uncertainty measure effectively captures the variability in human mobility patterns.

## Appendix F Broader Impact

Our work proposes an unsupervised approach to modeling uncertainty in sequence data, with a specific focus on human mobility. By leveraging both aleatoric and epistemic uncertainties, our model captures the variability inherent in complex behaviors and provides a robust way to detect anomalies without relying on labeled data. This approach is highly generalizable and can be applied across various domains, such as finance (e.g., detecting credit card fraud or money laundering), online shopping (e.g., analyzing user purchasing patterns and identifying bot activity), and app user behavior analysis (e.g., detecting shifts or unusual engagement). By modeling uncertainty, the proposed method can provide more informative anomaly scores, distinguishing between natural variability and significant deviations that require attention. This makes the model particularly useful in situations where uncertainty is expected, and traditional methods might lead to false positives.

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors elaborate on why only one event is masked per sequence during validation and testing? Would masking more events per sequence affect the performance or the uncertainty estimates?
2. Is the SimLA dataset open-sourced, and are there plans to release it publicly? This would be useful for reproducibility and benchmarking future models.
3. Why was Monte Carlo Dropout chosen for epistemic uncertainty estimation, and are there plans to experiment with other methods like deep ensembles or evidential neural networks?
4. Are there specific applications or downstream tasks, such as traffic signal adjustments or navigation, where the proposed model could be directly applied to demonstrate practical benefits?
soundness: 3
strengths: 1. Originality: The model’s design, utilizing a dual-Transformer architecture for feature- and event-level attention, effectively captures dependencies at different levels of human behavior.
2. Quality: The paper includes comparisons with various baselines, showing that UIFORMER achieves superior performance for anomaly detection.
3. Clarity: The paper is well-organized and clearly explains the model’s architecture, training dynamics, and uncertainty estimation.  Figures and tables enhance the understanding of the model’s performance and design.
4. Significance: The method’s ability to better capture uncertainties in human behavior could have significant implications for applications like security, surveillance, and health monitoring, where accurate anomaly detection is critical.
summary: The paper proposes UIFORMER, a dual-Transformer architecture for modeling human mobility patterns and detecting anomalies. It combines aleatoric and epistemic uncertainty estimation to better capture uncertainties inherent in human mobility data. The model aims to enhance robustness in anomaly detection by incorporating uncertainty into decision-making.
weaknesses: 1. Lack of Novelty in Uncertainty Quantification: The techniques for quantifying aleatoric and epistemic uncertainty, including Monte Carlo (MC) Dropout, have been widely used in other domains. The absence of a discussion on related works, specifically evidential deep learning methods, limits the novelty of the approach. Incorporating such references and discussions could strengthen the contribution of this paper.
2. There is insufficient detail about the SimLA dataset, particularly whether it is open-source or publicly available. Clarifying this point would improve transparency and reproducibility.
3. The decision to mask only one event per sequence in validation and testing phases is restrictive and may not reflect the full potential of the model in real-world scenarios where multiple events may be missing.
4. The use of MC Dropout for epistemic uncertainty estimation limits the exploration of alternative methods. Testing more methods, such as deep ensembles or evidential neural networks, could provide a more comprehensive understanding of uncertainty estimation in this context.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Kindly see my question in the weaknesses section, especially the questions that I raised in bullets 2, 4 and 5 above.
soundness: 2
strengths: 1) The problem of modeling human mobility data is an important one, with **important real-world implications**. 
2) The overall approach proposed for this particular problem is **a unique and new combination of existing ideas from literature**. It is of significance for the urban mobility domain. Especially the focus on modeling uncertainty when modeling human mobility data is something that can be very relevant for practical applications but is typically ignored in the urban mobility literature.  
3) The paper is **very well written**. The approach is mostly presented in a **clear** way (with some expectations, as mentioned below) and, in most cases, with a sufficient level of detail. 
4) The **empirical results show significant improvements** over the simpler baselines considered.
summary: This paper proposes an approach for anomaly detection of human mobility behaviour based on GPS data. The proposed approach is based on a Transformer architecture to model sequences of events/activities and it explicitly attempts to model epistemic and aleatoric uncertainty, which are then leveraged to devise a scoring function for rating abnormal behaviours. Using two simulated human activity datasets, the authors show that the proposed approach outperforms simpler baselines in modelling human mobility sequences and in detecting anomalous behaviors, especially showcasing and highlighting the contribution of modelling epistemic and aleatoric uncertainty to the results.
weaknesses: 1) Although the specific combination of existing ideas in the proposed approach is new from the application point of view, from a methodological point of view, the **low novelty** is a concern, especially **for the broader ICLR audience**. Since the contribution is of significance mostly for a transportation audience, I would encourage the authors to consider targeting a transportation journal instead since I believe this work would have more visibility and more impact there. Otherwise, for ICLR, the authors should motivate better the broader applicability of their work and their insights beyond the concrete application considered. For example, could the uncertainty-aware approach be applied to other sequence modeling tasks in different domains? Could you give some examples? 
2) The **motivation for the application considered should be improved**. Why would one want to detect abnormal mobility behaviours? The authors briefly mention that they can "indicate security threats or spread of infectious disease", but these use cases need to be better explained and backed up by related works where having such predictions of abnormal mobility behaviours would be relevant. This is the critical basis for the proposed work, so it needs to be articulated in a more convincing way. 
3) The **baselines considered in the paper are very naive**. However, modelling sequences of human activities is a very well-studied problem in the transportation field, and the literature is vast on different deep learning approaches, including approaches based on transformers. I encourage the authors to consider including stronger baselines from the state-of-the-art for modelling human activity sequences - i.e., at least for the predictive part (RQ1); although the extension of some of these for score-based anomaly detection is also trivial, and therefore can also be potentially considered as additional baselines for RQ2. 
4) The **soundness of the experimental setup is a concern**, given that the datasets used for validating the proposed approach are themselves simulated based on deep learning models that were trained on real-world data to generate synthetic sequences. The authors are essentially fitting a deep learning model to the output of another deep learning model. This raises at least three important questions: i) is the output of the deep generative model good enough to generate realistic synthetic sequences that capture well the data distribution, including the "long-tails" such that abnormal behaviors are captured? and ii) why not simply use the deep generative model that was used to generate the data to detect abnormal behaviors? What are the advantages of using the proposed transformer architecture instead? iii) why haven't the authors considered using real data to demonstrate their approach (of course, despite a quantitative evaluation being difficult/impossible)? I would suggest the authors discuss these concerns and discuss their rationale for using simulated data and the potential limitations or advantages of this approach in the paper, as well as discuss the challenges and potential approaches for evaluating real-world data.  
5) There are several aspects of the proposed approach that are either **not properly motivated or unclear**. For example, eq. 7 is presented as "the average predicted logits"; however, looking at eq. 7, it seems to be the average of the exponentiated logits (i.e., likelihoods), and then the authors take the logarithm. This should be clarified and properly motivated. Also, the authors have multiple passes T for the categorical loss in eq. 7, but not for the numerical features loss in Section 3.3.2. What is the reasoning behind this distinction in approaches? Similarly, what is the reasoning for the +1 in eq. 10? I would suggest the authors provide a more detailed explanation or derivation for equation 7, and explicitly state their reasoning for the different approaches used for categorical and numerical features, as well as the addition of 1 in equation 10. 
6) Lastly, there are **several inconsistencies** in the paper. For example, it is written that "MAPE for SD from 51.95% to 21.93%", however these numbers don't match the values in Table 1. Similarly, it is mentioned that "UIFORMER delivers the best performance across both datasets and for all features compared with the baselines.". However, the MAPE of UIFORMER for ST is significantly worse than a simple transformer. These aspects should be clarified in the paper. It is also written that "AU remains almost constant as training data size increases"; however, looking at the table there are several significant exceptions that should be considered and discussed. I suggest that the authors carefully review their results and claims for accuracy, and provide a more nuanced discussion of their findings, particularly where the results don't align perfectly with their general claims.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1.	What are the detailed experimental settings of the baseline methods used for comparison?
2.	Why not use real-world datasets for evaluation? 
3.	Can the proposed also use other anomaly scores apart from the one proposed in the paper, e.g., reconstruction error?
soundness: 1
strengths: 1.	The motivation of the paper is clearly presented, and the writing is easy to follow. 
2.	The proposed method estimates both epistemic uncertainty and aleatoric uncertainty. 
3.	Both feature-level and event-level information are taken into account. 
4.	An uncertainty-based score is proposed for anomaly detection.
summary: This paper presents a new method for human mobility modelling and anomaly detection via uncertainty estimation. The proposed method consists of a dual transformer encoder and an uncertainty-aware decoder.
weaknesses: 1.	My main concern with the paper is the lack of novelty. The proposed method is merely the combination of existing techniques. The authors should explore more advanced techniques in the field. For example, the epistemic uncertainty is estimated by MC dropout which is a weight-space method. It can also be studied from a function-space perspective, using methods such as Gaussian processes, Hamiltonian Monte Carlo, Stein variational gradient descent, etc. As for aleatoric uncertainty, methods such as quantile regression and conformal inference are worth exploring as well.

2.	Another major issue of the paper is that the evaluation is not convincing. the baselines used for comparison in the experiments do not reflect the current state-of-the-art methods for human mobility modelling or uncertainty estimation. 

Please refer to the relevant works listed below.
1) Abdar, Moloud, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth et al. "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Information fusion 76 (2021): 243-297.
2)Sun, Shengyang, Guodong Zhang, Jiaxin Shi, and Roger Grosse. "FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS." In International Conference on Learning Representations, 2019.
3) Maddox, Wesley J., Pavel Izmailov, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. "A simple baseline for bayesian uncertainty in deep learning." Advances in neural information processing systems 32 (2019).
4) Angelopoulos, Anastasios N., and Stephen Bates. "Conformal prediction: A gentle introduction." Foundations and Trends® in Machine Learning 16, no. 4 (2023): 494-591.

3.	The two datasets used for evaluation are synthetic. The proposed method should also be tested on real-world datasets.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How $\lambda$ is selected in the loss function?

2. There are various time series and trajectory anomaly detection methods. It would be interesting to investigate if these methods can be used to address human mobility anomaly detection.

3. It would be better to provide a running example to show how the uncertainties can be captured by the proposed uncertainty-aware decoder.

In addition, please see the weaknesses for other questions.
soundness: 3
strengths: **S1.** The paper introduces a novel and practical approach to model both aleatoric and epistemic uncertainties in trajectories, allowing the model to handle the inherent randomness of human behavior and data sparsity effectively.

**S2.** Overall, the paper is technically sound and provides valuable contributions on how to deal with aleatoric and epistemic uncertainties with numerical features and categorical features together. 

**S3.** The comprehensive experiments show the effectiveness of uncertainty modeling and ad performance.
summary: The paper proposes a model for detecting anomalies in human mobility using GPS data without labeled training data. It represents GPS data as sequences of stay-point events and utilizes Transformers for un/self-supervised learning. The model incorporates aleatoric and epistemic uncertainties to handle behavioral randomness and data sparsity, enabling robust training and decision-making. Experiments demonstrate its effectiveness over traditional forecasting and anomaly detection methods.
weaknesses: **W1.** It would be better to provide more case studies to intuitively show the learned AU and EU.

**W2.** It would be better to include an addition experiment to show how $\lambda$ is selected in the loss function.

**W3.** It would be more interesting to explain why the authors chose their current set of baselines. More baselines are required. For example, it would be interesting to compare the proposed method with the existing time series outlier detection methods [1]. It would help the community to see whether existing methods like the Anomaly Transformer will work for human mobility anomaly detection.

_[1]. Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy, ICLR 2022._

**W4.** It would be better to include a section to introduce anomaly detection in sequential data (e.g., trajectory and time series) with more additional references as follows,

_[2]. Mobile Trajectory Anomaly Detection: Taxonomy, Methodology, Challenges, and Directions, IoTJ 2024._

_[3]. Time-series anomaly detection service at Microsoft, SIGKDD 2019._

_[4]. Anomaly detection in time series: a comprehensive evaluation, PVLDB 2022._

_[5]. Unraveling the 'Anomaly'in time series anomaly detection: a self-supervised tri-domain solution, ICDE 2024._

_[6]. Revisiting VAE for Unsupervised Time Series Anomaly Detection: A Frequency Perspective, WWW 2024._

_[7]. PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection, SIGKDD 2024._

_[8]. Deep variational graph convolutional recurrent network for multivariate time series anomaly detection, ICML 2022._

In addition, it is suggested to discuss the key differences between general time series/trajectory anomaly detection and the studied human mobility anomaly detection.

**W5.** As efficiency is important in anomaly detection, especially for downstream real-world applications, it is encouraged to compare the theoretical space and time complexities of the proposed method and baselines. Additionally, it would be better to include an experiment to show the training time and inference time comparison.

Meta-review: While the submission presents an interesting and original application of uncertainty quantification for anomaly detection in human mobility data, several key issues must be addressed for it to be considered a strong contribution to the ICLR community:

The novelty of the approach is limited, as it relies on existing techniques for uncertainty estimation. Exploring alternative methods and providing a more comprehensive discussion of the state-of-the-art in uncertainty quantification would be beneficial. The use of synthetic datasets undermines the robustness of the experimental evaluation. Real-world datasets should be incorporated to strengthen the model's real-world applicability. The choice of baselines and the motivation behind the application of anomaly detection in mobility data need to be reconsidered and expanded upon.

Given the concerns about novelty, dataset choice, and experimental rigor, I recommend rejecting the paper in its current form.

Decision: Reject