Modeling Heart Rate and Activity Data for Personalized Fitness
Recommendation
Jianmo Ni

Larry Muhlstein

Julian McAuley

University of California, San Diego
jin018@ucsd.edu

University of California, San Diegolarrymuhlstein@gmail.com

University of California, San Diego
jmcauley@ucsd.edu

ABSTRACT
Activity logs collected from wearable devices (e.g. Apple Watch,
Fitbit, etc.) are a promising source of data to facilitate a wide range
of applications such as personalized exercise scheduling, workout recommendation, and heart rate anomaly detection. However,such data are heterogeneous, noisy, diverse in scale and resolution,and have complex interdependencies, making them challenging tomodel. In this paper, we develop context-aware sequential models
to capture the personalized and temporal patterns of fitness data.
Specifically, we propose FitRec – an LSTM-based model that captures two levels of context information: context within a specificactivity, and context across a user’s activity history. We are specifically interested in (a) estimating a user’s heart rate profile for a
candidate activity; and (b) predicting and recommending suitable
activities on this basis. We evaluate our model on a novel dataset
containing over 250 thousand workout records coupled with hundreds of millions of parallel sensor measurements (e.g. heart rate,GPS) and metadata. We demonstrate that the model is able to learncontextual, personalized, and activity-specific dynamics of users’heart rate profiles during exercise. We evaluate the proposed model
against baselines on several personalized recommendation tasks,
showing the promise of using wearable data for activity modelingand recommendation.

KEYWORDS
Fitness Recommendation; Sequential Modeling; Personalization
ACM Reference Format:
Jianmo Ni, Larry Muhlstein, and Julian McAuley. 2019. Modeling Heart Rateand Activity Data for Personalized Fitness Recommendation. In Proceedingsof the 2019 World Wide Web Conference (WWW’19), May 13–17, 2019, San
Francisco, CA, USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.
1145/3308558.3313643

1

INTRODUCTION

The development of wearable technologies has given people theopportunity to measure and track their activities via mobile devices (e.g. smart watches). Such devices collect various types ofdata, including health-related measurements (e.g. heart rate) andcontextual measurements (e.g. location, altitude, activity type). An
example of such data (from endomondo.com) is shown in fig. 1. Harnessing this data to model fitness is beneficial for developing safer
This paper is published under the Creative Commons Attribution 4.0 International(CC-BY 4.0) license. Authors reserve their rights to disseminate the work on theirpersonal and corporate Web sites with the appropriate attribution.
WWW ’19, May 13–17, 2019, San Francisco, CA, USA© 2019 IW3C2 (International World Wide Web Conference Committee), publishedunder Creative Commons CC-BY 4.0 License.
ACM ISBN 978-1-4503-6674-8/19/05.
https://doi.org/10.1145/3308558.3313643

Figure 1: A representative example of our data collectedfrom endomondo.com. The figure contains measurements
during one workout, including route, speed, altitude andheart rate sequences as well as overall statistics.
and more reliable activity-related services [1]. However, this remains a challenging task for a number of reasons: (1) the collectedsequential data are often heterogeneous and the interdependencies
among different variables are hard to capture; (2) users’ activity
patterns and health conditions change over time; and (3) the data
has high variance across users, and while large datasets can beeasily collected, the amount of data associated with each user islimited.
Researchers have recently sought to incorporate deep learningtechniques to capture the dynamics of sequential data. One of ourgoals is to adapt such models to capture personalized dynamics in
GPS and heart rate data. Recurrent Neural Network (RNN) basedmodels [6, 13] have been proposed to model multivariate sequential data and shown promising performance [5, 20]. Comparedwith traditional models, RNNs are able to learn complex non-linear
relationships between input variables. Furthermore, RNNs are convenient for modeling sequential problems with different input andoutput structures (e.g. sequential inputs and sequential outputs, orsequential inputs and numerical outputs). This allows us to straightforwardly design model variants for different prediction tasks on
our data. More recently, attention mechanisms [3] have further
boosted the performance of RNN based models due to their ability
to make the model aware of all previous hidden states.
In order to bring state-of-the-art models of sequential data tobear on large datasets of fitness activities, it is important to ask
how we can personalize models of sequence data to individual users.
Personalization is crucial when modeling people’s fitness due to thewide variation among individuals. For example, users may differ

Target:Finish in 50 minutes

User ID ... Sport Type
Historical workout
sequences

Route

T

Short term
prediction

Workout profile
forecastingT

What will the heart rate
profile look like?

Real time
workout
sequences

and activity data have attracted interest in the data mining andrecommendation fields because they contain rich contextual information (e.g. spatial and temporal information) about users thatmight complement existing recommender systems [2]. However,there have been few works studying how these data can be used inreal-world recommendation applications.
The key contributions in this paper are threefold:(1) We propose an LSTM-based model FitRec that considers context both within and across workouts. FitRec infers static
user embeddings from user attributes and meanwhile learns
temporal embeddings from users’ recent workout sequences.
FitRec can be applied to various tasks by using either a twolayer stacked LSTM module or an attention-based encoderdecoder module. We show the model’s performance compared with prior sequential modeling baselines such as Multilayer Perceptrons (MLP) [15] and Dual-stage Attentionbased RNNs (DA-RNN) [25].
(2) We demonstrate that FitRec can be used on two real-world
recommendation tasks: workout route recommendation andshort term heart rate prediction. We evaluate performancein terms of standard recommendation metrics (e.g. AUC,
F1-score) and show that our model outperforms strong baselines.
(3) We contribute a workout dataset that contains over 250 thousand workout records with multiple types of heterogeneoussequential signals as well as metadata. To the best of ourknowledge, we are the first to release a large-scale dataset of
this type. Beyond the recommendation problems considered
here, such a dataset has significant potential for re-use inresearch on heart rate data, sequence modeling, personalization, etc.

T+1?

How will the heart ratechange in next few seconds?

Figure 2: FitRec can be applied to two different tasks: workout profile forecasting and short term prediction.
significantly in comfort heart rate zones, adaptation ability for
different exercises, etc. Accounting for such differences requires
robust modeling techniques to capture the dynamics of both explicit
and implicit user-dependent features. An effective model needs toefficiently learn both static and temporal features from historicaldata.
In this paper, we address these challenges by exploring a largescale dataset of workout records and building a predictive model
leveraging context within and across workouts. Specifically, we
propose a Long Short-Term Memory (LSTM) based model FitRecthat takes as input user attributes and multiple workout measurement sequences. FitRec infers static user embeddings based on theattributes and temporal user embeddings from historical workout
measurements. By combining these inferred latent representationsof our users, FitRec can then be applied to (1) Quantitative tasks,
such as personalized sequential modeling to predict how workout
measurements (e.g. heart rate) will change across a workout, either
beforehand (i.e., based on a map of the intended route) or in realtime as the user exercises; (2) Qualitative tasks, such as identifying
important features that affect workout performance, or identifying
clusters of users based on common embedding structures; and (3)Recommendation tasks, such as recommending alternate routes
that will achieve a target heart rate profile.
In particular, we evaluate FitRec on two quantitative tasks: ‘workout profile forecasting’ and ‘short term prediction’ as shown in fig.
2. In the former, we are concerned with predictions such as estimating a user’s likely speed and heart rate profile given the activity
they intend to perform (e.g. cycling a particular GPS route); in thelatter we are interested in short-term prediction, e.g. determining
during an ongoing workout how the user’s heart rate will change
in the next few seconds or minutes. To apply to both tasks, FitRec
uses shared embedding layers while adapting a common predictive
component: a two-layer stacked LSTM module for workout profile
forecasting and an attention-based encoder-decoder module forshort term prediction. We also demonstrate the model’s capability
to provide personalized recommendations that could help users
toward their fitness objectives. Workout profile forecasting canbe used (for example) to recommend routes that match a user’s
desired exercise style, while short term prediction might be used
to dynamically help a user match their target heart rate. Fitness

While there are several works focused on analyzing human activities, we are (to the best of our knowledge) the first to investigate
personalized recommendation using sequential modeling methods
on fitness-oriented objectives and to conduct extensive experiments
on a real-world dataset.

2

RELATED WORK

Our work focuses on leveraging recent advances in RNN-based
sequential modeling for fitness prediction and recommendation.
We discuss the related work from each relevant area as follows.
Mining Sensor Data. One line of work falls into the category of
pervasive computing, which studies how to collect and handle datafrom mobile and wearable devices. The data may include various
types of human behavior such as sleep, exercise, surgery, health,etc. [16, 19, 23]. [1] discusses the challenges and recent advances in
population-scale pervasive health research. Their work emphasizes
the importance of sensor data for uncovering the causal relationships between human activity and health outcomes. [4] provides an
overview of data mining research in the healthcare domain and concludes that the capability to build context-aware and personalized
decision systems is important for the pervasive sensing market.
Recently, there is a growing trend toward modeling human wellness using sensor data. Farseev et al. collected users’ exercise dataand proposed a model to combine social network information topredict user wellness trends [9]. Particularly, they focused on BMI

(Body Mass Index) categorization during several periods for eachspecific user. [7] focused on detecting physical exercise types given
limited training data. They extracted features from exercise data
(e.g. heart rate, distance) and proposed an AdaBoost-based method
to predict the exercise type (e.g walking, aerobics, running, etc.).
Both of these works focus on handcrafting features from sensor
data, which are then applied to classification problems. Althoughsimilar in data modality, our work differs in terms of the prediction
tasks we consider, i.e., sequential modeling and sequence prediction.
Context-aware Modeling. Context-aware models have been successfully applied in many fields with abundant contextual information such as recommender systems, social networks, clinicalpredictions, etc. [8, 24]. Like these applications, fitness and exercise data naturally has heterogeneous input structure in the formof sequential measurements and contextual information. In [18]a recurrent model was proposed with a context-aware layer forpredicting a patient’s Blood Pressure (BP). The model considers
contextual input (e.g. age, gender, BMI) which is encoded via onehot vectors or numerical values. Experiments showed that adding
this contextual information to the model improves BP prediction
accuracy. Unlike their approach, we use attribute embedding andcontextual embedding modules consisting of LSTM layers to learn
from entire historical sequences. Furthermore, their blood pressure
measurement data does not consist of measurements within eachactivity, which differs from the data we consider.
RNN based sequential modeling. In recent years, Recurrent Neural Networks (RNNs) have shown remarkable effectiveness in modeling sequential data, such as text, audio, and clinical data [20],among others. [17] introduced an LSTM-based model to address
traditional multivariate time series prediction problems. They combined Convolutional Neural Networks (CNN) with a recurrent skip
layer to capture both long- and short-term dependencies in timeseries data. The model achieves significant improvements over LinearSVMs and ‘vanilla’ RNNs. [25] proposed a dual-stage attention
model to solve the problem of exogenous time series prediction.
They designed an input level attention mechanism along with temporal attention to capture the relations between input variables.
Similarly, we incorporate attention mechanisms in sequential data
to explore the relations between contextual variables in exercise
data.
Personalized Recommendation. Several works on personalizedrecommendation have tried to incorporate content (e.g. item attributes) and context (e.g. user clicks, purchases) to augment themodel in straightforward and typically ‘static’ scenarios (e.g. ecommerce). Recently, researchers from both industry and academia
have sought to develop systems for personalized fitness recommendation. [21] focused on personalized running route recommendation, by considering a variety of targets such as user preferences,
goals, and environment. [9] combined workout and social network
information from the same user in order to conduct wellness prediction. However, neither of these works focus on modeling sequential
fitness data (e.g. heart rate sequences). Though [9] released a public dataset, unlike ours it does not provide complete sequentialtraces for each workout, but rather includes hand-crafted statistics
extracted from each sequence.
While prior work has considered fitness and activity modelingfrom a variety of perspectives, we believe there is a research gap inTable 1: Notation
Notation

Description

Ttrain , Ttest

training set and test set for two prediction
tasks
contextual sequences of the current and themost recent workout
target sequence of the current and the mostrecent workout
attributes associated with the workout
length of input and target sequences
the number of sampled data points L
number of contextual sequences and eachsequence is associated with a variable.
number of attributes
number of distinct values of the ith attributedimension of attribute embeddings
dimension of contextual embeddings
embedding matrix of the ith attributeattribute embedding learned from the ith attributecontextual embedding learned from historical workout sequences

X, Z ∈ R N ×Ty, y′ ∈ RT
a
T
L
N
m
|a i |
D1
D2
Eai ∈ R |ai |×D 1
ea i ∈ R D 1
et ∈ RD 2

terms of studying low-level sequential information (e.g. heart rate,pace) as a means of learning more fine-grained models of users
and activities. This motivates us to model and publish a large-scale
workout dataset with heterogeneous sequential and attribute data,
and to conduct exploratory experiments on fitness recommendation
tasks. We expect that our work will facilitate future study of both
fitness and personalized heart rate profiling.

3 APPROACH
3.1 Notation and Tasks
Assume we are given N sequences X = (x1 , . . . , xT ) ∈ RN ×T
and one target sequence y = (y1 , . . . , yT ) ∈ RT , where xt =(x t1 , x t2 , . . . , x tN ) ∈ RN , yt ∈ R, and t ∈ [1,T ]. Here X can be considered as a combination of various contextual sequences (e.g. distance,
altitude) for the current workout and y is a target measurement
sequence (e.g. heart rate, speed). Further we assume there are N
historical contextual sequences Z = (z1 , . . . , zT ) ∈ RN ×T and one
historical target sequence y ′ = (y ′ 1 , . . . , y ′T ) ∈ RT , where Z andy ′ are from the most recent workout. For the dataset used in thiswork, the number of sampled data points L for each sequence of
the workouts is equal. Therefore, we also assume the same length T
for all sequences in our model. Note that this is not a requirementof our model, but merely a characteristic of the data we collect,
in which all sequences are quantized to have the same number ofsamples regardless of the original workout duration. The model
can be adapted to various sequence lengths by a simple padding
strategy. Each workout is also associated with multiple attributesa = (a 1 , . . . , am ) (e.g. userID, sport type, gender). Our notation issummarized in table 1.
We consider two types of sequential prediction problems: workout profile forecasting and short term prediction.
3.1.1 Workout profile forecasting. Given the contextual sequences
X, attributes a for a candidate workout, historical sequences Z andFitRec-Attn for short term prediction(Encoder-Decoder with attention)

FitRec for workout profile forecasting(2-layer stacked LSTM)

LSTM

LSTM

...

LSTM

LSTM

LSTM

LSTM

...

LSTM

Input attention

Contextual embedding module

...

LSTM

Temporalattention

LSTM

...

LSTM

Input attention

Attribute embedding layer

Linear Projection
LSTM

...

LSTM

LSTM

...

LSTM

Historicaluser
sequences

User input...

Sport input

Attributes

Detail of Contextual embedding module

Figure 3: Model structure for workout profile forecasting (FitRec) and short term prediction (FitRec-Attn). FitRec contains a
2-layer stacked LSTM and FitRec-Attn has an encoder-decoder module with dual-stage attention. The final outputs are coloredin blue.
y ′ , as well as the total time of the workout, predict the entire target
sequence y = (y1 , . . . , yT ). Notably, the input and target sequences
correspond to the whole workout and T equals the total length L of
the workout. This prediction task might correspond to a scenario
where a user specifies a route as well as their desired total time(i.e. how much time they plan to spend on the workout), and thesystem estimates their likely heart rate profile.1
3.1.2 Short term prediction. Given the contextual sequences X,
attributes a for a candidate workout, historical sequences Z and y ′ ,as well as the previous target sequence (y1 , . . . , yT −1 ), predict thetarget output yT at time step T . Here the input and target sequences
correspond to a fixed window of one workout (i.e. part of the whole
workout) andT equals the window size (i.e.T is smaller than L). Thistask is essentially a form of auto-regressive modeling that allows
us to predict short term heart rate dynamics during an activity.

3.2

Relation Between the Two Tasks

Our two tasks differ in terms of their given inputs, prediction outputs, and application scenarios as shown in fig. 2.
Workout profile forecasting aims to predict a user’s workout
performance (including speed and heart rate) given an expectedworkout time and route. The goal is to predict the overall trends
and characteristics of each measurement (e.g. max/min/avg heartrate) before the user starts the workout. As a result, the model canhelp the user find workout plans that meet their expectations on
1 We also experimented with a variant where the total time was not provided as input,

but found there to be excessive variance between (e.g.) running/sprinting/jogging
styles such that heart rates could not be reliably estimated without first knowing arough intended speed.

performance, or could be used to find alternate workout routes thatare similar to their previous routes in terms of how the user will
respond to them. This model can be adapted to applications suchas training schedule recommendation, alternative route recommendation, etc.
On the other hand, short term prediction focuses on forecasting
sequential measurements during an ongoing workout. This is similar to traditional problems of time series prediction with exogenous
variables [11, 25] but differs in that fitness data has measurements
both within and across activities which need to be modelled separately. Short term prediction is useful in scenarios like anomaly
detection and real-time decision making (e.g. advising a user thatthey should slow down in the next minute to avoid exceeding their
desired maximum heart rate).

3.3

Model Structure

As shown in fig. 3, we propose two models: FitRec for workout
profile forecasting and FitRec-Attn for short term prediction.
Both models share the same basic structure consisting of a contextual embedding module and an attribute embedding layer. Theselayers encode user attributes and historical information into anembedded representation that captures their latent individual attributes (e.g. their cardiovascular fitness, endurance, etc.) and canfacilitate personalized prediction. Following the embedding layers,
the two models have different predictive components. As shown on
the top left of fig. 3, FitRec consists of a 2-layer stacked LSTM which
simultaneously outputs the prediction of the target variable at alltime steps. FitRec-Attn includes an encoder-decoder network withan attention mechanism that outputs the prediction at each time

step one by one as shown on the top right of fig. 3. We describe thedetails for each component as follows.
3.3.1 LSTM to process sequences. In both models, we use Long
Short-Term Memory (LSTM) [13] to process the input and target
sequences. Given the input sequence X = (x1 , . . . , xt , . . . , xT ), an
LSTM learns an update function
ht = f (ht −1 , xt ),(1)

where ht , ht −1 are the hidden states of the LSTM at time step t
and t − 1, respectively, and xt is the input at time step t. Different
from traditional RNNs, the LSTM cell introduces several gates andmemory cells. We follow the LSTM structure from [25, 28]:i
sigmoid

© ª ©
ª 
ht −1
­ f ® ­sigmoid®
­ ®=­
®W
xt
­o® ­sigmoid®
«g¬ « tanh ¬st = st −1 ⊙ ft + it ⊙ gt

(2)

ht = tanh(st ) ⊙ ot ,where s is the cell state; g is the activation function applied on
the input; i, o and f are three sigmoid gates (‘input,’ ‘output,’ and‘forget,’ respectively); and h is the hidden state of the cell. W is thelearned weight matrix.
In the following sections, we use the notation ‘LSTM’ to represent the complete update function of an LSTM cell and ignore thecell state s for simplicity.
3.3.2 Context within and across activities. To address the difficulties in building personalized models, we consider learning attributeembeddings and contextual embeddings associated with the current workout. Specifically, for each attribute input (e.g. user, sport
type, gender), we use attribute embedding layers to obtain latent
representations of these attributes:
eai = Eai (ai ), ∀i ∈ [1, m](3)

where eai is the embedding for attribute ai .
To extract contextual features from historical activity measurements, we use two LSTMs to process the historical sequences Z
and y ′ respectively, then concatenate them and feed them into alinear projection layer to obtain a contextual embedding:h1,t = LSTM1 (zt , h1,t −1 ),h2,t = LSTM2 (yt′ , h2,t −1 ),(4)et = We [h1,t ; h2,t ] + be ,where h1,t and h2,t are the hidden states of the two LSTMs, et isthe projected contextual embedding, and We and be are trained
parameters.
After obtaining these embeddings, they are repeated at eachtime step and concatenated with the contextual embedding and the
input variables:
ut = [xt ; et ; ea1 ; . . . ; eam ],(5)

where ut is the new concatenated contextual sequence input attime step t. The sequences u = (u1 , . . . , uT ) ∈ RK ×T are then fed
into the cascaded predictive module for different tasks. Here K is
the dimension of each ut .

3.3.3 FitRec. We use a 2-layer LSTM model with a projection layeron the output for the workout profile forecasting problem, which
we call FitRec:
ht = LSTM(ut , ht −1 ),(6)yˆt = SELU(WNAT ht + bNAT ),where ht is the hidden state of the LSTM’s second layer, SELU is thescaled exponential linear unit activation function and WNAT andbNAT are learned parameters. The MLP output layer’s dimension
is set as the sequence length T . As shown in fig. 3, FitRec will
output the prediction for the whole workout (all T time steps)
simultaneously.
3.3.4 FitRec-Attn. Inspired by the success of the dual-stage attention RNN in modeling sequential data [25], we build FitRec-Attn – adual-stage attention encoder-decoder model on top of the attributeand contextual embedding module for short term prediction.
Given the concatenated contextual sequences u, we process themusing an encoder LSTM with input attention. Input attention helps
to find the importance of each input dimension among the concatenated contextual sequences. Consider uk = (u 1k , . . . , uTk ) for
the kth input dimension of u. An attention score stk is calculated
for uk and the previous encoder hidden state ht −1 , which represents the importance of that dimension at time step t. The weightis used to scale the original input and build a new input sequenceu˜t = (α t1ut1 , . . . , α tK utK ):stk = vs⊤ (Ws [ht −1 ; uk ] + bs ) 1 ≤ k ≤ K,
α tk = Íexp(stk ),K exp(s j )
t
j=1

(7)

where α tk is the normalized attention score, and vs , Ws and bs arelearned parameters. The newly computed ũt is used to update theencoder LSTMe :ht = LSTMe (ht −1 , ũt ).
(8)
Following the encoder, a decoder LSTM with temporal attention
is used to predict the target yt . Specifically, the temporal attention
score is calculated for each decoder hidden state dt and all encoder
hidden states, which represents how much the encoder hidden
state is related to the decoder. Then the attention scores are used
as weights to produce a context vector ct :lti = vl⊤ (Wl [dt −1 ; hi ] + bl ) 1 ≤ i ≤ T ,βti = Íexp(lti ),T exp(l j )
t
j=1

ct =

T
Õ

βti hi(9)i=1where lti and βti are the attention score and normalized attention
score for the t th decoder hidden state and the i th encoder hidden
state. vl , Wl and bl are learned parameters.
Once the context vector is obtained, it is concatenated with theprevious target variable yt −1 and used to update the decoder LSTM:
ỹt −1 = w̃⊤ [yt −1 ; ct −1 ] + b̃,
dt = LSTMd (dt −1 , ỹt −1 ),(10)where ỹt −1 is a linear transformation of the concatenated context
variable and target variable at time step t (as in [25]).

Given the previous target sequence (y1 , . . . , yT −1 ), each yt −1 isprocessed sequentially and ct and dt are updated. Finally, we obtainthe context vector and hidden state cT and dT at the last time step
T . These two are fed into a linear output layer to predict the target
variable yˆT as shown in fig. 3:yˆT = WAT [dT ; cT ] + bAT ,(11)

Table 2: Description of variables
Variable

Seq.

Unit

Heart Rate
MeasureTimestamp-ment
Distance
Speed

✓
✓
✓
✓

Beat per Minute (BPM)
Unix Timestamp
Mile
Mile per Hour (MPH)

Context-ual

UserID
Sport
Gender
AltitudeLongitude
Latitude

✗
✗
✗
✓
✓
✓

/
/
Male, Female
Meter
DegreeDegreeDerivedDer. Distance
Der. Speed✓✓

Kilometer (KM)
Kilometer per Hour
(KMPH)

where WAT and bAT are learned parameters.

3.4

Objective Function

For both workout profile forecasting and short term prediction, we
use the mean squared error as our loss function:
L=

1|N train |LÕ Õ(yˆt − yt )2 ,(12)y∈ Ttrain t =1

where |N train | is the number of total time steps for all workouts inthe training set, and L is the total length of each workout.

4

EXPERIMENTS

In this section, we first introduce the Endomondo dataset, a new realworld dataset for empirical study. Then we conduct experimentson this dataset and show the performance of our models for thetwo prediction tasks.

4.1

Table 3: Endomondo dataset statistics
Statistic

Original

FilteredRe-sampled# of workouts# of users
Avg. length (hours)
Avg. span (days)

253,020
1,104
5.998
/

167,373
956
1.486
766

102,343
887
1.059
733

The Endomondo Dataset

We collect data from endomondo.com. The data includes both measurements (i.e., heart rate, timestamps, distance, speed) and contextual data (i.e., longitude, latitude, altitude, gender, sport, user
identity). The original data has a high rate of missing measurements
for distance and speed, presumably due to the fact that these fields
are only available from certain devices. Hence we further introduce
two derived sequences: (1) derived distance: Calculate the distance between two data points given their latitudes and longitudes
via the Haversine formula [26]; (2) derived speed: Divide the derived distance and the time interval between two data points. In our
experiments, we use the derived speed instead of the original speed(which we found to be unreliable due to missing data or erroneous
magnitude values).
Table 2 lists the variables we use in this work (‘Seq.’ denotes
whether the variable is sequential or not; ‘Der.’ denotes that a variable is derived). Our data includes several other variables such as
weather, hydration, humidity, calories, cadence, etc., though thesevariables are much sparser than the variables listed (some are only
available for certain sport types, or from certain wearables); weleave the exploration of these data for future work.
We discard abnormal data points based on simple rules (e.g. overly
large magnitude, mismatching timestamps, abrupt changes in GPS
coordinates).2 We further keep users with at least 10 workout
records to make sure there is enough data to evaluate our personalization strategies. Table 3 summarizes our dataset. The original
dataset contains 253,020 workout records for 1,104 users.3 After
2 Which may occur due to sensor error, or more simply due to a user forgetting todisable the device after returning to their car (for example).
3 We collected a contiguous block of workouts, after which we collected the complete
workout histories for all users represented in that block.

filtering, there are 167,373 workout records for 965 users. On average each user has over 175 workout records that cover an averagespan of around 2 years. This makes the dataset promising for the
purpose of studying long-term temporal variation among users.
The original sequences as collected from Endomondo each contain L = 500 data points which are not necessarily sampled in
fixed-width intervals. The sampling interval may vary from seconds to minutes. Rather than considering the problem of imputing
missing values across long intervals, we simply discard workouts
with sampling intervals greater than 1 minute. Furthermore, theworkout data are truncated to 450 time points (out of an original 500)
to account for the fact that many of the workouts tend to complete
before the full 500 time-steps (e.g. during a user’s cooldown).
For the problem of short term prediction, it is important to have a
version of the dataset with a fixed sampling interval (i.e., so that our
predictions always correspond to a fixed amount of seconds in the
future). To this end, we produce a second version of the dataset thatis re-sampled into fixed intervals using cubic spline interpolation
followed by resampling in 10-second intervals. We discard workouts
in cases where resampling generates out-of-bounds values. After
resampling, we further discard those workouts with fewer than 300
samples (i.e., 300 × 10 seconds). Finally, we maintain a subset of102,343 workout records as shown in table 3.
Our data are highly multi-modally distributed, as the relationship between variables for one user differs significantly from that ofother users, and the structure of the data for one sport differs significantly from that of other sports (e.g. bicycling uphill vs. downhill
results in quite different heart rate patterns than running uphillvs. downhill). Our experiments show that the proposed models areable to learn these complex relations among variables.

Table 4: Parameter settings in our experiments

4.2

Model

FitRecFitRec-Attn

Hidden Size
Attribute embed. size
Contextual embed. size
Dropout
L2 Regularizer
Batch Size
Learning Rate

64
5
64
0.2
0.01
512
0.001

64
5
64
0.1
0.002
5120
0.005

Table 5: Workout profile forecasting on speed and heart rate.
∗ : significantly better than the second best score (p < 0.05).
Model

Global mean (train) 11.997
User mean (train)
10.690MLP
8.998
FitRec (U)
7.144
FitRec (U/S)
7.073
FitRec (U/S/G)
7.328
FitRec (U/S/C)
7.025∗

Training Procedure

We sort each user’s workouts in chronological order based on thefirst timestamp of each workout. Then we split and aggregate users’
first 80% of workouts for training, the next 10% for validation and thefinal 10% for testing. This is to make sure the contextual informationof a workout in the test set does not appear in the training set.
During training, the target data are not scaled while the input data
are scaled by representing the data points in terms of their z-scores.
We used Keras4 and Pytorch5 to implement our models. Ourmodels are trained using Adam [14]. We set the size of the LSTM
hidden state, attribute embedding and contextual embedding as
64, 5, and 64, respectively. Attribute embeddings are randomly
initialized and learned from scratch. We add dropout before andafter each LSTM layer and an ℓ2 regularizer on attribute embeddings
and contextual embeddings.
The best hyper-parameters are obtained via grid search on thevalidation set. The learning rate is chosen from {0.001,0.005,0.01},
the dropout rate from {0.1,0.2,0.3}, and our ℓ2 regularizer from {0.002,
0.005,0.01, 0.02}. The detailed parameter settings are as shown intable 4. Training is terminated after 50 epochs. The model that
achieved the best evaluation score on the validation set was run
on the test data to obtain the final performance score. To train themodels with contextual embeddings, we found it more stable to use
a pre-training technique. Specifically, we first initialize the model
parameters using the best model without contexutal embeddings,
then continue training until convergence. All code and data are
available publicly.6

4.3

Evaluation Metrics

We choose two popular metrics in sequential modeling: the Root
Mean Squared Error (RMSE) and Mean Absolute Error (MAE) toevaluate performance on our two prediction tasks:
v
u
u
t
L
Õ Õ
1
RMSE =(yt − yˆt )2(13)|N test |t =1y∈ Ttest

MAE =1|N test |LÕ Õ|yt − yˆt |,(14)y∈ Ttest t =1

where |N test | is the number of total time steps of all workouts on
the test set Ttest . Note that in our setting the two metrics havequite different semantics. For example, a model which correctly
4 https://keras.io/
5 https://pytorch.org/
6 https://github.com/nijianmo/fit-rec

Speed (KMPH)
RMSE MAE
8.532
6.418
4.411
2.472
2.381
2.396
2.384

Heart rate (BPM)
RMSE
MAE
24.302
20.11718.25618.78417.32518.20717.051∗18.18515.09013.91814.14213.012
13.831
12.847∗

identified heart rate ‘trends’ but was shifted in time or scale might
be considered an acceptable model; such predictions would incur ahigh MSE but a more moderate MAE.

4.4

Results: Workout profile forecasting

We compare FitRec to two baselines and also conduct ablation tests.
Our baselines are:
• Global mean. Predict the target variable at each time stepas the global mean of the training set.
• User mean. For each workout from a user u, predict thetarget variable at each time step as the mean value of all u’s
sequences in the training set. This serves as a naïve personalized baseline.
• MLP. A variant of a traditional Multilayer Perceptron thatconcatenates the learned attribute embeddings, contextual
embeddings and contextual inputs together as its input. The
MLP predicts the target variable at each time step simultaneously. The dimension of the MLP’s output layer is the total
length of the sequence. This baseline is used to demonstratethat the LSTM component of FitRec is better at modelingthese contextual inputs compared to an MLP.
• FitRec (U). FitRec model with user embedding only.
• FitRec (U/S). FitRec model with user embedding and sport
embedding.
• FitRec (U/S/G). FitRec model with user embedding, sport
embedding and gender embedding. This model is used tomeasure whether the gender embedding improves FitRec.
• FitRec (U/S/C). FitRec model with user embedding, sport
embedding and contextual embedding.
We use FitRec to predict two types of workout profile: (1) speedand (2) heart rate, given the expected workout time and a specified
route. In particular, the route is given as a distance sequence andan altitude sequence along the complete workout.
As shown in table 5, FitRec (U/S/C) with all embeddings outperforms each of the traditional baselines. Specifically, it decreases
the RMSE of the best baseline (MLP) by 21.93% and 6.6% on speed
prediction and heart rate prediction, respectively. The user meanbaseline achieves lower RMSE and MAE than the global mean baseline, demonstrating the need for personalized models for this task.
FitRec (U) with user embeddings achieves further improvements
over the user mean baseline, which shows the efficacy of the learned
embeddings in capturing individual patterns of users.

Table 6: Short term heart rate prediction. ∗ : significantly better than the second best score (p < 0.05).

(a) Biking

4.5

Model

Heart rate (BPM)
RMSE
MAE

Windowed MLP
Seq2Seq+A
DA-RNN
FitRec-Attn (U)FitRec-Attn (U/C)

3.035
2.807
2.816
2.795
2.783∗

1.919
1.720
1.733
1.705
1.695∗

Results: Short term prediction

For short term prediction, we consider three baselines for autoregressive modeling and conduct ablation tests to compare against
the performance of FitRec-Attn.
The baselines are:(b) Running

Figure 4: Examples of predicted heart rate for different
sports, altitude has much greater influence on heart rate andspeed change when biking as compared to running (for example).
Furthermore, our ablation tests exhibit improvements due to theattribute embeddings and the contextual embedding. FitRec (U/S/C)
outperforms FitRec (U) and FitRec (U/S), except for the MAE on
speed prediction. This may be because speed is sensitive to sport
type, and the previous workout that we use to learn the contextual
embedding has a different sport type from the current workout;
in such cases the contextual embedding may not be informative.
Missing data issues aside, this model could, in the future, be straightforwardly extended to consider other attributes such as weather,humidity, etc.
It is also surprising to find out that an auxiliary gender embedding does not help improve the prediction accuracy of our model:
FitRec (U/S/G) performs worse than FitRec (U/S). There are a few
possible reasons. One is that our data includes 863 males, 80 females
and 13 unknown users. Such an imbalance in data might make it
difficult to learn patterns across different genders. Another possibility is that the user embedding might already encode the user’s
gender information. As such, an auxiliary gender embedding does
not bring any further benefit to the model.
Figure 4 gives examples of the predicted heart rate and speed forbiking and running instances. In both cases the predicted sequences
closely match the trend of the ground-truth during the workout.
Notably, FitRec is able to learn the semantics of different sport types.
For example, entering a downhill portion of a bike ride causes auser’s heart rate to drop substantially, whereas running downhill
does not result in a heart rate reduction.

• Windowed MLP [10]. A conventional time series modelthat considers a sliding window of the target variable’s previous values, and predicts the target variable at the current
time step.
• Seq2Seq+A [6]. A competitive encoder-decoder baselinethat is widely used in sequential modeling. This model includes temporal attention between the encoder and decoder.
• DA-RNN [25]. A state-of-the-art attention based RNN model
for auto-regressive time series modeling with exogenous
variables. The model includes an input attention mechanismthat assigns weights to each input variable as well as temporal attention.
• FitRec-Attn (U). FitRec-Attn model with user embedding
only.
• FitRec-Attn (U/C). FitRec-Attn model with user embedding
and contextual embedding.
Because our ablation tests showed that the gender embeddings did
not improve results for short term prediction, we did not include
them in this section.
We use FitRec-Attn to predict the heart rate at each time step t
given the speed and altitude sequences, the previous T −1 heart rate
values, as well as the attribute embeddings and contextual embedding. We choose T = 10 in the experiments. We also experimented
with T = 5, 15, 20 and found that values greater than 10 yield marginal improvement. When implementing Seq2Seq+A and DA-RNN,
we only use speed and altitude sequences since they do not include
the embedding modules for user and contextual embeddings. The
baseline windowed MLP only considers the previous T − 1 heart
rate values as in traditional autoregressive models.
As shown in table 6, FitRec-Attn (U/C) achieves the best performance compared with the other baselines. When only considering
speed and altitude as inputs, attention-based Seq2Seq outperforms
the more sophisticated DA-RNN. This may be because that the
number of input variables is relatively small (2 in this case), leaving
the input attention of DA-RNN as but a source of extraneous complexity. On the other hand, by incorporating user embeddings andcontextual embeddings, FitRec-Attn (U/C) improves results on both
metrics compared to DA-RNN. This demonstrates the effectivenessof input attention when the input variable is high dimensional.

Table 7: Route recommendation performance

User mean (train)
MLP
FitRec (U)FitRec (U/S)
FitRec (U/S/C)

5

AUC

Hit@10

NDCG

0.500
0.506
0.639
0.643
0.673

0.100
0.106
0.214
0.234
0.272

/
0.212
0.262
0.266
0.284

PERSONALIZED RECOMMENDATION(a) Map of the ground-truth route.

So far we have shown the effectiveness of our model at generating ‘black box’ heart rate predictions given contextual inputs. In
this section we illustrate the ability of FitRec to provide personalized recommendations in practice. We consider the following two
recommendation tasks:• Workout route recommendation: Given expected workout
criteria (e.g. an expected workout time, an idiosyncratic heart
rate or speed curve), we suggest routes (i.e. historical routes
from all users) that will match the user’s expectation. Thismight be applied if a user has particular heart rate targets in
mind, or simply wishes to identify alternate routes that are‘similar to’ (in terms of heart rate profile) a route a user hastaken previously. Such a system could be used (for example)
by a user who is traveling but wants to maintain their regularexercise routine by identifying qualitatively similar routes.
• Short term heart rate prediction: Predict whether a user’s
heart rate will exceed some threshold if they continue at thecurrent pace during their workout. This might be used tobuild a personalized assistant that guides a user’s behavior(e.g. speed up or slow down) during a workout.

5.1

Route Recommendation

5.1.1 Evaluation Methodology. We evaluate our route recommendation task in terms of ranking performance. To do so, we first use
FitRec to generate a ranked list of candidate routes, given a user’s
expected workout criteria. Then, for each workout in the test set(positive sample), we consider its route (i.e. distance and altitude
sequences) as the positive route, and randomly sample another 100
workouts as negative routes. That is, given target workout criteria(heart rate etc.) as input, we are evaluating the model’s ability tocorrectly identify the real workout that generated these criteria
by assigning it a higher score than randomly chosen alternatives.
Given these positive and negative routes, FitRec generates a predicted heart rate sequence for each of them. The similarity between
the predicted sequences and the ground-truth heart rate sequenceis calculated using fast dynamic time warping (fastDTW) [27]. We
use DTW to measure similarity rather than the ℓ2 loss because it isless sensitive to temporal shifting.
After sorting these similarity scores in descending order, we
obtain the ranked list of all routes, and are interested in the ranked
position of the positive route amongst the negative routes. Herewe consider three recommendation metrics that are commonlyused in ranking-based recommendation [12]: AUC, Hit Rate@K(Hit@K) and Normalized Discounted Cumulative Gain (NDCG).
AUC accounts for the overall ranking of the positive route, Hit@K(b) Map of the recommended alternate route.

Figure 5: Examples of a recommended route. The model recommends an alternate route that is similar to the groundtruth (i.e., both routes are comparable hill-climbs).

measures whether the positive route appears in the top K of thelist, and the NDCG uses a graded relevance criterion to evaluate
the overall quality of the ranking.
5.1.2 Performance Analysis. Table 7 shows the performance ofroute recommendation on the test set. We find that baseline models exhibit poor performance on personalized route modeling: (1)The user mean baseline does not take into account the route (i.e.,
distance, altitude) information; and (2) the MLP model considers
routes as input during learning, but fails to rank the positive routes
higher than other routes (its AUC of 0.506 is close to that of the User
Mean baseline). On the other hand, FitRec with user embeddings isable to recognize the difference between routes and outperforms
baseline models in all metrics. After adding the attribute embeddings and contextual embedding, the model’s performance is further
improved.
Figure 5 presents an example of a recommended route given
by our model (i.e., the route ranked first on the ranked list, otherthan the ground truth) along with the ground-truth route. As theroute map shows (marked in blue),7 both the ground truth routeand the recommended route are ‘hill climbs’ of approximately thesame length and difficulty (the red bounding box in each route map
is the region with the greatest altitude during the workout). Thisshows the effectiveness of the model to recommend alternate but
semantically similar routes.

7 Snapshot from endomondo.com

Figure 6: Performance of short-term excessive heart rate prediction(a) User embedding learned forheart rate prediction.

(b) User embedding learned forspeed prediction.

Figure 8: 2d t-SNE of the learned user embedding, and its relationship to user segments in terms of heart rate and speed.

5.3

Figure 7: Example of short term excessive heart rate prediction.

5.2

Short term heart rate prediction

The goal of this task is to predict whether a user’s heart rate will
exceed some level if the user continues their workout at the current speed. We assume there is a threshold value predefined byeach user that she/he considers to be a desirable maximum heartrate. We consider four different threshold values: 180, 185, 190, 195
and compute three metrics to demonstrate the effectiveness of ourmodel to anticipate rapid heart rates in test sequences.
5.2.1 Evaluation Methodology. For each workout we construct
two binary sequences (b1 , . . . , b L ) and (b1∗ , . . . , b L∗ ) (of predictions),
where bt / bt∗ equals 1 if the groundtruth / predicted (respectively)
heart rate at timestep t exceeds a pre-defined threshold (zero otherwise). We then calculate Precision, Recall, and Macro-F1 scoresbased on these two binary arrays to evaluate the performance ofpredicting whether the speed will exceed the threshold.
5.2.2 Performance Analysis. Figure 6 demonstrates the performanceof FitRec-Attn when given different threshold values. As we can see,
FitRec-Attn (U/C) achieves a better F1-score compared to other baseline models for all threshold values. Specifically, the improvementof FitRec-Attn increases as the threshold value increases.
Figure 7 shows an example of short term heart rate prediction.
Note that for this task, the model takes as input the observed heart
rate values for the previous T −1 time steps; therefore the prediction
is more accurate than in our previous experiment. In the figure
we see an abrupt change in altitude, following which the model
successfully predicts the heart rate will reach a high value (i.e. over
185 BPM).

What have the user embeddings learned?

To qualitatively explain what has been captured by the learneduser embeddings, we use t-SNE [22] to project them into lowdimensional vectors. Figures 8a and 8b show the 2d t-SNE of theuser embeddings learned for the tasks of predicting heart rate andspeed, respectively.
For heart rate, we calculate the average heart rate of each user
based on all of their workouts. We classify them into four categories by threshold values: < 130, < 145, < 160, and ≥ 160 (BPM).
Similarly for speed, we calculate the average speed of each user
for all of their workouts. We classify them into three categories
by thresholds: < 10, < 25, and ≥ 25 (KM per Hour). As shown in
figs. 8a and 8b, users that fall into the same class (i.e., with similar
average heart rate and average speed) seem to have closer userembeddings.

6

CONCLUSIONSWe present FitRec – a system that addresses sequential prediction
problems in fitness and exercise data and can serve as a building
block for developing personalized fitness recommendation applications. We achieve this by learning embedded representations fromauxiliary information such as user identity, sport type and historical
workout sequences, each of which is incorporated into an LSTM
based sequential modeling framework. We verify that the modelis capable of providing high quality prediction on tasks centeredaround estimating users’ heart rate sequences, such as workout
profile forecasting and short term prediction. We conduct extensive
experiments on a new real-world dataset and demonstrate that themodel outperforms other baselines without auxiliary embedded
representations consistently on both tasks. Furthermore, the modelshows promising performance on recommendation tasks such asrecommending alternate workout routes and short-term excessive
heart rate prediction. Visualizations also show that the learned
embedded representations are useful for identifying distinct user
segments. In the future, we are interested in applying FitRec to recommendation tasks with more complex scenarios (i.e. safety-awareroute recommendation) and incorporating other auxiliary data (i.e.
interactive recommendation).
Acknowledgements. This work partly supported by NSF#1750063.

REFERENCES[1] Tim Althoff. 2017. Population-Scale Pervasive Health. IEEE Pervasive Computing
16 (2017), 75–79.
[2] Tim Althoff, Rok Sosic, Jennifer L. Hicks, Abby C. King, Scott L. Delp, and Jure
Leskovec. 2017. Large-scale physical activity data reveal worldwide activity
inequality. In Nature.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural MachineTranslation by Jointly Learning to Align and Translate. CoRR abs/1409.0473(2014).
[4] Hadi Banaee, Mobyen Uddin Ahmed, and Amy Loutfi. 2013. Data Mining for
Wearable Sensors in Health Monitoring Systems: A Review of Recent Trends andChallenges. In Sensors.
[5] Zhengping Che, Xinran He, Ke Xu, and Yan Liu. 2017. DECADE: A Deep Metric
Learning Model for Multivariate Time Series. In NIPS-ML4H.
[6] Kyunghyun Cho, Bart van Merrienboer, ÃĞaglar GülÃğehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase
Representations using RNN Encoder-Decoder for Statistical Machine Translation.
In EMNLP.
[7] Alok Kumar Chowdhury, Aleksandr Farseev, Prithwi Raj Chakraborty, Dian
Tjondronegoro, and Vinod Chandran. 2017. Automatic classification of physicalexercises from wearable sensors using small dataset from non-laboratory settings.
2017 IEEE Life Sciences Conference (LSC) (2017), 111–114.
[8] Miguel Ramos de Araujo, Pedro Manuel Pinto Ribeiro, and Christos Faloutsos.
2017. TensorCast: Forecasting with Context Using Coupled Tensors (Best Paper
Award). In ICDM.
[9] Aleksandr Farseev and Tat-Seng Chua. 2017. TweetFit: Fusing Multiple Social
Media and Sensor Data for Wellness Profile Learning. In AAAI.
[10] Ray J. Frank, Neil Davey, and Stephen P. Hunt. 2001. Time Series Prediction and
Neural Networks. Journal of Intelligent and Robotic Systems 31 (2001), 91–103.
[11] Tian Guo, Tao Lin, and Yao Lu. 2018. An interpretable LSTM neural network for
autoregressive exogenous model. CoRR abs/1804.05251 (2018).
[12] Xiangnan He, Tao Chen, Min-Yen Kan, and Xiao Chen. 2015. TriRank: Reviewaware Explainable Recommendation by Modeling Aspects. In CIKM.
[13] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory.
Neural computation 9 8 (1997), 1735–80.

[14] Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).
[15] Timo Koskela, Mikko Lehtokangas, Jukka Saarinen, and Kimmo Kaski. 1996. Time
Series Prediction with Multilayer Perceptron, FIR and Elman Neural Networks.
In World Congress on Neural Networks.
[16] Takeshi Kurashima, Tim Althoff, and Jure Leskovec. 2018. Modeling Interdependent and Periodic Real-World Action Sequences. WWW 2018 (2018), 803–812.
[17] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling
Long- and Short-Term Temporal Patterns with Deep Neural Networks. In SIGIR.
[18] Xiaohan Li, Shu Wu, and Liang Wang. 2017. Blood Pressure Prediction via
Recurrent Models with Contextual Layer. In WWW.
[19] Zhiyuan Lin, Tim Althoff, and Jure Leskovec. 2018. I’ll Be Back: On the Multiple
Lives of Users of a Mobile Activity Tracking Application. WWW 2018 (2018),
1501–1511.
[20] Zachary Chase Lipton. 2015. A Critical Review of Recurrent Neural Networks
for Sequence Learning. CoRR abs/1506.00019 (2015).
[21] Benedikt Loepp and JÃĳrgen Ziegler. 2018. Recommending Running Routes:
Framework and Demonstrator. In Workshop on Recommendation in Complex
Scenarios.
[22] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.
JMLR 9, Nov (2008), 2579–2605.
[23] Emma Pierson, Tim Althoff, and Jure Leskovec. 2018. Modeling Individual Cyclic
Variation in Human Behavior. In WWW.
[24] Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir
Zadeh, and Louis-Philippe Morency. 2017. Multi-level Multiple Attentions forContextual Multimodal Sentiment Analysis. In ICDM.
[25] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W.
Cottrell. 2017. A Dual-Stage Attention-Based Recurrent Neural Network forTime Series Prediction. In IJCAI.
[26] C Carl Robusto. 1957. The cosine-haversine formula. The American Mathematical
Monthly 64, 1 (1957), 38–40.
[27] Stan Salvador and Philip Chan. 2007. Toward accurate dynamic time warping inlinear time and space. Intell. Data Anal. 11 (2007), 561–580.
[28] Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent Neural
Network Regularization. CoRR abs/1409.2329 (2014).