Dynamical Wasserstein Barycenters for Time-series Modeling
Authors: Kevin C Cheng
Shuchin Aeron
Michael C Hughes
Eric Miller
Venue: NeurIPS 2021 Poster (2021)
Keywords: time series
transition
optimal transport
Wasserstein barycenter
displacement interpolation
simplex random walk
state space
dynamical model
Riemannian optimization
human activity recognition

Abstract: Many time series can be modeled as a sequence of segments representing high-level discrete states, such as running and walking in a human activity application. Flexible models should describe the system state and observations in stationary ``pure-state'' periods as well as transition periods between adjacent segments, such as a gradual slowdown between running and walking. However, most prior work assumes instantaneous transitions between pure discrete states. We propose a dynamical Wasserstein barycentric (DWB) model that estimates the system state over time as well as the data-generating distributions of pure states in an unsupervised manner. Our model assumes each pure state generates data from a multivariate normal distribution, and characterizes transitions between states via displacement-interpolation specified by the Wasserstein barycenter. The system state is represented by a barycentric weight vector which evolves over time via a random walk on the simplex. Parameter learning leverages the natural Riemannian geometry of Gaussian distributions under the Wasserstein distance, which leads to improved convergence speeds. Experiments on several human activity datasets show that our proposed DWB model accurately learns the generating distribution of pure states while improving state estimation for transition periods compared to the commonly used linear interpolation mixture models.

Dynamical Wasserstein Barycenters for Time-series Modeling
Kevin C. Cheng Tufts University
Shuchin Aeron Tufts University shuchin aeronêtufts.edu
kevin. cheng@tufts.edu
Michael C. Hughes
Eric L. Miller
Tufts University
Tufts University Fric.miller@tuffee
michael. m www..sututsssts....ooooooo.... edu
Abstract
Many time series can be modeled as a sequence of segments representing high- level discrete states, such rumning and walking in human activity application. Flexible models should describe the system state and observations in stationary "pure-state" periods well transition periods between adjacent segments, such as a gradual slowdown between running and walking. However, most prior work assumes instantaneous transitions between pure discrete states. We propose dynamical Wasserstein barycentric (DWB) model that estimates the system state over time well the data-generating distributions of pure states in an un- supervised manner. Our model assumes each pure state generates data from a multivariate normal distribution, and characterizes transitions between states via iissllacemen-interpllaaa
specified by the Wasserstein barycenter. The system state is represented by barycentric weight vector which evolves over time via a random walk on the simplex. Parameter learning leverages the natural Riemannian geometry of Gaussian distributions under the Wasserstein distance, which leads to improved convergence speeds. Experiments several human activity datasets show that our proposed DWB model accurately learns the generating distribution of pure states while improving state estimation for transition periods compared to the commonly used linear interpolation mixture models.
Introduction
We consider the problem of estimating the dynamically evolving state of a system from time-series data.' The notion of "sstate" in such contexts typically is modeled one of ways. For many problems, the system state is vector of continuous quantities (Kalman, 1960; Krishnan et al., 2016), perhaps constrained in some manner. Alternatively, discrete-state models take on one of countable number of options at each point in time, as exemplified by hidden Markov models (HMMs) (Rabiner, 1989) or "switching state" extensions (Ghahramani and Hinton, 2000; Linderman al., 2017).
Many time-series characterization problems of current interest warrant a hybrid of continuous and discrete state representation approaches, where the system gradually transitions in continuous manner among finite collection of "pure" discrete states. For example, in human activity recognition using accelerometer sensor data (Bi et al., 2021), some segments of data do correspond distinct activities (run, sit, walk, etc.), suggesting discrete state representation. However, when using sensors with gngghh sampling rates, transition periods when the system is evolving from one state to another (e.g. the individual accelerates from standing to running over few seconds) can be well
'Code available http::///gtthub..co//eeiii oatpa////o/aaaaaaaaaaaaaaaaaaaar Gaussiaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

resolved. Gradual evolution between pure states can also be observed in other domains of time series data, such as economics (Chang et al.. 2016) or climate science (Chang et al., 2020). Characterizing these systems requires a model with continuous state space to capture the gradual evolution of the system among the discrete set of pure states.
Motivated by this class of applications, we consider models for time series in which the system's ' dynamical state is specified by a vector of convex combination weights for mixing a set of data- generating distributions that define the individual pure states. Many approaches, such as mixture state vector (Rudin, 1976) as assignment probabilities;
models, interpret such dimplex-constraing that is, the system is assumed to be in pure state with uncertainty as to which. As result, the data-generating distribution at moments of transition is a convex, linear combination of the pure-state emission distributions. While useful in many applications, such linear interpolation does not capture the gradual transitions among pure states in the time series of interest to us.
To illustrate the shortcomings of linear interpolation, consider the toy data task in Fig. 1, where a system gradually transitions between three pure states over time. During the transition periods (e.g. at times 600 and 1400), the linear interpolation method infers a data-generating distribution that is multi-modal, shown in Fig. 1(b). If refer our pure states as "walk" and "run," " this approach models the walk-to-run transition as sometimes walk and sometimes run. This does not intuitively capture the gradual nature of accelerating from walk to run in our intended applications.
To overcome this limited representation, we consider another way to mix together pure-state distri- butions: papplaaemmnttinterpooaate
(McCann, 1997), which is related the Wasserstein distance (Peyré and Cuturi, 2019), a metric over the space of probability distributions (Sriperumbudur et al., 2010). While the work of McCann (1997) is limited combining two distributions, it is extended to multiple distributions using the notion of a Wasserstein barycenter (Aguch and Carlier, 201 I). Fig. I(c) shows how a Wasserstein barycenter approach to time-series modeling infers data-generating distributions during transitions that are not multi-modal but instead place mass in between where the two pure-state distributions do. This intuitively captures gradual transition between two pure states.
Inspired by this framework, in this work we develop a dynamical Wasserstein barycentric (DWB) model for time series intended to explain data arising system evolves between pure states. Our model uses a barycentric weight vector to represent the system state. Given an observed multivariate time-series and desired number of states K, all parameters are estimated in unsupervised way. Estimation simultaneously learns the data-generating distributions of K pure discrete states as well as the K- K-simplex valued barycentric weight vector state each timestep.
Given the nature of our model, we require that the state lie in the simplex at every timestep, a constraint not respected by the Gaussian noise that drives common continuous-state processes (Welch, 1997). Building on work by Nguyen and Volkov (2020), we employ a random walk where the driving noise comes from independent, identically distributed (IID) draws from mixture of two Beta distributions, representing stationary and transitional dynamics. By blending the current state and a mixture-of Betas draw in convex manner, we construct a new state that lies in the simplex.
To specify the emission distributions of our model, we assume that each pure state generates data from a multivariate Gaussian. While Gaussian model may not be suitable in all applications, this choice allows us to exploit useful properties of Gaussian densities under the Wasserstein distance (Takatsu, 2011), 1). Specifically, a dlosed-form expression exists for the Wasserstein distance between Gaussians, the Wasserstein barycenter among Gaussians can be computed via a simple recursion, and the estimation of the Gaussian mean vectors and covariance matrices can be performed conveniently over a Riemannian product manifold. Empirically, we find our proposed DWB model with Gaussian pure-states performs well on human activity datasets, accurately characterizing both pure-states emission distributions and capturing the system state in pure states and transition periods.
We introduce displacement papplacementtinteepplatiii model for time series where the data- generating distribution is given by the weighted Wasserstein barycenter of a set of pure-state emission distributions and a time-varying state vector. We propose a simplex-valued random walk with flexible dynamical structure to model the system state. We exploit the Riemannian siructure of Gaussian distributions under the Wasserstein distance for parameter estimation for faster convergence speed. We evaluate on human activity data and demonstrate the ability of our method to capture stationary and transition dynamics, comparing with the linear interpolation mixture model and with continuous state space model.
Contributions:
2

---

Ground Truth Distributions
(b)
interpolat
[c]
Wasserstein Barycenter
[d]
Estimated State
Pa
Pe=60o
Pt-euu
D
R2]
60
OCC State Parameeters
Ground Iruth State
600
1400
P1-1430
Sy 2
Figure I: (a) Three Gaussian distributions P1, P2, P3 each representing distinct activities with cor- responding means m1, m2, my that are marked ("x') in all plots as reference points. The time series is drawn from time-varying distribution according to the ground truth state vector as the system transitions linearly from P1 to P2. 1, ..., 1000, then continues to f3, 1000. 1800. (b) Under the linear interpolation state-transition model, the PDF at select times of ( 600. 1400 are linear combinations of Pl P2, P3. (c) Alternatively, the proposed displacemenn-interpolaaioo transition model between pure-states given by the Wasserstein barycenter translates the mass between pure states. (d) Following the Wasserstein barycentric model for time series, our proposed method accurately recovers both the pure state distributions and state vector from the observed time series.
Outline: Sec. 2 provides an overview of the Wasserstein distance, barycenter, and the associated geometry for Gaussian distributions. We then formalize our problem statement and estimation problem in Sec. 3. Sec. 4 discusses the model parameters, covering the dynamical simplex state-space model in Sec. 4.1 and the pure-state parameters in Sec. 4.2. Sec. 5 discusses the optimization of our model parameters leveraging geometric properties of the Wasserstein distance for Gaussians. Finally, Sec. 6 evaluates and discusses the advantages of our model in the context of human activity data.
Technical Background
A core component to approach is to model the intermediate transition states of time series using the Wasserstein barycenter (Agueh and Carlier, 2011) 1) of probability distributions, which generalizes the displacement- paaplacement-inerpplattiio framework (MeCann, 1997) beyond two distributions. We refer the works of (Peyré and Cuturi, 2019) and (Villani, 2009) for a detailed discussion on these concepts.
Consider the space of all Borel probability measures over Rd with finite second moment. The squared Wasserstein-2 distance for two distributions Pl. P2 with squared Fuclidean ground cost is defined as,
[ Rd
W P2) Mell(pI.e))
inf
|| B11 M (da, dB),
(I)
where 11(p), P2) is the of all joint distributions with marginals Pt P2. and M is the optimal transport plan, the element that minimizes the total transportation cost. When these measures are Gaussian, parameterized by their mean vectors m, E Rd and symmetric positive-defime covariance matrices S, E Symq. , the squared Wasserstein-2 distance has a closed form solution (Takatsu, 201 1),
W2 (pi (mi, S1),P2(m)) S2)) |m - m221 + (S) + S2 12-22(2 (
82(m1.m2)
(²)
This distance decomposes into sum of the squared Euclidean distance between mean vectors, E2 (m,, m2), and the squared Bures distance (Bhatia et al., 2017) between covariance matrices, B (S1, S2). Thus, the contributions of the mean and covariance to the Wasserstein distance between Gaussians decoupled, property that is uncommon for Gaussian distribution distances (Nagino and Shozakai, 2006) and has important implications for optimization and barycenter computation.
The Wasserstein barycenter extends the notion of the weighted average of points in Rd using the Fuclidean distance to the space of probability distributions with the Wasserstein distance. Given set of K measures and the barycentric coordinate vector on the K K-simplex, e A the Wasserstein
3

---

barycenter is the measure that minimizes this weighted Wasserstein distance to the set of measures,
PB (2, {px) k
argmin (EP)
(Pk.
(3)
When are Gaussian distributions, PB defined in (3) is itself Gaussian (Agueh and Carlier, 2011) with parameters m, Sg. Again, because of the decomposition of the Wasserstein distance in (2), the Wasserstein barycentric problem in (3) can be solved separately for its components,
mB argmin mER
[e[e)e m),
SB argmin
Σ = S).
(4)
SESym
The optimal mean be computed in closed-form: mB matrix can be solved via the fixed-point iteration proposed in Álvarez Esteban et al. (2016).
Σn x: [k)mm. The optimal covariance
Problem Formulation
Model Definition. Our DWB model's data-generating process is illustrated in Fig. 2. First, the pure-state emission parameters θ = {(m Sk)) de- fine a Gaussian distribution Pv for each pure state k. Second, the state vector 3t defines the system state at and lies on the simplex. Given O and æt, we can form a time-varying Gaussian dis- tribution PB, (z,, O) Nmmg, Sg.) for each f. which is a barycentric com- bination of the K pure Gaussians using weights x; via (4). We can write the states for entire sequence prised of an initial state and a sequence of simplex-valued state vectors, denoted X {oo, (=
Do.
r'
Y1
Y2
Y
Yr
X
X1
X2
X3
XT
+
m2
S
PB
PD3
Per
R 1:X
P1
P
PT
X, com-
Y,. Y2. Va. Ys. Y
yy
Figure 2: Graphical model diagram of proposed dy- namic Wasserstein barycenter (DWB) time series model. For each window of data, indexed by t. our model forms an emission distribution PB. that is the Wasserstein barycen- ter given known pure-state Gaussian parameters @ (m,, S k=1 simplex-valued state sequence {x))== is drawn from a random walk using Beta-mixture draws "Y (Sec. 4.1), with hyperparameters H = {w, 4p,1 bo.1) and initial state ag- Random variables are denoted by circles. Figure shown has n = 2, = Colors correspond to terms in (9).
Data Preprocessing. We are given vector-valued time series of observa- tions V- E Rd r 1,..., T. Instead of modeling this data directly, to im- prove smoothness pirical distribution of sliding windows of 2n samples (Aghabozorgi al., 2015) strided by 8 samples. We retain only windows with complete data, with start times corresponding to t = (6 + 126 + 1), sequentially as t E {1, 2... .). A window indexed at t corresponds to window centered at t T= (ō(t - 1) + n + 1), which provides an estimates the underlying distribution at y,. For each window location t, compute unbiased empirical Gaussian distribution P. N(m,, St) where,
> and time-varying weight vector Kt. The
model the em-
(T-(2x+ 1 Jō + 1, which we index
2n+1
2n++1
m. 2n. (=1
y5(-1)++
S.
Σ (18(1-1)4 - m. m)) -m.T" (5)
2n
Minimizing the Wasserstein distance between this sequence of empirical distributions Pr and the model-predicted distributions PB, drives our model's parameter learning.
Estimation Objective. In practice, we are given an observed sequence of empirical distributions {p.) and a desired number of states K. We wish estimate the state sequence X and emission parameters O We pose the estimation of X and θ as the solution to an optimization problem seeking to balance fidelity to a prior model on the parameters of interest with the desire to minimize the time integrated Wasserstein distance between the predicted and observed distributions:
T
X θ argmin - log (p (p(O) X,e
()))
+
W (pc, PB, (2., 9))
(6)

---

The scalar weight À > O trades off the model's s fit to data (measured by the Wasserstein distance) with the probability of the state sequence X and pure-state emission parameters O under assumed prior distributions. Our chosen priors p( X) and the p ()) are covered in the following section.
4 Model Parameter Priors
33 [0,0,1]
4.1 Prior on Simplex States over Time
Here we develop the transition model that generates the sequence of state vec- tors 2g. 21.... æT. We assume a first- order Markovian structure: p( X)
[3]e (1 31
Recall that each state vector lies on the K K-dimensiooal sim- plex. The geometry of the state space in the case of K = states is shown in Fig. 3, where x; lies in the convex hull of the three simplex vertices, the unit coordinate vec- tors e1, E2, e3. Each vertex is associated with a pure-state in our problem. For a more general K -tatae problem, this is gen- eralized to the K didimeesionnal simplex, denoted A
p(x+) II p(x,|_x,_
a, x + -x()xx
Yrllle; (1 >Y,D
xet1
[0,1,0]
= [1,0,0]
Figure 3: Given current state 2t, we transition to next state #++1 by averaging K step-to-vertex updates. For each = 1, 1,... K, step-length " [k] E [0, 1] represents proportional step from æ; simplex vertex e
straightforward manner.
To ensure that each new state at lies in the simplex, we define its update using a K-dimensional "innovations" vector, 7/- As seen in Fig. 3, we imagine taking K different steps from the previous state 2;-1. Each step (indexed by k) moves toward vertex eş with proportional step length 7 E [0, 1]. zero-length step (% [k] 0) leaves the state at its previous value -;-1 while full step (7% [7.[k] 1) jumps to the vertex eg. Unlike prior methods (Nguyen and Volkov, 2020), we repeat this process for each of the K components and average their results to achieve the next state It.
d Ek Yo [e]k)) Y By construction, (7) delivers a valid state x2 that lies in the K -isipeeex.
Dt
Inspired by ideas from dynamical Bayesian nonparametric models (Ren et al., 2008), a suitable prior over innovations Yt on the domain (0, 1] is the Beta distribution (Yates and Goodman, 2005). We draw independent innovation values for each component (indexed by K) as IID across time according two-component Beta mixture. The first component (index 0) captures stationary behavior and the second component (index 1) captures transitions between pure states:
T
K
p ((r) = IIII w Beta (Y][4): [k]+), bo [+]) + (1 w (k)) Beta (Yr[kk]: a [k]],bb, [A]). (8)
2=1 1=1
This Beta-mixture prior for allows flexibility in how æ; evolves the simplex. By requiring that the Beta parameters of each component are larger than one (a[k) > 1, b[k] 1), we induce uni-modal distributions on [0. 1]. For the stationary component (index 0), expect innovations close zero, which means we should set [u[ >> ag [k]. We fix ag k) = 1.1, by kk 20 in all experiments. For the transition component (index 1) of each state k allow Beta parameters a [k), b, [k] as well as the mixture weight w[k] to be learnable hyperparameters, denoted H = {w, 01 b. To prevent
mode collapse we constrain [k] 1.1,
a. [ > 0.15, and [k] [0.01, 0.99).
a.[k]tb.18
As mentioned in Sec. 3, the simplex-state 2; represents the barycentric mixing weights used to compute the model-predicted distribution of the data. In our current formulation, this sequence of states is deterministic according to (7), given the initial state and the sequence of innovation vectors. Since these innovations are the random variables interest, is convenient to replace in (6) with
r wo; (7r)7Tm) resulting in the estimation problem, ř, H
argmin log [T)(T) (()) T.e.H
E' Wi Pn, (T,
((T.).H.((AE]))
In our implementation, we are indifferent to the initial state and therefore set p(ag) as uniform over the simplex. The coloration in (9) is linked to the associated parameters in Fig. 2.
5

---

4.2 Prior on Pure-State Emission Parameters
The final component to (9) is p()), the prior on the pure-state emission parameters. Using reference normal distribution (m,, Gol), we can define a probability density function over the space of all Gaussian distributions derived from the Wasserstein distance to this reference distribution,
p(m, S) (x 00) exp
W2 ((m, (mo; ofr))
(10)
2s²
=x(s, 0o) exp
2s² 11 9012
where is scalar hyperparameter that controls the variance of this prior. A simple calculation shows that (10) is equivalent to a multivariate Gaussian distribution over q = [m, wi E R2d, the joint space of means m and the eigenvalues w the covariance matrices S E Sym This Gaussian
mean Go (mo, 0o; ..., ool and covariance equal to sI. Since the eigenvalues of S must be positive, it follows that the normalizing constant needed for (10) to valid distribution given the normal CDF
function & is x(s, are mutually independent. To ensure that this prior scales similarly to the other terms in (9), all of log (p( (m/mm,, Sk)).
((2*r²) ① (==) We assume that the pure-state distributions parameters
which scale with the length of the time series T, we set log (p(O) = T - km
5
Model Estimation
Algorithm 1: Dynamical Wasserstein Barycenter (DWB) Time-Series Estimation
Input:
Output:
Vr, 1... T: Time series observations
R=1 } Pure-state emission params {mr S
K
K: Number of pure states
Hyperparameters:
r Loo (re) ( Initial state and innovations
n: Window size, 5: Window stride
Wasserstein barycentric state vector (Computed from I via (7)) H {w, a1 b Beta mixture parameters for transition dynamics
{ar)/=
X: Weight data-fit term
8: Variance on prior for +
(Po, oo): Mean, var. of p(0) reference dist.
Convergence threshold
for l 1,...? T where T (T (T- + do
Di=1 2n+11
m.
Preprocessing windowed S. 2n Li-1 ( - - empirical distributions
(2++1)
9
-2++1
6e N N(m,, Si)
end
(0)) F (r()), (0); H (pt))=1)
Cost function F defined in (9)
do
p(++1) (n+1) = argminr . F ((n)) (n)), (m),, ((t)==)):
// Adam
(n+1) arggmnng F (r(++)) (n), H 100)==): // Riemannian 1ine search dn+1)= = F (T(++1), (++1) H(++1), (pili=n)
while (clm) - c(x++1))= > n
Given desired number of states K and multivariate time series dataset y, Alg. details the steps needed to learn all parameters of our DWB model: T, the initial state and innovations sequence that drive the dynamical state model; O the pure-state emission distribution means and covariance matrices; and H. the hyperparameters governing transition dynamics on the simplex.
The algorithm performs coordinate descent (updating some variables while fixing others) to optimize the objective function (9). We chose this structure because the update to θ able to exploit spe- cialized optimization structure. Gradient descent methods are used to implement each minimization step Alg. taking advantage of auto-differentiation in Py PoTchh (Paszke et al., 2017). The runtime cost of each step in Alg. I is (TT)d)), where d is the dimension of cach observed data vector y:-

---

Updates to r. H via Adam. The Adam optimizer (Kingma and Ba, 2017) is used to solve the r, H problem on line of Alg. 1. To ensure that Y [0. 1] for t 1,..., T. we clamp these parameters to [e, for le-6, The initial state vector is clamped and normalized to stay on the simplex in a similar manner and the parameters of H are clamped as mentioned in Sec. 4.1.
Updates to è via natural Riemannian geometry. The pure-state emission parameters O define the mean and covariance parameters for K Gaussian distributions. While a variety of methods each based on different geometries have been proposed for optimizing Gaussian parameters (Lin, 2019; Hosseini and Sra, 2015; Arsigny et al., 2007), in this work we choose to leverage the geometry of Gaussian distributions under the Wasserstein distance (Malagó et al., 2018). From the decomposition in (2), we k=1 K under Wasserstein geometry can be carried out over a see that optimization for θ {m,, Sx)
Riemannian product manifold (I Sym (Hu al., 2020) with standard Euclidean geometry on R2, and Wasserstein-Burne geometry on Sym (Malagò et al., 2018; Takatsu, 2011; Bhatia et al., 2017). Therefore, estimate @ over this Riemannian product manifold using gradient descent line search algorithm (Absil et al., 2008). The supplement provides further details and experimental results demonstrating improved optimization speeds compared to standard Euclidean geometry.
Real World Results
6.1 Datasets and Evaluation Procedures
Datasets. Our work is motivated by applications in human activity accelerometry where "pure" states correspond atomic actions such as walking, running, or jumping. We evaluate our algorithm two datasets where smooth transitions between states are observable and the number of states is known.
Beep Test (BT, proprietary): 46 subjects run between two points to a metronome with increasing frequency. In this setting the subject alternates between running and standing thus we estimate a two state model. Data captured from a three-axis accelerometer sampled at 100 Hz.
BT
MSR
100
250
5
25
125
100
100
Microsoft Research Human Activity (MSR, Morris et al. (2014)): 126 subjects perform exercises in gym setting. Exercises vary among subjects covering strength, cardio, cross-fit and static exercises. Each time series truncated five minutes. We set K to the number of labeled discrete states in the truncated time series (range: to 7). The three-axis accelerometer is sampled at 50 Hz.
1.0
1.0
le-4
le-4
Table Model hy- perparameters
Available labels. All models trained in unsupervised fashion: each method is provided only the 3-axis accelerometer signal y and desired number of states K as input. While some ground-trute state annotations available, each timestep is labeled belonging exclusively to one discrete state. This assumes instantaneous transition between pure states and belies the underlying gradual transitions (e.g. acceleration from stand to run) that actually occur in the data stream, which method is designed for. Because annotations that properly characterize the gradual transition between states are not available, we evaluate performance based on how well a given model's predicted emission distribution fits the observed data over the whole time series.
Performance metrics: We measure data fit quality using both the average Wasserstein er- ror (11), akin to the model-fit term in (9), as well as the negative log likelihood (12) of all samples in each window given the model's inferred barycentric distribution for that window.
T 2
1 T Σ (or, (11) Enll
Σ
Σ
- log (pn, (15()1)+4))) (12)
ew
T(2n
To the best of our knowledge. the problem of characterizing continuous transitions among discrete pure states is largely unexplored. Most relevant are continuous state space models which identify continuous latent state, but not a manner that identifies pure-states of the system. Therefore, we compare our proposed DWB model continuous-state deep neural state space (DSS) model. Additionally, we baseline the Wasserstein barycentric interpolation model against the linear interpolation model given by discrete-state Gaussian mixture models (GMM).
Baseline methods:
GMM Linear interpolation baseline. Under the linear interpolation model, each timestep's emission distribution a Gaussian mixture of the pure states, PG, = Σ x, [k].N St). We highlight that PB, and PG, are equivalent when the x; is in a stre-state, thus the difference between models
7

---

Accelerometer
5 & 0 8.
Nan Exarato
Squit
Not-Exereh
Oherreen THEADS Externsoy
Nar-Extroaa
Strup
Y2
Estimated State
-211
-221
X-2
1231
2000
40330
60000
60000
10000
120000
Time (1)
Figure 4: (top left) Three-axis accelerometer of a sample MSR time series consisting of 5 actions. (right) Estimated pure-state Gaussian distributions projected onto each pairwise axes. (bot left) The Wasserstein barycentric weights are given by the estimated system state.
lies in the transition regions. The GMM model is implemented in our framework by replacing PB, with PG, in Eqs. (9), (11), and (12). To compute the Wasserstein distance between Gaussian and Gaussian mixture, we use the upper bound Chen et al. (2018) for fast gradient-based parameter estimation, but use Monte-Carke estimation (Sriperumbudur al., 2010) for more precise evaluation.
Deep neural State Space (DSS, (Krishnan et al., 2016)). The DSS method uses neural networks to parameterize the transition dynamics and the emission distribution of the latent state. Unlike our DWB approach, continuous state space models do not identify pure-states of the system. Thus, post-processing of the latent state space would be required to identify such pure states under these frameworks. For our DWB model, the maximum number of parameters required for state transitions and emissions for the MSR dataset is p 91 (O ()). Therefore, we evaluate the DSS model under two different settings, one where both the transition and emission networks are given a comparable number of parameters to the DWB model (p 94) and one with many more (p s 88, 000) parameters. Exact configuration details for DSS are provided in the supplement.
Hyperparameters and initialization. Hyperparameter choices are documented in Tab. 1. with window size chosen capture 2-5 seconds of real-time activity in each dataset. We set 1.0 and À 100 according to the parameter selection study later in Sec. 6.2. We set mg to the mean of the observed data, and do to the average eigenvalue of the set of covariance matrices obtained from a K K-component GMM fit to the data using et al. (2011). Our O estimation problem non-convex, so the solution is dependent on initialization. To ensure fair comparison, we use the same initialization for each method: a time-series clustering method (Cheng et al., 2020a) that applies matched-filten change point detection (Cheng et al., 2020b) for the Wasserstein distance. Additional details regarding optimization and initialization are provided in the supplement.
canattiiommiiiizaiionnn via code from Pedregosa
6.2 Experimental Results and Analysis
Qualitative evaluation. Fig. 4 shows for one exemplary MSR time series how our model can estimate stre-state emission distributions for the five states (right) and capture the system state in both the stationary and transition periods over time (bot left). Of interest are the segments of "Non-Exercise" that have significant contributions from "Pushup" (e.g. dashed lines at = 2.5k, 7.5k, 10.5k). The data shows these regions are distinct from the pure "Non-Exercise regions (e.g. solid lines at t = 6.5k, 11.5k): the data mean and variance appear to be intermediate values between "Pushup" and "Non-Exercise" pure states, showing the model's ability to identify gradual transitions. Results from other MSR time series are included in the supplement.
We further visualize the learned state vectors for both our model and the baseline over time for the BT data Fig. revealing the benefits of our approach for transition modeling. This dataset has two pure states (stand, run), and transition periods can be clearly seen where the subject accelerates and decelerates between each running segment (e.g. Fig. 5(a) = 750, 1300). The GMM interpolation model in Fig. 5(b) identifies the alternating discrete states, however all of the transition regions appear identical, switching between states almost instantly. Only our Wasserstein barycentric model in Fig. 5(d) captures the varying rates of acceleration and deceleration in the transition regions. This

---

Accelerometer Data
(b)
GMM Interpolation
84-
200
460
02
ea DWDB
10
1000
1500
2003
2003
1000
1500
3200
2400
(c)
Model
(d)
Wasserstein Barycenter
101
60-
10
100 ew (GMM)
1000
1900
2000
2000
1.300 1690
2000 2600
time (f)
time [e]
Figure 5: BT alternates between two discrete states (run and stand). (a) The transition between states are observable in the three-axis accelerometer.
Figure 6: Comparison for each MSR time series between the GMM and Wasserstein barycentric interpolation model (DWB) under the ew evaluation metric. Points below
(b) The state model learned by the GMM linear model is akin to binary switch in states whereas (c) the Wasserstein barycentric model closely tracks both grad- ual and quick state changes. (d) This difference is highlighted the discrepancy in W (Pt: PB.) the transition regions.
indicate better data for the Wasserstein model.
discrepancy between the models is highlighted in Fig. 5(c), where the improvement of the Wasserstein relative to the GMM model accentuated in the transition regions.
Quantitative comparison to GMM linear interpolation. Fig. shows how our Wasserstein barycentrie interpolation model improves data fit quality compared to the GMM linear interpo- lation model, as shown by the decrease in the ew metric across all the 126 MSR time series. Evaluation with respect to e,r (lower is better) shows similar improvement with average of 1.02 for our Wasserstein barycenter model versus 1.50 for the GMM model. Because the interpolated distributions pn, and pG, equivalent when the system exists in pure-state, the benefits of displacemenn-ineepplattion barycentric model come from improved fit during the transition periods.
Quantitative comparison to Deep State Space (DSS). As shown in Tab. 2. when given a similar number of pa- rameters, our DWB method outperforms the DSS model regardless the evaluation metric chosen. When given many more parameters, DSS still has worse ew but bet- ter eny scores. This difference can be attributed to the fact that the DSS objective is minimized with respect to enu whereas the DWB objective is minimized with respect to ew- These results suggest that in terms of characterizing a time series' underlying data distribution, our method is competitive with deep learning methods. Our DWB approach has the added benefit of learning the pure-states of the system, something that would require additional post-proet post-processing of the latent space for the DSS and other continuous state space models.
DWB
DSS
DSS
p= 91
2(94)
= 2 (88))
0.27
4.34
3.07
extt
1.02
1.49
-0.508
Table 2: Evaluation of DWB and DSS with 94 and 80k parameters on MSR dataset. DWB performs best according to ew, and better than DSS when comparable number of parameters are used under exil.
Ablation study of the dynamics prior. We also consider variation of our model using (non- mixture) single Beta distribution prior for 7. with parameters [[k] = 1.1, b[k] = 3.0, = 10, 2.0. single Beta for learning both stationary and transition dynamies, the model is more sluggish in adapting to fast changing states as seen in the supplement. Under this configuration, the MSR dataset has an average ew 0.50 compared the average ew 0.27 shown in Fig. 6 for the Beta-ri learnable prior. A sample plot is included in the supplement.
Because this
Riemannian optimization. The supplement provides experimental results demonstrating improved optimization speed for estimating θ using the Riemannian product manifold discussed in Sec. compared to using the standard Euclidean geometry.
Hyperparameter sensitivity. We explore the sensitivity of the results to variations in two key hyperparameters: the scalar À that controls the strength of the data term during learning and the pure state variance 8, whose inverse also plays the role of regularization parameter (9). From Tab. increasing À generally improves the model fit is expected from (9). For À Z 100, there is an

---

optimal value s = 1.0 for the MSR dataset which corresponds to "reasonable" pure state distributions seen in Fig. 7. For too small, the distributions are constrained the centroid of the data. For s too large, the pure state distributions become disjoint from the data themselves, a result allowed by the simplicial structure of the model. In this regime the the barycentric state vector moves away from the vertices towards the interior of the simplex causing more perceived uncertainty between states.
Sensitivity to initialization. The supplement includes a plot showing similar results to Fig. obtained when initializing & using ground truth activity labels, suggesting our chosen initialization unbiased.
7 Conclusions and Future Work
0.1
8 1.0
5 10
1000 100
Addressing recent trends in technology where the sampling rate of sensors can capture both stationary and transient behaviors of the system, we propose a dynamical Wasserstein barycen- tric model (DWB) to learn both pure-state emis- sion distributions and the time-varying state vector under a pettlaaemeee--nteroolloooo sition model between states in an unsupervised setting. For applications such as human ac- tivity recognition where transitions are often gradual, the displacement- inttecemenntntteoolatioonn given by the Wasserstein barycenter fits data more accurately than the mixture transition model commonly used in the literature. The proposed method can be applied to a wide range of time- series problems including segmentation, clus- tering, classification, and estimation.
tran-
1000 A=
Figure 7: (a) Estimated pure state distributions and (b) estimated state for MSR data varying X,
As further contributions, we provide dynami- cal state-evolution model of barycentric weight vectors over time. Inspired by previous work in state-space and Bayesian domains, this simplex random walk applies other temporal model- ing applications requiring simplex-valuee rep- resentations. We also show how tailoring the optimization geometry to the problem leads to improved convergence speeds.
0.01
0.1
1.0
100
0.1
596
601
581
395
434
1.0
597
593
455
389
378
10
572
533
284
203
214
521
362
139
157
173
1000
410
175
128
137
137
Table 3: Hyperparameter sensitivity results. Mean
ew for 25 MSR time series varying X, s. Reported results this paper use À 100, 8 = 1.0.
Limitations. Due to the need to estimate pure- state distributions in time-sensitive manner, our method is primarily useful for applications where the data is low-dimensional and densely sampled. We have assumed knowledge of the true number of states; in practice at least upper bound might be known but model size selection remains an open problem. Moreover, we have made strong assumption that all distributions are Gaussian primarily to leverage the associated geometry and simplify the barycenter computation and optimization.
Future Work. Because the Wasserstein barycenter in (3) is defined for any set of distributions with finite second moment (Peyré and Cuturi, 2019), the displlcemenn-interpolaaii
data model outlined in this work can in principle be extended to non-Gaussian distributions. Constructing tractable algorithms based on non non-parametric pure state models is certainly an interesting task. We also observe that the simplex random walk is amenable to natural extensions. For example, in our work. we assume that the transition parameters 7 are IID over time. However, by coupling these parameters, we can obtain higher-order smoothness in the simplex-state vector's trajectory.
Finally, the only barrier for making (9) a true likelihood is building probabilistic model for the model fit based on the Wasserstein distance to the observed data. This has proven difficult to implement as normalization factor (10) is dependent on the reference distribution. However, with this modification, we can use posterior analysis to properly assess uncertainty in the model and aid in setting the model parameters including the total number of states K, which currently is assumed to be known a-priori.
10

---

8 Acknowledgements
This research was sponsored by the U.S. Army DEVCOM Soldier Center, and was accomplished under Cooperative Agreement Number 1911QY-19-2-006 this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Army DEVCOM Soldier Center, or the U.S. Government. The U. S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.
The views and conclusions contained in
We also acknowledge support from the U.S. National Science Foundation under award HDR-1934553 for the Tufts T-TRIPODS Institute. Shuchin Aeron supported in part by NSF CCF: 155:055,,75, NSF RAISE 1931978, NSF ERC planning 1937057, and AFOSR FA9550- 18- 1-065. Michael C. Hughes supported in part by NSF IIS-1908617. Eric Miller supported in part by NSF grants 1934553,
1935555, 1931978, and 1937057.
References
P.-.. Absil, R. Mahony, and R. Sepulchre. 2008. Optimization algorithms on matrix manifolds. Princeton University Press, Princeton, N.J.; Woodstock. OCLC: ocn174129993.
Saced Aghabozorgi, Ali Seyed Shirkhorshidi, and Tch Ying Wah. 2015. Time-series clustering A decade
review. Information Systems 53 (Oct. 2015), 16-38. https: //doi. org/10. 1016/j j.1i6/j.ie i.......0.. 2015. .0.007
Martial Agueh and Guillaume Carlier. 2011. Barycenters the Wasserstein Space. SIAM Journal on Mathemat- ical Analysis 43,2 2 (Jan. 2011), 904-924. httpss://oi.. http:://doo.ooog11/// 10. 11377100805774
Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache. 2007. Geometric Means a Novel Vector Space Structure on Symmetric Positive-Deefiniee Matrices. SIAM J. Matrix Anal. Appl. 29, 1 (Jan. 2007), 928-347. btttps://di... www//o.//////... 1137/050637996
Rajendra Bhatia, Tanvi Jain, and Yongdo Lim, 2017. On the Bures distance between positive definite matrices. 172:0:171 [math] (Dee. 2017). http://aaxxdv.org/bbs//Tr 01504 arXiv: 1712.01504.
H. Bi, M. Perello-Nieto, R. Santos-Rodriguez and P. Flach. 2021. Human Activity Recognition Based on Dynamic Active Learning. IEEE Journal of Biomedical and Health Informatics 25, 4 (April 2021), 922-934. httpp:// /tt:://doo /oi. org/10. wtrg/0..119/JBII.. 202. .3034003 Conference Name: IEEE Journal of Biomedical and Health Informatics.
Yoosoon Chang, Robert K. Kaufmann, Chang Sik Kim, J. Isaac Miller, Joon Y. Park, and Sungkeun Park. 2020. Evaluating trends in time series of distributions: A spatial fingerprint of human effects climate. Journal of Econometrics 214, (Jan. 2020), 274-294.
1016/j - 2019.00 05.0010..4
Yoosoon Chang, Chang Sik Kim, and Joon Y. Park. 2016. Nonstationnaiity in time series of state densities. Journal of Econometrics 192, (May 2016), 152-167.
1016/j j jeconom.mm....... 2015. 06. 025
Yongxin Chen, Tryphon T. Georgion, and Allen Tannenbaum. 2018. Optimal transport for Gaussian mix- ture models. 171.:77710 [cs, math] (Jan, 2018). http:: 1710.07876.
http:///ardv.oog//aaa///ta orgg//s///////a///o 00876
arXiv:
Kevin C. Cheng, Shuchin Aeron, Michael C. Hughes, Erika Hussey, and Eric L. Miller. 2020a. Optimal Transport Based Change Point Detection and Time Series Segment Clustering, arXiv:1911.01325 (cs, eess] (Feb. 2020). http: http://arxiv.com abs /9911. 01325 arXiv: 1911.01325.
Kevin C. Cheng, Fric I.. Miller, Michael C. Hughes, and Shuchin Aeron. 2020h. On Matched Filtering for Statistical Change Point Detection. IEEE Open Journal of Signal Processing 1 (2020), 159-176. https:
1109/0JSP. 2020 3035070 Conference Name: IEEE Open Journal of Signal Processing.
Zoubin Ghahramani and Geoffrey Hinton. 2000. Variational Learning for Switching Space Models. Neural Computation 12, 4 (2000), 831-864. https://doi.orge http://....rr//11.1.1/////////// 1162/00896660000000111111
Reshad Hosseini and Suvrit Sra. 2015. Manifold Optimization for Gaussian Mixture Models, arXiv:1506.0777 arXiv: 1506.07677.
[cs, math, stat] (June 2015). http:/ //tp://///xiv.//// wwwpxii..orgg/bbeeee
07677
Jiang Hu, Xin Liu, Zai-Wen Wen, and Ya- Xiang Yuan. 2020. A Brief Introduction to Manifold Optimization. Journal of the Operations Research Society of China 8, 2 (June 2020), 199-248. httpss///oi. 00./10. 1007 9400//40005-o0 00295-9

---

R. E. Kalman. 1960. A New Approach to Linear Filtering and Prediction Problems. Journal of Busic Engineering 82, (March 1960), 35-45. htttpa///oii. http:///ooo.org/../.. 116521.6666625
Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization, arXiv:1412.69800 [cs]
2017). http://arxiv. org/ /412. 6980
arXiv: 1412.6980.
Rahul G. Krishnan, Uri Shalit, and David Sontag. 2016. Structured Inference Networks for Nonlinear State
Space Models. urXiv:1609.09860 [cx, stat] (Dec. 2016). http://arxivy wttp://ww...//////////////// 09669 arXiv: 1609.09869
Zhenhua Lin. 2019. Riemannian Geometry of Symmetric Positive Definite Matrices via Cholesky Decompo- sition. arXiv: [math, stat] (Aug. 2019). htttp:://do.orgg/// 1137/19M11221084 arXiv: 1908.09326.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. 2017. Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems. In Proceedings of the 20th International Conference on Artifical Intelligence and Statisticy. 914-922. https://proceedings mlr. wwwpppews///eeeppppppppp Trtess//v5//inndemaaaat inderman html
Luigi Malagö, Luigi Montrucchio, and Giovanni Pistone. 2018. Wasserstein Riemannian Geometry of Positive Definite Matrices. arXiv:1801.0922 [math, star] (Sept. 2018). http://arxiv. wwwpxiii.org//bbbe aarggabb/8801.... 09269 arXiv: 1801.09269.
Robert J. McCann. 1997. A Convexity Principle for Interacting Gases. Advances in Mathematics 128, (June 1997), 153-179. https://doi.co org/ 1006/ aama. 1997. 1634
Dan Morris, T. Scott Saponas, Andrew Guillory, and Ilya Kelner. 3014. RecoFit: using wearable sensor to find, recognize, and count repetitive exercises. In Proceedings the SIGCHI Conference Human Factors in Computing Systems. ACM, Toronto Ontario Canada, 3225-3234. nttps://doi.or 1145/2556288. 2557116
Goshu Nagino and Makoto Shozakai. 2006. Distance Measure Between Gaussian Distributions for Discriminat- ing Speaking Styles. (2006), 4.
Tuan-Minh Nguyen and Stanislav Volkov. 2020, On class of random walks in simplexes. Journal of Applied Probability 57, (June 2020). 409-428. attps:///olt org/10. 1017/ 2020. 19 arXiv: 1709.00174.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch, (2017), 4.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn Machine Leamiing Python. Journal of Machine Learning Research 12, 85 (2011), 2825 2830. http://jmlr.org/ papers/v12. http://////epp//e/e//ee egwsseeao.ll....am html
Gabriel Peyré and Marco Cuturi. 2019. Computational Optimal Transport: With Applications to Data Science. Foundations and Trends@ in Machine Learning 11, 5-6 (Feb. 2019), 355-601 607. https://doi. org/10 100 1561/2200000073 Publisher: Now Publishers, Inc.
L. R. Rabiner. 1989 A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, (Feb. 1989), 257-286. https://do1. of IEFF.
1109/5. 18626 Conference Name: Proceedings
Lu Ren, David B. Dunson, and Lawrence Carin. 2008. The dynamic hicrarchical Dirichlet process. In Proceedings of the 25th international conference on Machine learning ICML '08. ACM Press, Helsinki, Finland, 824 831. https://doi. org/10. 1tt///111115/1191111111 1390260
Walter Rudin. 1976. Principles of mathematical analysis (3d ed ed.). McGraw-Hil New York.
Bharath K. Sriperumbudur, Arthur Gretton, Kenji Fukumizu. Bernhard Schölkopf, and Gert R. G. Lanckriet. 2010. Hilbert Space Embeddings and Metries on Probability Mcasures. Journal of Machine Learning Research 11, 50 (2010), 1517-1561. http: //jmx.. org/
10a. html
Asuka Takatsu. 2011. Wasserstein geometry of Gaussian measures. Osaka Journal of Mathematics 48, 4 (Dec. 101 1026-1026. https://doi. org/10. 18910/4973 University, Departments of Mathematics.
Publisher: Osaka University and Osaka City
12

---

Cédric Villani. 2009. Optimal transport: old and new. Number 338 in Grundlehren der mathematischen Wissenschaften. Springer, Berlin. OCLC: 0en244421231.
Greg Welch. 1997. An Introduction to the Kalman Filter. (1997), 16.
Roy D. Yates and J. Goodman. 2005. Probability and Stochastic Processes; a Friendly Introduction, for Electrical and Computer Engineers (2nd ed ed.). John Wiley & Sons, Hoboken, NJ.
Pedro C. Álvarez Esteban, E. del Barrio, J. A. Cuesta-Albertos, and C. Matrán. 2016. A fixed-point approach barycenters Wasserstein space. arXiv:1511.05355 [math, statj (April 2016). http://arxiv. orwpziiii..r///aaee abs/ 1511. 05355 arXiv: 151 1.05355.
13


Reviews:
[{"review_id": "T2Z87hj8DHP", "reviewer": "Reviewer_r2KJ", "summary": "To study systems with pure-states and transition periods between pure-states, the authors propose a dynamical model which uses multivariate normal distributions to represent pure-states and characterizes transitions between states via the displacement-interpolation  specified by the Wasserstein barycenter. Learning parameters of pure-state distributions and simplex-valued barycentric weight vector evolving over time or dynamics of the system are performed in a joint way with the help of ADAM and Riemannian line search. Experimental results on human activity datasets are promising.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. Employing Wasserstein barycenter for state transition and interpolation sounds a very interesting idea.\n2. System dynamics can be characterized by simplex-valued barycentric weight vector evolving over time and learned from data. Such kind of modeling techniques might be applicable to other problems.\n\nCons\n1. The authors employ Beta distribution to model prior over innovations \\gamma_t or random factors in a dynamic system. I am wondering whether such kind of prior has enough descriptive power to model complex time series in reality with characteristics like non-stationary, heavy tail distribution, etc. In addition, distributions of pure-states are modeled using Gaussian distribution which has nice analytical properties and close-form solution for the Wasserstein barycenter computation. However, in practice it may not be optimal solution for state space modeling due to its limited discriminative power regarding complicated data distribution. \n2. Algo. 1 is a block-wise coordinate descent algorithm in essence. Convergence analysis is  expected which may help readers to understand how challenge the problem is and to what extent solutions found by the algorithm is optimal. \n3. Fig. 1 is not intuitive. From it I am not sure why the Wasserstein barycenter is superior to linear interpolation.\n4. In the experiments, GMM is employed as a baseline method which is not sufficient. Are there any other state-of-the-art methods in literature which could be used to illustrate the advantages of the proposed method?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GJdGY1vC05y", "reviewer": "Reviewer_nnyz", "summary": "This paper deals with the problem of modelling continuous-space time series as sequences of high-level hidden states.\nIn such models, transitions are usually represented as multinomial mixtures over hidden states.\nIn this work, another approach is considered in which transition distributions are modelled as Wasserstein barycenters between pure distributions attached to hidden states.\nIn the case of Gaussian distributions, such barycenters are also Gaussians, and when the match between empirical distributions and those model-predicted distributions is assessed using Wasserstein distance, closed-form formulations can be used.\nFrom those observations, a loss is derived that can be optimized by alternating between (i) gradient descent on the \"innovation\" parameters $H$ and $\\Gamma$ and (ii) gradient descent on a Riemannian manifold.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well motivated, the introduction is insightful and Figure 1 is a very good illustration of the benefits of the method.\nThe main idea (using Wasserstein barycenters in place of Euclidean ones to better model transitions between states) makes sense and the experiments conducted tend to validate this idea.\nThough the paper clearly illustrates the benefits of this approach for human activity recognition, experiments on other kinds of data would have been a plus to better motivate the significance of the contribution outside this specific application setting.\n\nIn the proposed approach, one needs to compute the Wasserstein distance between an empirical Gaussian distribution at time $t$ and the Wasserstein barycenter at the same timestamp.\nSince only a single sample is observed at time $t$, the parameters of the empirical Gaussian distribution are estimated using a window centered at $t$.\nI have two questions related to this:\n* How critical is the choice of the window size $n$?\n* Why rely on this window centered in $t$ instead of evaluating the likelihood of the observation $y_t$ under the distribution $\\rho_{B_t}$?\n\nBelow are some minor remarks:\n* In Fig 1 (d): the indexes of the Gaussians don't seem to match those of the estimated states\n* The Figure provided in Appendix that compares single-component and two-component Beta distributions for $\\Gamma_t$ is interesting and would deserve a place in the paper\n* It is a pity that Figure 2 does not present prior-related parameters stored in $H$\n\n**EDIT: I have read author rebuttal and it confirms my positive opinion on this paper (and answers my questions related to the use of a sliding window).**", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have included a section about the limitations of their approaches, which is sound.\nThey have not discussed potential negative societal impact of their work which makes sense since, even if their method can be used for human activity recognition, their contribution does not seem to introduce new societal threats compared to the existing literature in the field.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["Research Integrity Issues (e.g., plagiarism)"], "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "7rXk5kpDiLV", "reviewer": "Reviewer_p5FH", "summary": "\nSummary\n=======\n\nContext:\n--------\nThis paper proposes a reconstruction model for time series that are modeled using:\n- a sequence of K distributions (\\rho_1, ..., \\rho_K) each representing a certain \"state\"\n- a transition vector of weights (x_t)_t \\in Simplex_K that weight the different states at each\ntime point.\n\nStarting from observed vector valued data y_1, ..., y_T, the goal is to identify the state distributions \\rho_k\nand the transition weights (x_t).\n\nContribution\n------------\nThis paper proposes to do so by considering Gaussian distributions (\\rho) and modeling each observation \ny_t as a sample of the Wasserstein barycenter of (y_1, ..., y_K) weighted by (x_t[0], ... x_t[K]). \n\nThe transition states x_t are modeled using a Markov chain with a Beta distribution prior on its parameters (Gamma).\nAll parameters (Gamma, means and covariances of \\rho) are estimated via fitting the data (with the Wasserstein barycenter\nloss) and the prior on the parameters.\n\nThe proposed model is compared to a GMM (i.e replacing OT with a linear interpolation) on two experiments illustrating\nthe flexibility of using OT in estimating the transient state weight vector x. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "review\n======\n\nGeneral assessment and main concerns\n-----------------------------------\n\nOverall, the paper is well presented and very easy to read. The illustrated examples figs 1-3 play helped a lot.\nThe proposed idea is simple and intuitive: if samples between states are more likely to smoothly transition from one distribution\nto another than simply \"disappear / appear\" then an OT interpolation makes more sense than a linear one. While the proposed model\nas well as its implementation are convincing, my main concern is that the experiments as well as the introduction is a bit too \nrestricted / niche. \n\n(A) I would have like to know for instance the nature of the data for which such dynamic models apply ? why would\na Gaussian assumption be realistic ? Are there any scenarios where a linear interpolation would be more suited than OT ? \n\n(B) All experiments used do not have ground truth. While this is sometimes inevitable, the performance of a new model should always be assessed \nin controlled experiments. The authors resort to using a data fitting term to compare the OT interpolation against the linear one which is biased\nsince it is literally minimized in their proposed loss function. While the presented evidence is interesting and encouraging, it is inconclusive\nat best. These concerns as well as (A) could have been adressed using simulations where the strengths / limits of the proposed model could be \npinpointed. \n\nSpecific comments\n-----------------\n\n1) Equations (6) and (9) display the W_2 distance. However I believe it should be the Bures metric instead ? Moreover, the empirical measures\n\\rho_t should be replaced by their empirical moments since the model doesn't exactly fit the empirical observations but their moments instead.\nThis is even more confusing since it is the same as the fit error (11) which a priori should be using the exact empirical data.\n\n2) The Gaussian approximation is a very strong assumption (perhaps necessary if d and n are large) and OT decomposes as a Euclidean distance between\nthe means and Bures between the covariances. It would have been interesting to perform comparisons where the covariance term is replaced by another\nmetric over the cone of PD matrices or even entirely removed. Perhaps discriminating using the means would be enough ? \n\n3) Using (11) as an evaluation metric introduces a systemic bias here since it is quite literally minimized in the proposed method:\nit would naturally be lower than any other benchmark.\n\n4) For initialization, why not use the output of the GMM fit entirely and use one mean m_0  and one sigma_0 ?\n\n5) If you use such an initialization where you have initial estimates of all the means and covariances, another benchmark\nwould be to estimate the weights x_k directly by projecting on the simplex.\n\n6) Were the specified values in L184-L185 manually data driven ?\n\nMinor remarks and typos:\n---------------------\n\n- L44 unfinished sentence \n- L145 something strange here in this sentence", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "4OUhmjxDYgT", "reviewer": "Reviewer_Vw6y", "summary": "The authors present a model for the state of a system over time as a Wasserstein barycenter as a mixture between a (known) number of pure states modeled as Gaussian distributions. Modeling states as Gaussian distributions results in more efficient computation given the closed form of Wasserstein barycenters between Gaussian distributions.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This model is empirically validated by comparing to a GMM model over two time series datasets with state annotations using the a Wasserstein distance error between predict and ground truth distributions.\n\nQuestion: Why should the Beta distribution be unimodal? I would think that there are many instances when it should not be unimodal, and could more often have a number of modes equal to the number of pure states. \n\nThis is an interesting idea to improve over gaussian mixture models in some time series. However, this manuscript does not relate to any of the work on time series in the ML conference cycle and is not compared to any recent methods, and further is only applied to a very specific time series. It would be interesting to see how this model performs against some of the more recent deep learning models and on a wider variety of datasets. Without this it is difficult to evaluate where this model fits into the broader literature.\n\nN-BEATS: https://arxiv.org/abs/1905.10437\nM4 Competition:https://www.sciencedirect.com/science/article/pii/S0169207019301128 \n-----------------\nPost author response update:\nThank you for clearing up these issues and the problem setting you are tackling in this paper. I now understand that this is different from the traditional time series interpolation and extrapolation problem. Since a large number of the results use error from known states as ground truth this seems quite similar to me. I agree that it would be a good idea to compare to deep state-space models as mentioned by the authors and more recent work in this line.\n\nI am updating my score to weak reject as I still believe it would be useful to have comparison to other related work on deep state space modeling and further comparison to DeepSS (although preliminary results provided by the authors looks promising).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes dynamical Wasserstein barycenter for modeling time series, observation are considered to be sampled from Bures Wasserstein barycenter of states with time dependent mixing coefficients.  The paper compares wasserstein interpolation of states to linear interpolation of states. 

Reviewers were positive about the paper in the sense it demonstrated an advantage on a GMM model with linear interpolation. Several question were raised on why restricting to Bures barycenter and to go beyond gaussianity assumption and on using maybe of other baselines such as neural networks and in providing more experiments on more challenging benchmarks. 

I think the paper has a value being a new application of Wasserstein barycenter to time series modeling , and the authors were transparent about the limitations of the work. Weak accept 
Decision: Accept (Poster)