Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: 
Summary
=======

Context:
--------
This paper proposes a reconstruction model for time series that are modeled using:
- a sequence of K distributions (\rho_1, ..., \rho_K) each representing a certain "state"
- a transition vector of weights (x_t)_t \in Simplex_K that weight the different states at each
time point.

Starting from observed vector valued data y_1, ..., y_T, the goal is to identify the state distributions \rho_k
and the transition weights (x_t).

Contribution
------------
This paper proposes to do so by considering Gaussian distributions (\rho) and modeling each observation 
y_t as a sample of the Wasserstein barycenter of (y_1, ..., y_K) weighted by (x_t[0], ... x_t[K]). 

The transition states x_t are modeled using a Markov chain with a Beta distribution prior on its parameters (Gamma).
All parameters (Gamma, means and covariances of \rho) are estimated via fitting the data (with the Wasserstein barycenter
loss) and the prior on the parameters.

The proposed model is compared to a GMM (i.e replacing OT with a linear interpolation) on two experiments illustrating
the flexibility of using OT in estimating the transient state weight vector x. 

main_review: review
======

General assessment and main concerns
-----------------------------------

Overall, the paper is well presented and very easy to read. The illustrated examples figs 1-3 play helped a lot.
The proposed idea is simple and intuitive: if samples between states are more likely to smoothly transition from one distribution
to another than simply "disappear / appear" then an OT interpolation makes more sense than a linear one. While the proposed model
as well as its implementation are convincing, my main concern is that the experiments as well as the introduction is a bit too 
restricted / niche. 

(A) I would have like to know for instance the nature of the data for which such dynamic models apply ? why would
a Gaussian assumption be realistic ? Are there any scenarios where a linear interpolation would be more suited than OT ? 

(B) All experiments used do not have ground truth. While this is sometimes inevitable, the performance of a new model should always be assessed 
in controlled experiments. The authors resort to using a data fitting term to compare the OT interpolation against the linear one which is biased
since it is literally minimized in their proposed loss function. While the presented evidence is interesting and encouraging, it is inconclusive
at best. These concerns as well as (A) could have been adressed using simulations where the strengths / limits of the proposed model could be 
pinpointed. 

Specific comments
-----------------

1) Equations (6) and (9) display the W_2 distance. However I believe it should be the Bures metric instead ? Moreover, the empirical measures
\rho_t should be replaced by their empirical moments since the model doesn't exactly fit the empirical observations but their moments instead.
This is even more confusing since it is the same as the fit error (11) which a priori should be using the exact empirical data.

2) The Gaussian approximation is a very strong assumption (perhaps necessary if d and n are large) and OT decomposes as a Euclidean distance between
the means and Bures between the covariances. It would have been interesting to perform comparisons where the covariance term is replaced by another
metric over the cone of PD matrices or even entirely removed. Perhaps discriminating using the means would be enough ? 

3) Using (11) as an evaluation metric introduces a systemic bias here since it is quite literally minimized in the proposed method:
it would naturally be lower than any other benchmark.

4) For initialization, why not use the output of the GMM fit entirely and use one mean m_0  and one sigma_0 ?

5) If you use such an initialization where you have initial estimates of all the means and covariances, another benchmark
would be to estimate the weights x_k directly by projecting on the simplex.

6) Were the specified values in L184-L185 manually data driven ?

Minor remarks and typos:
---------------------

- L44 unfinished sentence 
- L145 something strange here in this sentence
limitations_and_societal_impact: None
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors present a model for the state of a system over time as a Wasserstein barycenter as a mixture between a (known) number of pure states modeled as Gaussian distributions. Modeling states as Gaussian distributions results in more efficient computation given the closed form of Wasserstein barycenters between Gaussian distributions.
main_review: This model is empirically validated by comparing to a GMM model over two time series datasets with state annotations using the a Wasserstein distance error between predict and ground truth distributions.

Question: Why should the Beta distribution be unimodal? I would think that there are many instances when it should not be unimodal, and could more often have a number of modes equal to the number of pure states. 

This is an interesting idea to improve over gaussian mixture models in some time series. However, this manuscript does not relate to any of the work on time series in the ML conference cycle and is not compared to any recent methods, and further is only applied to a very specific time series. It would be interesting to see how this model performs against some of the more recent deep learning models and on a wider variety of datasets. Without this it is difficult to evaluate where this model fits into the broader literature.

N-BEATS: https://arxiv.org/abs/1905.10437
M4 Competition:https://www.sciencedirect.com/science/article/pii/S0169207019301128 
-----------------
Post author response update:
Thank you for clearing up these issues and the problem setting you are tackling in this paper. I now understand that this is different from the traditional time series interpolation and extrapolation problem. Since a large number of the results use error from known states as ground truth this seems quite similar to me. I agree that it would be a good idea to compare to deep state-space models as mentioned by the authors and more recent work in this line.

I am updating my score to weak reject as I still believe it would be useful to have comparison to other related work on deep state space modeling and further comparison to DeepSS (although preliminary results provided by the authors looks promising).
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: To study systems with pure-states and transition periods between pure-states, the authors propose a dynamical model which uses multivariate normal distributions to represent pure-states and characterizes transitions between states via the displacement-interpolation  specified by the Wasserstein barycenter. Learning parameters of pure-state distributions and simplex-valued barycentric weight vector evolving over time or dynamics of the system are performed in a joint way with the help of ADAM and Riemannian line search. Experimental results on human activity datasets are promising.

main_review: Pros:
1. Employing Wasserstein barycenter for state transition and interpolation sounds a very interesting idea.
2. System dynamics can be characterized by simplex-valued barycentric weight vector evolving over time and learned from data. Such kind of modeling techniques might be applicable to other problems.

Cons
1. The authors employ Beta distribution to model prior over innovations \gamma_t or random factors in a dynamic system. I am wondering whether such kind of prior has enough descriptive power to model complex time series in reality with characteristics like non-stationary, heavy tail distribution, etc. In addition, distributions of pure-states are modeled using Gaussian distribution which has nice analytical properties and close-form solution for the Wasserstein barycenter computation. However, in practice it may not be optimal solution for state space modeling due to its limited discriminative power regarding complicated data distribution. 
2. Algo. 1 is a block-wise coordinate descent algorithm in essence. Convergence analysis is  expected which may help readers to understand how challenge the problem is and to what extent solutions found by the algorithm is optimal. 
3. Fig. 1 is not intuitive. From it I am not sure why the Wasserstein barycenter is superior to linear interpolation.
4. In the experiments, GMM is employed as a baseline method which is not sufficient. Are there any other state-of-the-art methods in literature which could be used to illustrate the advantages of the proposed method?


limitations_and_societal_impact: No
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper deals with the problem of modelling continuous-space time series as sequences of high-level hidden states.
In such models, transitions are usually represented as multinomial mixtures over hidden states.
In this work, another approach is considered in which transition distributions are modelled as Wasserstein barycenters between pure distributions attached to hidden states.
In the case of Gaussian distributions, such barycenters are also Gaussians, and when the match between empirical distributions and those model-predicted distributions is assessed using Wasserstein distance, closed-form formulations can be used.
From those observations, a loss is derived that can be optimized by alternating between (i) gradient descent on the "innovation" parameters $H$ and $\Gamma$ and (ii) gradient descent on a Riemannian manifold.
main_review: The paper is well motivated, the introduction is insightful and Figure 1 is a very good illustration of the benefits of the method.
The main idea (using Wasserstein barycenters in place of Euclidean ones to better model transitions between states) makes sense and the experiments conducted tend to validate this idea.
Though the paper clearly illustrates the benefits of this approach for human activity recognition, experiments on other kinds of data would have been a plus to better motivate the significance of the contribution outside this specific application setting.

In the proposed approach, one needs to compute the Wasserstein distance between an empirical Gaussian distribution at time $t$ and the Wasserstein barycenter at the same timestamp.
Since only a single sample is observed at time $t$, the parameters of the empirical Gaussian distribution are estimated using a window centered at $t$.
I have two questions related to this:
* How critical is the choice of the window size $n$?
* Why rely on this window centered in $t$ instead of evaluating the likelihood of the observation $y_t$ under the distribution $\rho_{B_t}$?

Below are some minor remarks:
* In Fig 1 (d): the indexes of the Gaussians don't seem to match those of the estimated states
* The Figure provided in Appendix that compares single-component and two-component Beta distributions for $\Gamma_t$ is interesting and would deserve a place in the paper
* It is a pity that Figure 2 does not present prior-related parameters stored in $H$

**EDIT: I have read author rebuttal and it confirms my positive opinion on this paper (and answers my questions related to the use of a sliding window).**
limitations_and_societal_impact: The authors have included a section about the limitations of their approaches, which is sound.
They have not discussed potential negative societal impact of their work which makes sense since, even if their method can be used for human activity recognition, their contribution does not seem to introduce new societal threats compared to the existing literature in the field.
review_title: null

Meta-review: The paper proposes dynamical Wasserstein barycenter for modeling time series, observation are considered to be sampled from Bures Wasserstein barycenter of states with time dependent mixing coefficients.  The paper compares wasserstein interpolation of states to linear interpolation of states. 

Reviewers were positive about the paper in the sense it demonstrated an advantage on a GMM model with linear interpolation. Several question were raised on why restricting to Bures barycenter and to go beyond gaussianity assumption and on using maybe of other baselines such as neural networks and in providing more experiments on more challenging benchmarks. 

I think the paper has a value being a new application of Wasserstein barycenter to time series modeling , and the authors were transparent about the limitations of the work. Weak accept 
Decision: Accept (Poster)