Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper aims to develop a method for trajectory prediction/ interpolation by learning trajectory
representations in a latent space. The paper uses a transformer encoder to learn parameters of
a distribution, and decoder to reconstruct the original trajectory point from an embedding in the
latent space. The paper uses the conventional triplet loss to force similarity between
embeddings from the same trajectory segment, and a reconstruction loss for the decoder.
strengths_and_weaknesses: Strengths
- The proposed method is sufficiently novel, with most previous works focusing on autoregressive predictions or the latent space representations not modeling future representations adequately.
- The paper is generally well written, with the qualitative examples demonstrating the network’s ability to interpolate/extrapolate in diverse contexts.
- The method allows sampling from a distribution, which comes with its benefitsirregularly spaced sampling, flexibility on the prior for 
 distributions, generalizing to unobserved latent spaces.
- The method performs better than the existing state of the art method of Trajectron++ on the human pose prediction task.

Weaknesses
- The paper lacks additional ablation studies, such as:
    - How does the network performance vary with the number of samples, M?
    - How does the choice of the encoding distribution impact network performance?
    - How sensitive is the method to irregularly sampling from the distribution and how does it impact the final decoded trajectory?
- It would be interesting to understand how non-compliant trajectories (irregular actions such as hand waving for a few timesteps) impact the learned latent space, both for interpolation and extrapolation. Additionally if the authors could comment on repetitive trajectories (actions that repeat periodically), that would be interesting.
- The paper evaluates the method on the single task of joint-space trajectory domain- it would be interesting to see how this translates to other domains for trajectory prediction eg. autonomous driving, crowd prediction.
- The paper mentions the lack of context aggregation in the method, which is understandable for simplicity reasons. However, to generalize outside of the joint-space prediction problem, context aggregation would be vital.
questions: Questions have been asked along with the weaknesses mentioned above.
limitations: Yes, in Appendix A.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a spatial trajectory representation framework. The framework represents a partial observation of a trajectory as a probability distribution in a learned latent space. With this representation, the framework is able to perform a variety of different tasks, with examples including 1) future and past prediction, 2) continuous reconstruction given a discrete input, 3) interpolation between two segments, 4) comparison of different trajectories, and 5) modifying existing trajectories. The framework consists of an encoder and a decoder. The encoder is trained with self-supervised triplet loss. The decoder is trained with regression loss on ground-truth trajectories. The authors evaluated the method on public human movement datasets and compared its performance against VRNN and Trajectron++. The result shows that it achieves significantly better performance than the baselines.
strengths_and_weaknesses: --- Strengths

- The paper is well written and easy to follow.

- The framework is able to perform a variety of different tasks without re-training.

- The proposed method was evaluated on public datasets and achieved significantly better performance than the public baselines.


--- Issues and suggestions

- The baselines used in this work are a bit old. VRNN is from 2015, and Trajectron++ is from 2020.
questions: --- Other comments

- Line 192: Typo "Transformer++".

limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a new representation learning method for spatial trajectories, named by TrajRep. TrajRep is in an encoder-decoder structure and optimized based on the reconstruction loss, as well as a self-supervised triplet loss to refine the learned representations. For the self-supervised loss, the authors define the positive and negative loss by analyzing the structure of trajectories in detail and define the distance function based on the box embeddings under the conditional scenario. TrajRep is evaluated on the human movement dataset.
strengths_and_weaknesses: ### Strengths

1. In general, I think this paper is interesting. Concretely, I think the novelty of this paper mainly lies in two aspects:
- The whole pipeline for trajectory modeling, namely especially Equation (2) that decodes the raw trajectory from z and t.
- The distribution formalization method in sections 2.4 and 2.5.

2. The qualitative Experiments are impressive, especially the Figure 5.

3. The related work is well researched.

### Weaknesses
1. I do not think this paper is well-organized. Due to the lack of details, it is difficult to clear up the method design from the main text. Too many details are hidden in the supplementary materials, which affects the reading of the main text.

2. Some sentences of the paper are not well-supported.

3. The experiments are insufficient, including the aspects of comparing baselines, ablations studies, analysis of hyperparameter sensitivity and training curves.

See the next section for more details of the weaknesses.
questions: 1. Paper organization.
- Figure 6 in the supplementary materials should be placed in the main text. Section 2.3 of the original paper does not clarify the authors' design well.
- The formalization of box embedding (Equation (11)-(12)) should be mentioned in section 2.5 or related works. In the original version, it is hard to know the formalization of Vol in line 141.

2. Some sentences of the paper are not well-supported.
- Line 165: the authors claim that “our method is generally designed for any kind of spatial trajectory”, but the experiments only include the human movement dataset. More datasets are expected.
- Line 148-line 150: the authors present two types of the distance function D and choose the box-embedding way without experiments of comparison.
- Figure 6. The authors do not explain the reason why the relationship between two segments is labeled as HP or SP or others.

3. Insufficient experiments.
- Comparing baselines. This paper only includes two baselines. More baselines are expected, e.g. some GNN-based models, which are widely used in skeleton prediction.
- Ablations studies. The design space of the paper is large. More ablations are expected, including the definition of D (symmetric distance or conditional), w/ or w/o triple loss, and the sampling strategy mentioned in line 530.
- Hyperparameter sensitivity. Why choose $\alpha$ as 1 (line 541)? Does the training process and model performance is sensitive to this hyperparameter?
- Training curves. Since the extreme point may affect the triple loss, I would like to see if the training process is steadily convergent during training.

Considering the above questions, I think this paper needs much more effort in presentation and experiments.

=====================
In the rebuttal, the author addressed the above questions well. I would like to raise my score.


limitations: Yes, the authors have discussed the limitations and potential negative social impact of this paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a framework for learning representations for trajectory data from partial observations. The learnt representation is in the form of a probability distribution such that it captures the uncertainty of the unobserved segments. This distribution can be used for sampling unobserved segments of trajectories (interpolation, extrapolation, continuous time sampling), compare different trajectories or modify the given trajectory. A distance measure is defined in the latent space of distributions, and the framework is trained using a self-supervised triplet loss that maximizes distance between similar trajectory segment representations and minimizes the same between dissimilar pairs. 

Experiments are conducted on human pose dataset, with Variational RNN and Trajectron++ as baselines, with results for prediction and interpolation tasks. Qualitative results are also provided to show learned representation manipulation for speed change and temporal offset.

strengths_and_weaknesses: Strengths :
1) The paper is well written and easy to read. 
2) The idea of learning trajectory representation as a probability distribution, which can be decoded in continuous time is novel, and is of interest to the community. The various aspects of problem formulation in explored in good detail. It is interesting to see the use of box embeddings for trajectory data representation learning. 
3) The idea is presented in a fairly generic form, and it could have relevance in diverse problem domains involving trajectory data. 


Weaknesses :

The main weaknesses center around lack of experimental evidence for some of the central claims in the paper.  
1)  While the authors claim the learned representation captures the uncertainty in unobserved segments, the multimodality of prediction is not explored in the experiments section. Can the framework be used to decode diverse realistic pose variations?
2) The claim of learned representations being useful for comparing trajectories (line 160) is also not evaluated. The authors have referred to prior work on trajectory similarity detection / trajectory clustering, but the proposed approach is not compared with prior work for this task.

######## EDIT after rebuttal : The authors have addressed my main concerns as well as those from the other reviewers. Hence, I would like to raise my rating.
questions: (Please see the weaknesses section.)


Minor errors, typos:

L141 Shouldn’t it be a division operation instead of subtraction?

L192 Trajectron++

limitations: Adequately addressed by the authors.
review_title: null

Meta-review: metareview: This paper presents a new method for learning spatial partial trajectories. The trajectories are embedded as probability distributions in a learned latent space. The proposed framework is shown to interpolate and extrapolate partially observed trajectories. Experiments on three real datasets show that the proposed method outperforms existing state of the art methods.
The reviewers find the paper well-written and the proposed method novel, technically strong and interesting. The reviewer raised a few issues regarding the lack of additional ablation studies and the fact that the method is evaluated on the single task of joint-space trajectory domain. These issues are not considered by the reviewers as very crucial. 
confidence: Certain
Decision: Accept