Representing Spatial Trajectories as Distributions
Authors: Didac Suris Coll-Vinent
Carl Vondrick
Venue: NeurIPS 2022 Accept (2022)
Keywords: representation learning
human pose
video

Abstract: We introduce a representation learning framework for spatial trajectories. We represent partial observations of trajectories as probability distributions in a learned latent space, which characterize the uncertainty about unobserved parts of the trajectory. Our framework allows us to obtain samples from a trajectory for any continuous point in time—both interpolating and extrapolating. Our flexible approach supports directly modifying specific attributes of a trajectory, such as its pace, as well as combining different partial observations into single representations. Experiments show our method's superiority over baselines in prediction tasks.

Representing Spatial Trajectories as Distributions
Didac Surís Columbia University didac. sidac.suris$colle www.sscoolumba..eeet.. edu
Carl Vondrick Columbia University vondrick@cs. www.soia..l.a.eeeooooooeoooeeee. edu
Abstract
We introduce representation learning framework for spatial trajectories. We represent partial observations of trajectories as probability distributions in a learned latent space, which characterize the uncertainty about unobserved parts of the trajectory. Our framework allows us to obtain samples from a trajectory for any continuous point time-both interpolating and extrapolating. Our flexible approach supports directly modifying specific attributes of trajectory. such as its pace, as well as combining different partial observations into single representations. Experiments show our method's s advantage over baselines in prediction tasks. See trajectories cs col columbia. edu for video results and code.
Introduction
The visual world is full of objects moving around in predictable ways. Examples of these spatial trajectories include human motion, such as people dancing or exercising: objects moving, such as a ball rolling; trajectories of cars and bicycles; or animal migration patterns. Evidence suggests that the human perceptual system encodes motion into high-level neural codes that represent the motion holistically, going beyond the specific input observations [22]. Humans use this abstract representation for downstream tasks like inferring intention ( Computer vision systems likely need similar mechanisms to encode trajectories and motions into global representations.
Representation learning has been transformative in other domains such images and text for its ability to obtain high-level representations that reorganize the information in the input, and are better downstream tasks than the original signals. A global representation of trajectories would allow us to evaluate trajectory at any point in time. even ones not yet observed. However, modeling trajectories presents a series of challenges for representation learning. First, in real-time scenarios, the future of the trajectory is never observed. Second, temporal and spatial occlusions may impede observing part of trajectory. Third, trajectories are by nature continuous in time. And finally, trajectory-level metric is usually not well defined and application-dependent
We propose a representation learning framework for trajectories that deals with all these challenges in a unified way. Our key contribution is the representation of a partial observation of a trajectory probability distribution in learned latent space, that represents all the possible trajectories the observation could have been sampled from. Our framework's 's simplicity and generality allows to be flexible: it does not constrain the input-space metric, accepts observations of different lengths and at any (irregularly sampled) point in time, can be implemented using different families of latent space distributions, and is capable of performing inference- time tasks for which has been explicitly trained. Our experiments on human movement datasets show that our method can accurately predict the past and future of trajectory segment, well as the interpolation between two different segments, outperforming autoregressive baselines. Additionally, it can do so for any continuous point in time. We also show how we can modify given trajectories by manipulating their representations.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Input
Output (extrapolation)
O
Input
Output (interpolation)
Input
GUINOT
NOTH
Figure I: Predictions on figure skating data (FisV). Our model is capable of predicting the future
(top row), past, and interpolation (bottom row) of a trajectory given partial observations, any continuous time. The imputs to the model are the keypoints in the images, See more examples in Fig.
2 Method
2.1 Framework, Definitions and Notation
The imput to framework is sequence of samples obtained from (continuous in time and infinite) spatial trajectory u, which we define as the continuous temporal evolution of a set of spatial coordinates. We call each sample point æ, which lives in the impul space R.K. We call the sequence of points, together with the times at which they were sampled, a segment 8, which can be understood as partial observation of u. We define distance metric 8 between points æ in the input space.
Our goal is to transform these measurements of motion s into a representation Z that will be useful for downstream tasks. We define latent space R of trajectories full extent of a trajectory, both in time and in space. We use Q to represent probability distributions over trajectories 2, in the latent space. We define distance function D between distributions of trajectories, which assumes an underlying distance function d between trajectories 2.
Each in this space represents the
We use an encoder θ to encode every segment 8 to probability distribution Q(::)) (s)) over trajectories, where Q(2; 8) represents the probability that 8 was sampled from the trajectory represented by z. Additionally, obtaining a point x = 2(z,t), t). & takes any continuous as input. See Fig. 2 for a schematic.
can decode trajectory a specific time t by using decoder 4
2.2 Representation Learning
When observing segment one may have some uncertainty about the specific trajectory it was sampled from. For instance, a segment showing a person jumping may correspond trajectory that continues with the person falling, or to a trajectory that proceeds with them doing a backflip and landing on their feet, but will not belong represent the segment as distribution over trajectories, where Q(2; s) represents the likelihood of trajectory given the segment. During training, the goal is learn this mapping from the input space (segments of trajectories) to the latent space (distributions over trajectories).
trajectory of a person swimming. Therefore, we
segments sa 8 that have been obtained from the same underlying trajectory, we want some z to exist such that its likelihood under the distributions Qo and Q" representing each of the segments is high. To encourage this, we train the model to maximize the overlap between the distributions Q" and Q². Similarly, we minimize the overlap between (the distribution repre- sentations of) segments sampled from different trajectories, under the assumption that no trajectory z exists that contains both segments. Specifically, we minimize self-supervised triplet loss:
Concretely, given
Lene
Σ (i,6) Kk JET
max [D -D('e0))-((('*+)) +
(Q'.
+a,0
where is margin hyperparameter, and T is set of triplets: for every segment i in the dataset, define several triplets by sampling pairs consisting of positive segment k (such that (i, k+) is positive pair) and negative segment -- (such that ki,k-) is negative pair).
In addition to learning representations of trajectories, we also wish to be able decode them back to input-space points. To achieve this, we train decoder & that allows to obtain the specific value

---

K
Latent Space R N
Input Space R
Encoder (Transformer)
s
Et
C dec
L enc
P
U
Decoder
Figure 2: Schematic of our framework. We show the input space Ex the latent space IR and the mappings between the two (encoder & and decoder Φ). A segment belonging to trajectory is encoded into a distribution Q, from which a trajectory 2 is sampled and decoded at a time t, to get 2,.
of any trajectory any continuous time t. In order to train the decoder + sample trajectories 2 ~ Q(::s) from each segment representation, and decode them at specific time-steps t which were contained in obtaining prediction &, $(2,t) () for which have ground truth dy. There is uncertainty in this prediction, as was part of the segment in the first place; the decoder is only explicitly trained for reconstruction, not extrapolation. We train the decoder via regression, using the point-wise distance ò. Note that we never explicitly define a trajectory-level distance in the imput space; it is implicitly learned by the model. The reconstruction loss is mathematically defined
Laec
1
E (
Σ (0(2, ()a))
where N is the number of segments in the dataset, and T is the number of points in segment s'. We minimize Eqs. ( and (2) jointly and end-to-end. implement the encoder using a Transformer Encoder architecture [54] and the decoder & using ResNet 17 See Appendix C for more details.
2.3 Creating Positive and Negative Pairs
In order to define positives and negative pairs for Eq. 11, the following:
Input-space relationships. The simplest way is to take segments from the same trajectory as positives and segments from other random trajectories as negatives. The initial segments can have different relationships, such as precedence, containment, overlap 3 In our experiments,
sample three segments for every trajectory: a past segment (P in Fig. 3). a future segment (F) whose starting time comes right after the end of the past segment, and combinution segment (C), which contains both the past and the future segments.
Intersection. An intersection of two distributions Q in the latent space will represent all the trajectories that have a high likelihood for both intersected segments. Note that an intersection in the latent space is a union in the input space: the intersection constrains the possible trajectories to those that consistent simultaneously for the two segments. Similarly, an intersection in the input space (assuming an overlap between segments) a union in the latent space. In the latent space, the intersection of the past and future segments should be equal to the representation of the combination segment, and therefore the pair (C, I) is a positive one.
Re-encoding. Given a trajectory 2, we can decode into any set of times f, obtaining new segment. This segment can be reecdddd using O, and representation can be obtained for it. resulting in a new positive or negative for other segment representations. For example, when given the past randomly sample possible the future, the resulting segment (FP -fture given past) will be different than the ground truth future, so the pair (F, FP) will be treated as a negative.
We exemplify combination of these possibilities Fig. 3l In order to determine which pairs of segments are positive, and which negative, the rule is always the same: if they belong the

---

Input Space R
K
Latent Space R N
PF
Legend
CP
Past
PP
F Future
C Combination
CP
Future given past
Past given future
CF
CP Combination given past
Combination future
Other
F
PF
Figure 3: Examples of segments. We illustrate how spatial trajectories (left) are ideally encoded into the latent space (right). The intersection between two segment representations (boxes in the figure) represents the trajectories that contain the two segments. "Future given past" represents a segment decoded at a future time, from trajectory sampled from the past representation. is effectively a sample possible future given the past. Other segments are defined similarly. For clarity, we do show other options like "past given past", which would be the same box as past P. Best viewed in color
same trajectory they are positives, otherwise they are negatives. For example, looking at Fig. i clear that, as discussed above, there is no trajectory that can contain both F and FP. We list negative and positive pairs in Appendix C.I
2.4 Comparing Distributions
Eq. I uses the distance function D to compare distributions of trajectories. In this section, we introduce two different ways of designing D, resulting in different intuitions about the latent space.
Symmetric Distance two segments can belong to the same trajectory, the distributions Q representing each segment should be similar and close to each other (positives), and the (symmetric) distance between them should be small. For example, in Fig. 3 the representations the past and future F segments belonging the same trajectory are treated positives.
Conditional Instead of computing a distance or a similarity, we compute the probability that segment 8 belongs to the same trajectory as another segment sh, We model this as a conditional probability P(Q*(Q*). There are four possibilities:
1. P(Q% = when s includes s" like the combination segment C including the past
2. 0 < P(Q" < 1, when Q" is possible but not necessary given by Q*, like P and F in Fig
3. 0 < P(Q"IQ") < 1, defined in similar manner.
4. P(Q"QQ") = 0, for unrelated segments, like C and O.
We treat the first three cases positives, and the last case negative. Because pairs belonging to the first case have a stricter correspondence than those belonging to the second and third cases, we sample them more often during training. Note that under this interpretation, past P and future F from the same trajectory do not have a strong correspondence (first case), but a softer one (second and third cases): one does not fully define the other. This approach results in probability values that we either maximize (positives) or minimize (negatives), so we define D ( B) = P(A
The previous approaches require a way of computing either distance between the distributions Q, or a conditional probability between them. In the next section, we show two families of distributions for which these can be defined.

---

2.5 Trajectory Segments as Distributions
In order to obtain Q, the encoder θ predicts the parameters distribution family. Conditions for the distribution families are: 1) we can sample from it in a differentiable way. 2) we can parameterize it. 3) we can compute, closed form, an intersection that returns distribution from the same family, and 4) we can compute either similarity function or a conditional probability, or both (see Sec 2.4) Next, we introduce two distribution families that meet the previous criteria.
Normal distributions We use uncorrelated multivariate normal distributions, and parameterize them with a mean f and a standard deviation o. We compute the intersection as the product two normal distributions, which remains normal when the dimensions are uncorrelated (see Appendix D We use the symmetrized Kullback-Leible function. This distance is not proper metric; alternatives are discussed in Appendix D Normal distributions assume an underlying Euclidean distance metric d between trajectories 2.
(KL) divergence between distributions as a distance
Box embeddings Box embeddings (55] represent objects with high -igh-dimensional products-of- intervals (or boxes), parameterized by their two extreme vertices V and z². The intersection be- tween box embeddings is well defined and results in another box embedding. This makes them natural choice to represent conditional probabilities, which can be computed as P(A[B) = Vol A n B)/Vol(B), B), where (ol(A) = II: m of 0) is the volume of the box, and O represents the intersection operation. These operations are straightforward to compute. Boxes are not actual distri- butions, as they need not integrate to one. However, they are easily normalized by dividing by their volume, and therefore they can be treated as distributions for all the practical purposes required in our framework (i.e. sampling, where we approximate the boxes with uniform distribution). Symmetric distance functions can also be defined on box embeddings; we define a few in Appendix [.22
In both cases, we use the reparameterization trick (24) in order to sample from the distributions while keeping gradient information. We found the best-performing option was using box embeddings under the conditional scenario; the values reported in Section 3.2 use this setting.
2.6 Inference
Once trained, our decoder &b is able decode a trajectory at any continuous time t, including times that were not part of the input. For example, our framework can decode future segment given an input past segment, by sampling from its representation, and evaluating that sample at some future times. This future segment will not necessarily be equal to the ground truth future segment (in case it exists), because a single past can have multiple futures.
Overall, our framework is capable of doing 1) future and past prediction, by decoding a segment at times outside of its range; 2) continuous reconstruction given discrete input, by decoding at any continuous time /; 3) interpolation between two segments, by decoding trajectories in their latent- space intersection; and 4) modifying existing trajectories, by manipulating the latent space. All the previous tasks are possible without explicitly training to do any of them. We show examples in Sec. 3
3 Experiments
3.1 Datasets
For our experiments, we selected data adhering to the following criteria. First, there has to be uncertainty in the trajectory when given just segment (for instance, the future is not fully specified given the past). Second, the prediction should not require external contextual information. Context can be seamlessly added to our architecture, but involves additional task-specific enginecring decisions, and we want our evaluation to be orthogonal to them. Similarly, we avoid trajectories that require highly-engineene point-level distances 6. Finally, we prefer our trajectories to be obtained from real- world data. For all the previous reasons, we implement our framework on human movement datasets.
Specifically, we extract keypoints from human action datasets using OpenPose 10 For every video, we keep the most salient human trajectories. This results in sequences of dimension [L. 25, 2]. where T, the number of frames in the trajectory, 25 the number of joints in a human skeleton extracted by OpenPose, and 2 corresponds to the number of spatial coordinates for every joint. We refer to the
5

---

Table Prediction results. We report the mean squared error (the lower the better) across keypoints, after normalizing each trajectory to be contained in region of size 100 x 100. F, P and stand for "future", "past" and "interpolation", respectively. Values are obtained over runs with different test-time random seeds (changes include sampled segments and sampled 2). An extended table with standard deviations is in Appendix
(a) Long sequences
FineGym
Diving48
FisV
F
F
P
F
P
I
VRNN
I3
15.85
15.93
16.10
23.51
27.97
25.66
14.95
15.03
15.08
Trajectron++ uni.
44
9.54
9.98
9.73
11.67
16.52
11.98
11.42
11.85
11.68
Trajectron++ 44
9.72
10.01
9.89
11.59
16.23
12.68
11.41
11.71
11.63
TrajRep (ours, ablation)
8,82
9.07
7.57
10.00
11.74
10.06
10.62
11.27
9.70
+ re-encoding (ours)
8.50
8.83
7.11
9.81
12.00
9.58
10.32
10.77
9.22
(b) Short sequences
FineGym
Diving48
FisV
F
F
P
F
P
VRNN B3
12.77
13.20
13.40
18.36
20.14
19.86
13.26
13.44
13.45
Trajectron++ uni. 44
7.80
8.28
7.48
9.05
10.36
8.29
9.23
9.68
8,86
Trajectron++
7.26
7.93
6.94
8.74
11.35
8.31
8.70
9.28
8.28
TrajRep (ours, ablation)
6.49
6.59
5.15
6.94
6.99
5.00
7.83
8.17
6.01
+ re-encoding (ours)
6.20
6.36
4.88
6.76
6.85
5.04
7.54
7.78
5.88
whole skeleton at every time-step the combination of all joints, resulting in a K = -00-dimenioonal vector- as point. As distance function between points (i.e. skeletons) we use (2-norm distance per-joint, and average across all visible joints. We extract human movement trajectories from the FineGym [45), Diving48 (29) and FisV (57) datasets, which correspond to gymnasties, diving and figure skating, respectively.
For each of the datasets, we experiment with short sequences (up to time-steps, or slightly over one second) and long ones (up to 30 time-steps, representing slightly under four seconds of the trajectory), and report results for both. We provide more details on the dataset creation in Appendix
3.2 Quantitative Experiments
Baselines and ablations As baselines, we select vujectory-modeile methods that are capable of encoding uncertainty about the future. Variational RNNs [33 extend recurrent neural networks (RNNs) to the non-deterministic case, by modeling every step with a variational auto-ens outo-encoder ( (VAE) 4 a state-of-the-ar rrjjectory-modeling framework which also builds on top of RNNs and (conditional) VAFs (23). Uncertainty is modeled as a Gaussian mixture model (GMM). We adapt Trajectron++ to our data, making the encoding and decoding as similar to our setting as possible (for fairness), while keeping the core of the framework intact. We train two Trajectron++ versions, one with uniformly-sampled inputs and outputs ("Trajectron++ uni."), and a second one with non-uniform sampling, following the setup in our models ("Trajectron*+*") We also ablate model, and report results with and without training with re-encoded segments.
24]. Trajectron++
Tasks and metric We evaluate our framework on three different tasks: future prediction, past prediction, and interpolation between two segments. Future prediction consists in predicting points from future segment given a past segment. Past prediction is defined symmetrically. In the interpolation task, we input two sepurated segments (past and future) from trajectory, and predict the segment in between them. In our model, we do by decoding from the latent-space intersection of the two input segments. Baselines (which are autoregressive) are not capable of doing this combination, so only use the past segment input. Baselines are also not capable of directly

---

Input: 8 frames, second
Input: past
0
1.
tw0.0s
0.8
2.0 2.16
Prediction: future
(a) Future prediction. We show an example of future prediction, where the input are eight inegullrll sampled frames during one second (we only show three of them), and we predict up to three seconds into the future.
Input: frames, second
Input: future
t(s)
t=0.0s
0.68
0.84
1.32
2.04
2.68
3.0
4.2
Prediction: past
(b) Past prediction. We show example of past prediction, where the input are eight irregularly sampled frames during one second (we only show three them), and we predict up to three seconds into the past.
Input: past future
Prediction: middle
S
SAC
SA
IIS
CH
V VISAC
Vincs
(c) Interpolation. We provide the model with skeleton keypoints coming from two separate segments, and sample points at times in between these two segments, from the intersection of the two segments in latent space. The model produces sensible interpolations that are simply a linear interpolation at the joint level: in the first example, the gymnast first stands, then turns; in the second example, the gymnast swings right and left, just in time to end up meeting the future segment at the right position.
Figure 4: Predictions on gymnastics data (FineGym). We show examples past, future, and interpolation predictions. The input to our model are (irregularly in time) sampled keypoints obtained from human movement datasets, and the outputs are predictions of the trajectories at different continuous times (past, future, between the inputs). The only input to the model are the keypoints, not the images. Results show our model's capabilities for modeling trajectories well outside of the input's temporal range, for dealing with spatial and temporal occlusions, and for doing so at large temporal resolution. See Section 3.3 for deeper amalysis.
7

---

x1.0
x1.2
x1.4
x1.6
(a) Speed change.
+0.0
+1.0
+2.0
t
(b) Temporal offset.
Figure 5: Temporal editing. We decode, for the same times t, different trajectories in the latent space. These trajectories have been obtained by encoding the segment in the top row, and moving in small increments in the latent space along directions that represent speed (a) temporal offset (b). We highlight in green and pink some correspondences between different decoded trajectories, to emphasize that the spatial trajectories are the same but with variations in some time-related attribute, such as speed or temporal offset.
performing past prediction, we use the average of the (2-norm distance across joints, which used by all methods during training, and report the best out of M 10 samples, account for multiple modes and uncertainty in the prediction. Note that our model has never been explicitly trained to perform any the previous tasks.
predict the future of the reversed trajectory instead. As metric,
We show results in Tables la and Tb for long and short trajectories respectively. Our model outperforms baselines in all the tasks, which proves its value and flexibility. We also show how creating more interesting negatives with the re-encoding of decoded trajectories results in more accurate prediction results. However, our method performs well even without re-encoding.
3.3 Qualitative Experiments
We show examples of our model's inputs and outputs Fig. 4 Specifically, we show future, past and interpolation predictions. The results reflect that the model learns to predict sequences up to four times longer than the input. They also show the large temporal resolution of our model: the model predictions evolve smoothly and sensibly for time-steps separated by few hundredths of second (Figs. 4a and When not all joints are present in the input (first frame in Fig, 4a), our model is still capable of reconstructing the full spatial extent of the position. Finally, note how the model can take

---

Input (past)
Prediction (future)
Ground truth
Sample
Sample #2
Ground truth
Sample
Sample #2
Ground truth
Sample
Sample #2
Ground truth
Sample #1
Sample #2
Figure 6: Multiple futures. Given a few input frames, our model is capable of predicting the future. It does so by modeling distribution over the possible trajectories: by sampling from this distribution, we can obtain different plausible futures given the input (past) segment. for specific inputs, the ground truth future, well two different futures sampled from the input segment distribution, which have been sampled randomly. This figure shows that our model is indeed capturing the molti nature of the trajectories under uncertainty.
this figure we show,

---

as inputs irregularly sampled time-steps (Fig. 4b), which makes it adaptable to temporal occlusions. Baselines are only capable of predicting the future, and are restricted to predicting uniform time-steps.
Temporal editing The segments are directly tied to the temporal span they represent. For example, two segments with the exact same coordinates and evolution across time, but starting at different times will result in different (albeit similar) representations. The speed of a movement and the time the trajectory where the movement is done are important attributes of that movement, and the representation should not be invariant to them: they belong to different trajectories. However, because these trajectories are very similar, our model learns to represent them close in the latent space. In Fig. 5 we show that the model encodes different temporal variations. Moving along specific directions in the latent space results in progressively faster trajectories (Fig. or in trajectories with an increasing temporal offset with respect to the original one (Fig. 5bb. See Appendix E for details.
Representing multiple futures A crucial aspect of our formulation the assumption that the future is uncertain, and that model has to be capable of modeling the different modes of the trajectory distribution. In Fig. 6 we show examples of multiple predicted futures given a single past segment, proving that our model captures the multi-mon modal nature of the trajectories under uncertainty.
4 Related work
Modeling trajectories Spatial trajectories are usually modeled in the literature in an autoregressive
44 30 58 52 25 19). where trajectories are defined conditioned on previous time-steps. Despite their success, AR models are incapable of dealing with some of the challenges stated in Section | most notably they do not represent time as a continuous variable, they cannot model the full extent of a trajectory (simultaneously both past and future), and no learned trajectory-level metric can be obtained from them. Some of them model the uncertainty in the prediction [52 25 19 20,
(AR) fashion [47 33 26
2 49 20
49]. and we use two representative ones [3] as our baselines. different line of work focused on representing segments of trajectories (not just points) as points 61
44 13
in latent space [62 65 60)
28 64 9 31]. However, they are not capable of modeling the full extent of trajectory outside the limits of the considered segment. Additionally, the segment-level
metric is either unstructured 622, or is explicitly given 65 63 644
Continuous time
Modeling time as a continuous signal has gained traction recently in fields such as
graphics [56 53 39 46 or physics modeling [I), because it accurately represents the underlying (continuous) world being modeled. In the graphics neural implicit functions literature, time used to condition the prediction of the network. We adopt the same approach in our decoder. We encode the set of continuous times in a segment by using a Transformer network [54], which by construction is permutation invariant, but allows temporal embeddings to be concatenated with the input, both discrete [5] and continuous [53 51]
Self-supervised representation learning Finding self superviedd representations for temporal data has been the subject of a large amount of work in domains such as trajectories [1], video [35 40) or audio processing [2] 42 [5). Most methods, however, represent segments simple points in Euclidean space. Structured representations for temporal data [50] 36 allow the latent space to follow certain inductive biases, like our framework's idea that segments compose trajectories. We model segments as either normal distributions or box embeddings [55). The latter have been used to represent hierarchical relationships in domans such as text [38 34 knowledge bases 01 or images [41). We use them to represent temporal information. In recent work, Park et al. (36] also model segments using normal distributions, where trajectories are weighted sums of the segment representations.
Acknowledgments and Disclosure of Funding
We thank Arjun Mani and Mia Chiquier for helpful feedback. This research is based on work partially supported by the NSF NRI Award #2132519 and the DARPA MCS program. DS is supported by the Microsoft PhD Fellowship.
10

---

References
[1] Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems (NeurIPS), 33:9649-9661, 2020.
[2] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio
Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEF/CVF
Conference on Computer Vision and Pattern Recognition (CVPR), pages 961-971, 2016.
[3] James F. Allen. Maintaining knowledge about temporal intervals. In Communications of the ACM, Volume
26 Issue 11, 1983.
[4] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages
2021.
[5] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learming (ICML), July 2021. [6] Sarah-Jayne Blakemore and Jean Decety, From the perception of action to the understanding of intention. Nature reviews neuroscience, 2(8):561-566 -676115672222 2001.
[7] Paul Bromiley. Products and convolutions of gaussian probability density functions. Tina- Vision Memo,
3(4):1, 2003.
[8] Michael Bronstein. Beyond message passing: physics-inspired paradigm for graph neural networks. The Gradient, 2022.
[9] Judith Butepage, Michael J Black, Danica Kragic, and Hedvig Kjellstrom. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 6158 6666, 2017.
[10] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[11] Ben Chamberlain, James Rowbottom, Maria Gorinova, Michael Bronstein, Stefan Webb, and Emanuele Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning, pages 1407-
1418. PMLR, 2021.
[12] Tejas Chheda, Purujit Goyal, Trang Tran, Dhruvesh Patel, Michael Boratko, Shib Sankar Dasgupta, and Andrew McCallum. Box embeddings: An open- source library for representation learning using geometric structures. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2021.
[13] Junyoung Chung. Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.
[14] Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Lorraine Li, and Andrew McCallum. Improving local identifiability in probabilistic box embeddings. In Conference on Neural Information Processing Systems (NeurIPS). 2020.
[15] Eduardo Fonseca, Diego Ortego, Kevin McGuinness, Noel E O'Connor, and Xavier Serra. Unsupervised contrastive learning of sound event representations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 371-3 -375, IEEE, 2021.
[16] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learnigg by dense predictive coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR) Workshops, pages 0-0, 2019.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[18] John R Hershey and Peder A Olsen. Approximating the kullback leibler divergence between gaussian mixture models. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing- ICASSP volume 4, pages IV-317. TEEF, 2007.
[19] -inn Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems, 31, 2018.
[20] Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi ageet trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2375-2384, 2019.
[21] Aren Jansen, Manoj Plakal, Ratheet Pandya, Daniel PW Ellis, Shawn Hershey, Jiayang Liu, R Channing Moore, and Rif A Saurous. Unsupervised learning of semantic audio representations. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 126-130. IEEF, 2018.
[22] Gunnar Johansson. Visual perception of biological motion and model for its analysis. Perception & psychophysics, 14(2):201-2) 2211 1973.
11

---

[23]
Durk P Kingma. Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. Advances in neural information processing systems, 27, 2014.
[24]
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014.
[25] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018. [26] Philipp Kratzer. Mare Toussaint, and Jim Mainprice. Prediction of human full-body movements with motion optimization and recurrent neural networks. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1792-1798 1798. IEEE, 2020.
[27] Xiang Li. Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. Smoothing the geometry of probabilistic box embeddings. In International Conference on Learning Representations (ICLR), 2019. [28] Xiucheng Li, Kaiqi Zhao, Gao Cong, Christian S Jensen, and Wei Wei. Deep representation learning for trajectory similarity computation. In 2018 IEEE 34th international conference on data engineering
(ICDE), pages 617-628. IEEE, 2018.
[29] Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In European Conference on Computer Vision (ECCV), pages 513-528, 2018.
[30] Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Todrake. Propaga- tion networks for model-based control under partial observation. In 2019 International Conference on Robotics and Automation (ICRA), pages 1205-1211. IEBE, 2019.
[31] Xiang Liu, Xiaoying Tan, Yuchun Guo, Yishuai Chen, and Zhe Zhang. Cstrm: Contrastive self-supervised trajectory representation model for trajectory similarity computation. Computer Communications, 2022.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations (ICLR), 2019.
[33] Julieta Martinez. Michael J Black, and Javier Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2891-2900, 2017.
[34] Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. Modeling fine-grainer entity types with box embeddings. In Annual Meeting the Association for Computational Linguistics (ACL).
page 2051-20664 2021.
[35] Tian Pan, Yibing Song. Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11205-11214, 2021.
[36] Jungin Park, Jiyoung Lee, Ig-Jac Kim, and Kwanghoon Sohn. Probabilistic representations for vidco contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d' d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Newral Information Processing Systems (NeurIPS), pages 8024 8035. Curran Associates, Inc., 2019.
[38] Dhruvesh Patel. Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew McCallum. Representing joint hierarchies with box embeddings. In Automated Knowledge Base Construction, 2020.
[39] Albert Pumarola, Enric Corona, Gerard Pons Moll, and Francese Moreno-Nopee Noguer. D-nerf: Neural radiance fields for dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition ( (CVPR), pages 10318-10327, 2021.
[40] Rui Qian. Tinjjaan Meng. Boqing Gong. Ming Hsuan Yang. Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6964-6974, 2021.
[41] Anita Rau, Guillermo Garcia-Hernande Danail Stoyanov, Gabriel J Brostow, and Daniyar Turmukhambe- tov. Predicting visual overlap of images through interpretable non-metrie box embeddings. In European Conference on Computer Vision (ECCV), pages 629-646. Springer, 2020.
[42] Aaqib Saced, David Grangier, and Neil Zeghidour. Contrastive learning of general-purpose audio represen- tations. In ICASSP 202/-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 1875-3874 IEEE, 2021.
[43] Antoine Salmona, Julie Delon, and Agnès Desolneux. Gromov-wasserstein distances between gaussian distributions. arXiv preprint arXiv:2104. 0007000 2021.
[44] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++:: Dynamically- feasible trajectory forecasting with heterogeneous data. In European Conference on Computer Vision (ECCV), pages 683-700. Springer, 2020.
[45] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine- finengrained action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2020.
12

---

[46]
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In arXiv, 2020.
[47] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng. and Jiaying Liu. An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[48] Nishant Subramani, Samuel Bowman, and Kyunghyun Cho. Can unconditional language models recover
arbitrary sentences? Advances in Neural Information Processing Systems, 32, 2019.
[49] Chen Sun, Per Karlsson. Jiajun Wu, Joshua Tenenbaum, and Kevin Murphy. Stochastic prediction of multi-agent interactions from partial observations. International Conference on Learning Representations (ICLR), 2019.
[50]
Dídae Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[51] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 7537-7547, 2020.
[52] Charlie Tang and Russ Salakhutdinov. Multiple futures prediction. Advances in Neural Information Processing Systems, 32, 2019.
[53] Basile Van Hoorick. Purva Tendulkar, Didac Surís, Dennis Park, Simon Stent, and Carl Vondrick. Revealing occlusions with 4d neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[55] Luke Vilmis, Xiang Li, Shikhar Murty, and Andrew McCallum. Probabilistic embedding of knowledge graphs with box lattice measures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages (63-272, 2018.
[56] Wenqi Xian, Jia- Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free- viewpoint video. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9421-9431, 2021.
[57] Chengming Xu, Yanwei Fu, Bing Zhang, Zitian Chen, Yu-Gang Jiang, and Xiangyang Xue. Learning to score figure skating sport videos. JEEE transactions on circuits and systems for video technology, 30(12)4578-45500 2019.
[58] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018
[59] Sijie Yan, Yuanjun Xiong, Jingho Wang, and Dahua Lin. Mmskeleton. https:/ /ttt:/ub///ubb...../. c
peen-mmlabmmmskeeeton
ppen-mmlab/ mmskeleton 2019.
[60] Di Yao, Gao Cong, Chao Zhang, and Jingping Bi. Computing trajectory similarity in linear time: A generic sced-guided neural metric learning approach. In 2019 JEEE 35th international conference on data engineering (ICDE), pages 1358-1369. IEEE, 2019.
[61] Di Yao, Gao Cong, Chao Zhang, Xuying Meng. Rongchang Duan, and Jingping Bi. lincar time approach to computing time series similarity based on deep metric learning. IEEE Transactions on Knowledge and Data Engineering, 2020.
[62] Yao, Chao Zhang, Zhihua Zhu, Jianhui Huang, and Jingping Bi. Trajectory clustering via deep representation learning. In 2017 international joint conference on neural networks (JJCNN, pages 3880-
3887. IEEE, 2017.
[63] Guan Yuan, Penghui Sun, Jie Zhao, Daxing Li, and Canwei Wang. review of moving object trajectory clustering algorithms. Artificial Intelligence Review, 47(11:123-144, 2017.
[64] Wenyuan Zeng. Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun. Dsdnet: Deep structured self-driving network. European Conference on Computer Vision (ECCV), pages 156-172. Springer, 2020.
[65] Hanyuan Zhang, Xingyu Zhang, Qize Jiang. Baihua Zheng, Zhenbang Sun, Weiwei Sun, and Changhu Wang. Trajectory similarity learning with auxiliary supervision and optimal matching. International Joint Conference on Artificial Intelligence (IJCAI), 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes], see Sections 2 and S
(b) Did you describe the limitations of your work? [Yes] we discuss them Appendix A
(c) Did you discuss any potential negative societal impacts of your work? [Yes], we discuss them in Appendix A
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplementary material or as a URL)? [Yes], they are in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. see Appendix C
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]. see Table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes], see Appendix C
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes], we cite the datasets being used.
(b) Did you mention the license of the assets? [No], the licenses are mentioned in the respective dataset and library papers, which we cite.
(c) Did you include any new assets either in the supplementary material or as a URL? (es]. data and code are provided.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "lst4fjrKHW", "reviewer": "Reviewer_bCK7", "summary": "The paper aims to develop a method for trajectory prediction/ interpolation by learning trajectory\nrepresentations in a latent space. The paper uses a transformer encoder to learn parameters of\na distribution, and decoder to reconstruct the original trajectory point from an embedding in the\nlatent space. The paper uses the conventional triplet loss to force similarity between\nembeddings from the same trajectory segment, and a reconstruction loss for the decoder.", "questions": "Questions have been asked along with the weaknesses mentioned above.", "limitations": "Yes, in Appendix A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed method is sufficiently novel, with most previous works focusing on autoregressive predictions or the latent space representations not modeling future representations adequately.\n- The paper is generally well written, with the qualitative examples demonstrating the network’s ability to interpolate/extrapolate in diverse contexts.\n- The method allows sampling from a distribution, which comes with its benefitsirregularly spaced sampling, flexibility on the prior for \n distributions, generalizing to unobserved latent spaces.\n- The method performs better than the existing state of the art method of Trajectron++ on the human pose prediction task.\n\nWeaknesses\n- The paper lacks additional ablation studies, such as:\n    - How does the network performance vary with the number of samples, M?\n    - How does the choice of the encoding distribution impact network performance?\n    - How sensitive is the method to irregularly sampling from the distribution and how does it impact the final decoded trajectory?\n- It would be interesting to understand how non-compliant trajectories (irregular actions such as hand waving for a few timesteps) impact the learned latent space, both for interpolation and extrapolation. Additionally if the authors could comment on repetitive trajectories (actions that repeat periodically), that would be interesting.\n- The paper evaluates the method on the single task of joint-space trajectory domain- it would be interesting to see how this translates to other domains for trajectory prediction eg. autonomous driving, crowd prediction.\u0000\n- The paper mentions the lack of context aggregation in the method, which is understandable for simplicity reasons. However, to generalize outside of the joint-space prediction problem, context aggregation would be vital.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fJY4oGgJItC", "reviewer": "Reviewer_vvBV", "summary": "This paper proposes a spatial trajectory representation framework. The framework represents a partial observation of a trajectory as a probability distribution in a learned latent space. With this representation, the framework is able to perform a variety of different tasks, with examples including 1) future and past prediction, 2) continuous reconstruction given a discrete input, 3) interpolation between two segments, 4) comparison of different trajectories, and 5) modifying existing trajectories. The framework consists of an encoder and a decoder. The encoder is trained with self-supervised triplet loss. The decoder is trained with regression loss on ground-truth trajectories. The authors evaluated the method on public human movement datasets and compared its performance against VRNN and Trajectron++. The result shows that it achieves significantly better performance than the baselines.", "questions": "--- Other comments\n\n- Line 192: Typo \"Transformer++\".\n", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--- Strengths\n\n- The paper is well written and easy to follow.\n\n- The framework is able to perform a variety of different tasks without re-training.\n\n- The proposed method was evaluated on public datasets and achieved significantly better performance than the public baselines.\n\n\n--- Issues and suggestions\n\n- The baselines used in this work are a bit old. VRNN is from 2015, and Trajectron++ is from 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E3yH2coeRxX", "reviewer": "Reviewer_Ph8V", "summary": "This paper presents a new representation learning method for spatial trajectories, named by TrajRep. TrajRep is in an encoder-decoder structure and optimized based on the reconstruction loss, as well as a self-supervised triplet loss to refine the learned representations. For the self-supervised loss, the authors define the positive and negative loss by analyzing the structure of trajectories in detail and define the distance function based on the box embeddings under the conditional scenario. TrajRep is evaluated on the human movement dataset.", "questions": "1. Paper organization.\n- Figure 6 in the supplementary materials should be placed in the main text. Section 2.3 of the original paper does not clarify the authors' design well.\n- The formalization of box embedding (Equation (11)-(12)) should be mentioned in section 2.5 or related works. In the original version, it is hard to know the formalization of Vol in line 141.\n\n2. Some sentences of the paper are not well-supported.\n- Line 165: the authors claim that “our method is generally designed for any kind of spatial trajectory”, but the experiments only include the human movement dataset. More datasets are expected.\n- Line 148-line 150: the authors present two types of the distance function D and choose the box-embedding way without experiments of comparison.\n- Figure 6. The authors do not explain the reason why the relationship between two segments is labeled as HP or SP or others.\n\n3. Insufficient experiments.\n- Comparing baselines. This paper only includes two baselines. More baselines are expected, e.g. some GNN-based models, which are widely used in skeleton prediction.\n- Ablations studies. The design space of the paper is large. More ablations are expected, including the definition of D (symmetric distance or conditional), w/ or w/o triple loss, and the sampling strategy mentioned in line 530.\n- Hyperparameter sensitivity. Why choose $\\alpha$ as 1 (line 541)? Does the training process and model performance is sensitive to this hyperparameter?\n- Training curves. Since the extreme point may affect the triple loss, I would like to see if the training process is steadily convergent during training.\n\nConsidering the above questions, I think this paper needs much more effort in presentation and experiments.\n\n=====================\nIn the rebuttal, the author addressed the above questions well. I would like to raise my score.\n\n", "limitations": "Yes, the authors have discussed the limitations and potential negative social impact of this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. In general, I think this paper is interesting. Concretely, I think the novelty of this paper mainly lies in two aspects:\n- The whole pipeline for trajectory modeling, namely especially Equation (2) that decodes the raw trajectory from z and t.\n- The distribution formalization method in sections 2.4 and 2.5.\n\n2. The qualitative Experiments are impressive, especially the Figure 5.\n\n3. The related work is well researched.\n\n### Weaknesses\n1. I do not think this paper is well-organized. Due to the lack of details, it is difficult to clear up the method design from the main text. Too many details are hidden in the supplementary materials, which affects the reading of the main text.\n\n2. Some sentences of the paper are not well-supported.\n\n3. The experiments are insufficient, including the aspects of comparing baselines, ablations studies, analysis of hyperparameter sensitivity and training curves.\n\nSee the next section for more details of the weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2HmoHzSiUpn", "reviewer": "Reviewer_1wa9", "summary": "The authors propose a framework for learning representations for trajectory data from partial observations. The learnt representation is in the form of a probability distribution such that it captures the uncertainty of the unobserved segments. This distribution can be used for sampling unobserved segments of trajectories (interpolation, extrapolation, continuous time sampling), compare different trajectories or modify the given trajectory. A distance measure is defined in the latent space of distributions, and the framework is trained using a self-supervised triplet loss that maximizes distance between similar trajectory segment representations and minimizes the same between dissimilar pairs. \n\nExperiments are conducted on human pose dataset, with Variational RNN and Trajectron++ as baselines, with results for prediction and interpolation tasks. Qualitative results are also provided to show learned representation manipulation for speed change and temporal offset.\n", "questions": "(Please see the weaknesses section.)\n\n\nMinor errors, typos:\n\nL141 Shouldn’t it be a division operation instead of subtraction?\n\nL192 Trajectron++\n", "limitations": "Adequately addressed by the authors.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths :\n1) The paper is well written and easy to read. \n2) The idea of learning trajectory representation as a probability distribution, which can be decoded in continuous time is novel, and is of interest to the community. The various aspects of problem formulation in explored in good detail. It is interesting to see the use of box embeddings for trajectory data representation learning. \n3) The idea is presented in a fairly generic form, and it could have relevance in diverse problem domains involving trajectory data. \n\n\nWeaknesses :\n\nThe main weaknesses center around lack of experimental evidence for some of the central claims in the paper.  \n1)  While the authors claim the learned representation captures the uncertainty in unobserved segments, the multimodality of prediction is not explored in the experiments section. Can the framework be used to decode diverse realistic pose variations?\n2) The claim of learned representations being useful for comparing trajectories (line 160) is also not evaluated. The authors have referred to prior work on trajectory similarity detection / trajectory clustering, but the proposed approach is not compared with prior work for this task.\n\n######## EDIT after rebuttal : The authors have addressed my main concerns as well as those from the other reviewers. Hence, I would like to raise my rating.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept