Title: FUTURE PREDICTION WITH ADVERSARIAL GRAMMARS

Abstract: In this paper, we propose a differentiable adversarial grammar model for future prediction. The objective is to model a formal grammar in terms of differentiable functions and latent representations, so that their learning is possible through standard backpropagation. Learning a formal grammar represented with latent terminals, non-terminals, and productions rules allows capturing sequential structures with multiple possibilities from data. The adversarial grammar is designed so that it can learn stochastic production rules from the data distribution. Being able to select multiple production rules leads to different predicted outcomes, thus efficiently modeling many plausible futures. We confirm the benefit of the adversarial grammar on two diverse tasks: future 3D human pose prediction and future activity prediction. For all settings, the proposed adversarial grammar outperforms the state-of-the-art approaches, being able to predict much more accurately and further in the future, than prior work.
Code will be open sourced.

1. INTRODUCTION:

Future prediction in videos is one of the most challenging visual tasks. Being able to accurately predict future activities, human or object pose has many important implications, most notably for robot action planning. Prediction is particularly hard because it is not a deterministic process as multiple potential 'futures' are possible, and in the case of human pose, predicting real-valued output vectors is further challenging. Given these challenges, we address the long standing questions: how should the sequential dependencies in the data be modeled and how can multiple possible long-term future outcomes be predicted at any given time.

To address these challenges, we propose an adversarial grammar model for future prediction. The model is a differentiable form of a regular grammar trained with adversarial sampling of various possible futures, which is able to output real-valued predictions (e.g., 3D human pose) or semantic prediction (e.g., activity classes). Learning sequences of actions or other sequential processes with the imposed rules of a grammar is valuable, as it imposes temporal structural dependencies and captures relationships between states (e.g., activities). At the same time, the use of adversarial sampling when learning the grammar rules is essential, as this adversarial process is able to produce multiple candidate future sequences that follow a similar distribution to sequences seen in the data. More importantly, a traditional grammar will need to enumerate all possible rules (exponential growth in time) to learn multiple futures. This adversarial stochastic sampling process allows for much more memory-efficient learning without enumeration. Additionally, unlike other techniques for future generation (e.g., autoregressive RNNs), we show the adversarial grammar is able to learn long sequences, can handle multi-label settings, and predict much further into the future.

The proposed approach is driven entirely by the structure imposed from learning grammar rules and their relationships to the terminal symbols of the data and by the adversarial losses which help model the data distribution over long sequences. To our knowledge this is the first approach of adversarial grammar learning and the first to be able to successfully produce multiple feasible long-term future predictions for high dimensional outputs.

The approach outperforms previous state-of-the-art methods, including RNN/LSTM and memory based methods. We evaluate future prediction on high dimensional data and are able to predict much further in the future than prior work. The proposed approach is also general -it is applied to diverse future prediction tasks: 3D human pose prediction and multi-class and multi-label activity forecasting, and on three challenging datasets: Charades, MultiTHUMOS, and Human3.6M.

2. RELATED WORK:

Grammar models for visual data. The notion of grammars in computational science was introduced by Chomsky (1956) for description of language, and has found a widespread use in natural language understanding. In the domain of visual data, grammars are used to parse images of scenes (Zhu & Mumford, 2007; Zhao & Zhu, 2011; Han & Zhu, 2008) . In their position paper, Zhu & Mumford (2007) present a comprehensive grammar-based language to describe images, and propose MCMC-based inference. More recently, a recursive neural net based approach was applied to parse scenes by Socher et al. (2011) . However, this work has no explicit representation of grammar. In the context of temporal visual data, grammars have been applied to activity recognition and parsing (Moore & Essa, 2002; Ryoo & Aggarwal, 2006; Vo & Bobick, 2014; Pirsiavash & Ramanan, 2014) but not to prediction or generation. Qi et al. (2017) used used traditional stochastic grammar to predict activities, but only within 3 seconds.

Generative models for sequences. Generative Adversarial Networks (GANs) are a very powerful mechanism for data generation by an underlying learning of the data distribution through adversarial sampling (Goodfellow et al., 2014) . GANs have been very popular for image generation tasks (Emily L Denton, 2015; Isola et al., 2017; Wang et al., 2018; Brock et al., 2019) . Prior work on using GANs for improved sequences generation (Yu et al., 2017; Fedus et al., 2018; Hu et al., 2017) has also been successful. Fraccaro et al. (2016) proposed a stochastic RNN which enables generation of different sequences from a given state.

Differentiable Rule Learning Previous approaches that address differentiable rule or grammar learning are most aligned to our work (Yang et al., 2017) . However, they can only handle rules with very small branching factors and have not been demonstrated in high dimensional output spaces.

Future pose prediction. Previous approaches for human pose prediction (Fragkiadaki et al., 2015; Ionescu et al., 2014; Tang et al., 2018) are relatively scarce. The dominant theme is the use of recurrent models (RNNs or GRUs/LSTMs) (Fragkiadaki et al., 2015; Martinez et al., 2017) . Tang et al. (2018) use attention models specifically to target long-term predictions, up to 1 second in the future. Jain et al. (2016) propose a structural RNN which learns the spatio-temporal relationship of pose joints. The above models, contrary to ours, cannot deal with multi-modality and ambiguity in the predictions, and do not produce multiple futures. These results are also only within short-term horizons and the produced sequences often 'interpolate' actual data examples.

Video Prediction. Without providing an exhaustive survey on video prediction, we note that our approach is related to the video prediction literature (Finn et al., 2016; Denton & Fergus, 2018; Babaeizadeh et al., 2017) where adversarial formulations are also common (Lee et al., 2018) .

3. APPROACH:

Overview and main insights. Our approach is driven by learning the production rules of a grammar, with which we can learn the transitions between continuous events in time, for example 3D human pose or activity. While an activity or action may be continuous, it can also spawn into many possible futures at different points, similarly to switching between rules in a grammar. For example, an activity corresponding to 'walking' can turn into 'running' or continuing the 'walking' behaviour or change to 'stopping'. These production rules are learned in a differentiable fashion with an adversarial mechanism which allows learning multiple candidate future sequences. This enables robust future prediction, which, more importantly, can easily generate multiple realistic futures.

3.1. DIFFERENTIABLE RULE GENERATION FOR SEQUENCE MODELING:

A formal regular grammar is represented as the tuple (N, Σ, P, N 0 ) where N is a finite non-empty set of non-terminals, Σ is a finite set of terminals (or output symbols), P is a set of production rules, and N 0 is the starting non-terminal symbol, N 0 ∈ N . Productions rules in a regular grammar are of the form A → aB, A → b, and A → , where A, B ∈ N , a, b ∈ Σ, and is the empty string. Applying multiple productions rules to the starting non-terminal generates a sequence of terminals. Note that we only implement rules of form A → aB in our grammar, allowing it to generate sequences infinitely.

Our objective is to learn such non-terminals (e.g., A) and terminals (e.g., a) as latent representations directly from training data, and model the production rules P as a (differentiable) generative neural network function. That is, at the heart of the proposed method is learning nonlinear function G : N → {(N, Σ)} that maps a non-terminal to a set of (non-terminal, terminal) pairs. We denote each element (i.e., each production rule) derived from the input non-terminal as {(A i , t i )}. Note that this mapping to multiple possible elements enables modeling of multiple, different sequences, and is not done by existing models (e.g., RNNs).

For any latent non-terminal A ∈ N , the grammar production rules are generated by applying the function G, to A as (here G is a neural network with learnable parameters):

EQUATION

Each pair corresponds to a particular production rule for this non-terminal. More specifically,

A → t 1 B 1 A → t 2 B 2 . . . A → t K B K , where B 1 , B 2 , . . . B K ∈ N (2)

This function is applied recursively to obtain a number of output sequences, similar to prior recurrent methods (e.g., RNNs and LSTMs). However, in RNNs, the learned state/memory is required to abstract multiple potential possibilities into a single representation, as the mapping from the state/memory representation to the next representation is deterministic. As a result, when learning from sequential data with multiple possibilities, standard RNNs tend to learn states as a mixture of multiple sequences instead of learning more discriminative states. By learning explicit production rules, our states lead to more salient and distinct predictions which can be exploited for learning long-term, complex output tasks with multiple possibilities, as shown later in the paper.

For example, suppose A is the non-terminal that encodes the activity for 'walking'. An output of the rule A → walkingA will be able to generate a sequence of continual 'walking' behavior.

Additional rules, e.g., A → stoppingV , A → runningU , can be learned, allowing for the activity to switch to 'stopping' or 'running' (with the non-terminals V, U respectively learning to generate their corresponding potential futures). Clearly, for high dimensional outputs, such as 3D human pose, the number and dimensionality of the non-terminals required will be larger. We also note that the non-terminals act as a form of memory, capturing the current state with the Markov property.

To accomplish the above task, G has a special structure. The model contains a number of nonterminals and terminals which are learned: |N | non-terminals of dimensionality D, and |Σ| terminals of dimensionality C (the latter naturally correspond to the number and dimensionality of the desired outputs). G takes input of A ∈ N , then using several nonlinear transformations (e.g., fully connected layers), maps A to a vector r corresponding to a set of rules: r = f R (A). Here, r is a vector with the size |P | whose elements specify the probability of each rule given input non-terminal. We learn |P | rules which are shared globally, but only a (learned) subset are selected for each non-terminal as the other rule probabilities would become zero. This is conceptually similar to using memory with recurrent neural network methods (Yogatama et al., 2018) , but the main difference is that the rule vectors are used to build grammar-like rule structures which are more advantageous in explicitly modeling of temporal dependencies.

In order to generate multiple outputs, the candidate rules, r are followed by the Gumbel-Softmax function (Jang et al., 2017; Maddison et al., 2017) , which allows for stochastic selection of a rule. This function is differentiable and samples a single rule from the candidate rules based on the learned rule probabilities. These probabilities model the likelihood of each generated sequence.

Two nonlinear functions f T and f N are additionally learned, such that, given a rule r, output the resulting terminal and non-terminal: B = f N (r), t = f T (r). These functions are both a sequence of fully-connected layers followed by a non-linear activation function (e.g., softmax or sigmoid depending on the task). As a result,

G(A) = {(f N (f R (A)), f T (f R (A)))}.

The schematic of G is visualized in Figure 1 , more details on the functions are provided in the later sections.

The non-terminals and terminals are modeled as sets of high dimensional vectors with pre-specified size and are learned jointly with the rules (all are tunable parameters and naturally more complex datasets require larger capacity). For example, for a simple C-class classification problem, the terminals are represented as C-dimensional vectors matching the one-hot encoding for each class.

Difference to stochastic RNNs Standard recurrent models have a deterministic state, given some input, while the grammar is able to generate multiple potential next non-terminals (i.e., states). Stochastic RNNs (Fraccaro et al., 2016) address this by allowing the next state to be stochastically generated, but this is difficult to control, as the next state now depends on a random value. In the grammar model, the next non-terminal is sampled randomly, but from a set of deterministic candidates. By maintaining a set of deterministic candidates, the next state can be selected randomly or by some other method, giving more control over the generated sequences.

Learning the starting non-terminal. Given an initial input data sequence (e.g., a short video or pose sequences), we learn to generate its corresponding starting non-terminal (i.e., root node). This is used as input to G to generate a sequence of terminal symbols starting from the given nonterminal. Concretely, given the initial input sequence X, a function s is learned which gives the predicted starting non-terminal N 0 = s(X). Then the function G is applied recursively to obtain the possible sequences where j is an index in the sequence and i is one of the possible rules:

   {(B 1 i , t 1 i )} i = G(N 0 ), j = 0 {(B j+1 i , t j+1 i )} i = G(B j ), for j > 0 (3) 3.2 ADVERSARIAL RULE SAMPLING

The function G generates a set of (non-terminal, terminal) pairs, which is applied recursively to the non-terminals, resulting in new rules and the next set of (non-terminal, terminal) pairs. Note that in most cases, each rule generates a different non-terminal, thus sampling G many times will lead to a variety of generated sequences. As a result, an exponential number of sequences will need to be generated during training, to cover the possible sequences. For example, consider a branching factor of k rules per non-terminal with a sequence of length L. This results in k L terminals and nonterminals (e.g., for k = 2 we have ∼ 1000 and for k = 3 ∼ 60, 000). Thus, enumerating all possible sequences is computationally prohibitive beyond k = 2. Furthermore, this restricts the tasks that can be addressed to ones with lower dimensional outputs because of memory limits. With k = 1 (i.e., no branching), this reduces to a standard RNN during training, unable to generate multiple possible future sequences (i.e., we observed that the rules for each non-terminals become the same).

We address this problem by using stochastic adversarial rule sampling. Given the non-terminals, which effectively contain a number of potential 'futures', we learn an adversarial-based sampling, similar to GAN approaches (Goodfellow et al., 2014) , which learns to sample the most likely rules for the given input. The use of a discriminator network allows the model to generate realistic sequences that may not match the ground truth without being penalized.

We use the function G, which is the function modeling the learned grammar described above, as the generator function and build an additional discriminator function D. Following standard GAN training, the discriminator function returns a binary prediction which discriminates examples from the data distribution vs. generated ones. Note that the adversarial process is designed to ultimately generate terminals, i.e., the final output sequence for the model. D is defined as:

EQUATION

More specifically, D is tasked with the prediction of p ∈ {T rue, F alse} based on if the input sequence of terminals, t = t 0 t 1 t 2 . . . t L , is from the data or not (L is the length of the sequence). Note that our discriminator is also conditioned on the non-terminal sequence (n = n 0 n 1 n 2 . . . n L ), thus the distribution on non-terminals is learned implicitly, as well.

The discriminator function D is implemented as follows: given an input non-terminal and terminal sequence, we apply several 1D convolutional layers to the terminals and non-terminals, then concatenate their representations followed by a fully-connected layer to produce the binary prediction.

(Note that we also tried a GRU/LSTM instead of 1D conv, and it did not making a difference).

The discriminator and generator (grammar) functions are trained to work jointly, as is standard in GANs training. This constitutes the adversarial grammar. The optimization objective is defined as:

EQUATION

where p data (x) is the real data distribution (i.e., sequences of actions or human pose) and G(z) is the generated sequence from an initial state based on a sequence of frames (X).

The sequences generated by G could be compared to the ground truth to compute a loss during training (e.g., maximum likelihood estimation), however, doing so requires enumerating many possibilities in order learn multiple, distinct possible sequences. Without enumeration, the model converges to a mixture representing all possible sequences. By using the adversarial training of G, the model is able to generate sequences that match the distribution observed in the dataset. This allows for computationally feasible learning of longer, higher-dimensional sequences.

Architectures and implementation details. The functions G, f N and f t , f R , mentioned above, are networks using several fully-connected layers, which depend on the task and dataset (specific details are provided in the supplemental material). For pose, the function s is implemented as a two-layer GRU module (Cho et al., 2014) followed by a 1x1 convolutional layer with D N outputs to produce the starting non-terminal. For activity prediction, s is implemented as two sequential temporal convolutional layers which produce the starting non-terminal. The model is trained for 5000 iterations using gradient descent with momentum of 0.9 and the initial learning rate set to 0.1. We follow the cosine learning rate decay schedule. Our models were trained on a single V100 GPU. 

4. EXPERIMENTS:

We conduct experiments on two sets of problems for future prediction: future activity prediction and future 3D human pose prediction and three datasets. Our experiments demonstrate strong performance of the proposed approach over the state-of-the-art and the ability to produce multiple future outcomes, to handle multi-label datasets, and to predict further in the future than prior work.

4.1. ACTIVITY FORECASTING IN VIDEOS:

We first test the method for video activity anticipation, where the goal is to predict future activities at various time-horizons, using an initial video sequence as input. We predict future activities up to 45 seconds in the future on well-established video understanding datasets MultiTHUMOS (Yeung et al., 2015) for multi-class prediction and Charades (Sigurdsson et al., 2016) which is a multi-class and multi-label prediction task. We note that we predict much further into the future than prior approaches, that reported results within a second or several seconds (Yeung et al., 2015) .

Evaluation metric To evaluate the approaches, we use a standard evaluation metric: we predict the activities occurring T seconds in the future and compute the mean average precision (mAP) between the predictions and ground truth. As the grammar model is able to generate multiple, different future sequences, we also report the maximum mAP over 10 different future predictions. We compare the predictions at 1, 2, 5, 10, 20, 30 and 45 seconds into the future. As little work has explored longterm future activity prediction, we compare against four different baseline methods: (i) repeating the activity prediction of the last seen frame, (ii) using a fully connected layer to predict the next second (applied autoregressively), (iii) using a fully-connected layer to directly predict activities at various future times, and (iv) an LSTM applied autoregressively to future activity predictions.

MultiTHUMOS dataset. The MultiTHUMOS dataset (Yeung et al., 2015) is a popular video understanding benchmark with 400 videos spanning about 30 hours of video and 65 action classes.

Table 1 shows activity prediction accuracy for the MultiTHUMOS dataset. In the table, we denote our approach as 'Adversarial Grammar -max' but also report our approach when limited to generating a single outcome ('Adversarial Grammar'), to be consistent to previous methods which are not able to generate more than one outcome. We also compare to grammar without adversarial learning. As seen, our approach outperforms alternative methods including LSTMs. We observe that the gap to other approaches widens further in the future: 3.9 mean accuracy for the LSTM vs 11.2 of ours at 45 seconds in the future, as these autoregressive approaches become noisy. Due to the structure of the grammar model, we are able to generate better long-term predictions. We also find that by predicting multiple futures and taking the max improves performance, confirming that the grammar model is generating different sequences, some of which more closely match the ground truth.

Charades dataset. Charades (Sigurdsson et al., 2016 ) is a challenging video dataset containing longer-duration activities recorded in home environments. Charades is a multi-label dataset in which multiple activities often co-occur. We use it to demonstrate the ability to handle this complex data. It consists of 9858 videos (7990 training, 1868 test) over 157 activity classes. Table 2 shows the future activity prediction results for Charades. Similarly, we observe that the adversarial grammar model provides more accurate future prediction than previous work, slightly outperformed by grammar only. We note that Charades is more challenging than others on both recognition and prediction tasks, and that grammar only, while performing well here, is not feasible for high dimensional tasks.

Figure 2 shows a true sequence and several other sequences generated by the adversarial grammar. As Charades contains many different possible sequences, generating multiple futures is beneficial.

4.2. HUMAN POSE FORECASTING:

We further evaluate the approach on forecasting 3D human pose, a high dimensional structuredoutput problem. This is a challenging task (Jain et al., 2016; Fragkiadaki et al., 2015) but is of high importance, e.g., for motion planning in robotics. It also showcases the use of the Adversarial Grammar, as using the standard grammar is not feasible.

Human 3.6M dataset. We conduct experiments on a well established future pose prediction benchmark, the Human 3.6M dataset (Ionescu et al., 2014; Catalin Ionescu, 2011) , which has 3.6 million 3D human poses of 15 activities. The goal is to predict the future 3D locations of 32 joints in the human body. We use quaternions to represent each joint location, allowing for a more continuous joint representation space. We also predict differences, rather than absolute positions, which we found leads to more stable learning. Previous work demonstrated prediction results up to a second on this dataset. This work can generate future sequences for longer horizons, 4 seconds in the future.

We compare against the state-of-the-art methods on the Human 3.6M benchmark (Fragkiadaki et al., 2015; Jain et al., 2016; Ionescu et al., 2014; Martinez et al., 2017; Tang et al., 2018) using the Mean Angle Error (MAE) metric as introduced by Jain et al. (2016) . Table 3 shows average MAE for all (Fragkiadaki et al., 2015; Martinez et al., 2017; Tang et al., 2018) 

5. CONCLUSION:

We propose a novel differentiable adversarial grammar and apply it to several diverse future prediction and generation tasks. Because of the structure we impose for learning grammar-like rules for sequences and learning in adversarial fashion, we are able to generate multiple sequences that follow the distribution seen in data. Our work outperforms prior approaches on all tasks and is able to generate sequences much further in the future. We plan to release the code.

A APPENDIX:

A.1 IMPLEMENTATION DETAILS Activity Prediction For activity prediction, the number of non-terminals (|N |) was set to 64, the number of terminals (|Σ|) was set to the number of classes in the dataset (e.g., 65 in MultiTHUMOS and 157 in Charades). We used 4 rules for each non-terminal (a total of 256 rules). G, f N and f t each used one fully connected layer with sizes matching the desired inputs/outputs. s is implemented as a two sequential temporal convolutional layers with 512 channels.

3D Pose estimation For 3D pose, the number of non-terminals (|N |) was set to 1024, the number of terminals (|Σ|) was set to 1024, where each terminal has size of 128 (32 joints in 4D quaternion representation). The number of rules was set to 2 per non-terminal (a total of 2048 rules). G was composed of 2 fully connected layers, f N and f t each used three fully connected layers with sizes matching the desired inputs/outputs. s was implemented as a 2-layer GRU using a representation size of 1024.

A.2 SUPPLEMENTAL RESULTS:

Table 5 provides results of our approach for future 3D human pose prediction for all activities in the Human3.6M dataset. Figure 5 shows more examples of future predicted 3D pose at different timesteps.

Table 5 : Evaluation of future pose of our appoach for both short-term and long-term prediction horizons for all activities. Human3.6M benchmark. 

:



Figures and Tables:

Figure 1: Overview of the adversarial grammar model. The initial non-terminal is produced by an encoder based on some observations. The grammar then generates multiple possible sequences from the non-terminal. The generated and real sequences are used to train the discriminator.

Figure 2: Example video and activity sequence from Charades. At various times, we show multiple futures predicted by the grammar, some matching the true sequence and others very different.

Figure 3: Example results for 3D pose predictions. Top: walking, middle: greeting, bottom: posing.

Figure 5: Various predicted 3D pose sequences for walking, greeting, taking photos, sitting, posing.

Prediction mAP for future activities (higher is better) from 1 seconds to 45 seconds in the future on the MultiTHUMOS dataset.

Table content: <table><tr><td>Method</td><td colspan="7">1 sec 2 sec 5 sec 10 sec 20 sec 30 sec 45 sec</td></tr><tr><td>Random</td><td>2.6</td><td>2.6</td><td>2.6</td><td>2.6</td><td>2.6</td><td>2.6</td><td>2.6</td></tr><tr><td>Last Predicted Action</td><td colspan="3">16.5 16.0 15.1</td><td>12.7</td><td>8.7</td><td>5.8</td><td>5.9</td></tr><tr><td>FC Autoregressive</td><td>17.9</td><td>17</td><td>14.5</td><td>7.7</td><td>4.5</td><td>4.2</td><td>4.7</td></tr><tr><td>FC Direct</td><td>13.7</td><td>9.8</td><td>11.0</td><td>7.3</td><td>8.0</td><td>5.5</td><td>8.2</td></tr><tr><td>LSTM (Autoregressive)</td><td colspan="3">16.5 15.7 12.5</td><td>6.8</td><td>4.1</td><td>3.2</td><td>3.9</td></tr><tr><td>Grammar only (Ours)</td><td colspan="3">18.7 18.6 13.5</td><td>12.8</td><td>10.5</td><td>8.2</td><td>8.5</td></tr><tr><td>Adversarial Grammar (Ours)</td><td colspan="3">19.3 19.6 13.1</td><td>13.6</td><td>11.7</td><td>10.4</td><td>11.4</td></tr><tr><td colspan="4">Adversarial Grammar -max (Ours) 22.0 19.9 15.5</td><td>14.4</td><td>13.3</td><td>10.8</td><td>11.4</td></tr></table>

Prediction accuracy for future activities for 45 seconds in the future on the Charades dataset.

Table content: <table><tr><td>Method</td><td colspan="7">1 sec 2 sec 5 sec 10 sec 20 sec 30 sec 45 sec</td></tr><tr><td>Random</td><td>2.4</td><td>2.4</td><td>2.4</td><td>2.4</td><td>2.4</td><td>2.4</td><td>2.4</td></tr><tr><td>Last Predicted Action</td><td colspan="3">15.1 13.8 12.8</td><td>10.2</td><td>7.6</td><td>6.2</td><td>5.7</td></tr><tr><td>FC Autoregressive</td><td colspan="3">13.5 14.0 12.6</td><td>6.7</td><td>3.7</td><td>3.5</td><td>5.1</td></tr><tr><td>FC Direct</td><td colspan="3">15.2 14.5 12.2</td><td>9.1</td><td>6.6</td><td>6.5</td><td>5.5</td></tr><tr><td>LSTM (Autoregressive)</td><td colspan="3">12.6 12.7 12.4</td><td>10.8</td><td>7.0</td><td>6.1</td><td>5.4</td></tr><tr><td>Grammar only (Ours)</td><td colspan="3">15.7 14.8 12.9</td><td>11.2</td><td>8.5</td><td>6.6</td><td>8.5</td></tr><tr><td>Adversarial Grammar (Ours)</td><td colspan="3">15.9 15.0 13.1</td><td>10.5</td><td>7.4</td><td>6.2</td><td>8.8</td></tr><tr><td colspan="4">Adversarial Grammar -max (Ours) 17.0 15.9 13.4</td><td>10.7</td><td>7.8</td><td>7.2</td><td>9.8</td></tr></table>

Evaluation of future pose for short-term and long-term prediction horizons. Measured with Mean Angle Error (lower is better) on Human3.6M. No predictions beyond 1 second are available for prior work. Results are from(Fragkiadaki et al., 2015;Martinez et al., 2017;Tang et al., 2018)

Table content: <table><tr><td>Method</td><td colspan="6">80ms 160ms 320ms 560ms 640ms 720ms</td><td>1s</td><td>2s</td><td>3s</td><td>4s</td></tr><tr><td>ERD [1]</td><td>0.93</td><td>1.07</td><td>1.31</td><td>1.58</td><td>1.64</td><td>1.70</td><td>1.95</td><td>-</td><td>-</td><td>-</td></tr><tr><td>LSTM-3LR [1]</td><td>0.87</td><td>0.93</td><td>1.19</td><td>1.49</td><td>1.55</td><td>1.62</td><td>1.89</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Res-GRU [2]</td><td>0.40</td><td>0.72</td><td>1.09</td><td>1.45</td><td>1.52</td><td>1.59</td><td>1.89</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Zero-vel. [2]</td><td>0.40</td><td>0.71</td><td>1.07</td><td>1.42</td><td>1.50</td><td>1.57</td><td>1.85</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MHU-MSE [3]</td><td>0.39</td><td>0.69</td><td>1.04</td><td>1.40</td><td>1.49</td><td>1.57</td><td>1.89</td><td>-</td><td>-</td><td>-</td></tr><tr><td>MHU [3]</td><td>0.39</td><td>0.68</td><td>1.01</td><td>1.34</td><td>1.42</td><td>1.49</td><td>1.80</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Adv. Gram. (Ours)</td><td>0.36</td><td>0.65</td><td>0.98</td><td>1.27</td><td>1.40</td><td>1.49</td><td colspan="4">1.74 2.25 2.70 2.98</td></tr></table>

Evaluation of future pose for specific activity classes. Results are Mean Angle Error (lower is better). Results are from

Table content: <table/>

Starting from a neutral pose, the grammar is able to generate multiple difference sequences by selecting different rules. Top row: a walking sequence, middle: eating, bottom: sitting. activities compared to the state-of-the-art methods and Table4shows results on several activities, consistent with the protocol in prior work. As seen from the tables, our work outperforms all prior methods. Furthermore, we are able to generate results at larger time horizons of four seconds in the future. In Fig3, we show some predicted future poses for several different activities, confirming the results reflect the characteristics of the actual behaviors. In Fig.4, we show the grammar's ability to generate different sequences from a given starting state. Here, given a starting state, we select different rules, which lead to different sequences corresponding to walking, eating or sitting.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>.</td></tr><tr><td colspan="2">Human3.6M dataset.</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Methods</td><td colspan="2">Walking</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>80ms</td><td colspan="7">160ms 320ms 400ms 560ms 640ms 720ms 1000ms</td></tr><tr><td>ERD [1]</td><td/><td>0.77</td><td/><td>0.90</td><td>1.12</td><td>1.25</td><td>1.44</td><td>1.45</td><td>1.46</td><td>1.44</td></tr><tr><td>LSTM-3LR [1]</td><td/><td>0.73</td><td/><td>0.81</td><td>1.05</td><td>1.18</td><td>1.34</td><td>1.36</td><td>1.37</td><td>1.36</td></tr><tr><td>Res-GRU [2]</td><td/><td>0.27</td><td/><td>0.47</td><td>0.68</td><td>0.76</td><td>0.90</td><td>0.94</td><td>0.99</td><td>1.06</td></tr><tr><td>Zero-velocity [2]</td><td/><td>0.39</td><td/><td>0.68</td><td>0.99</td><td>1.15</td><td>1.35</td><td>1.37</td><td>1.37</td><td>1.32</td></tr><tr><td>MHU [3]</td><td/><td>0.32</td><td/><td>0.53</td><td>0.69</td><td>0.77</td><td>0.90</td><td>0.94</td><td>0.97</td><td>1.06</td></tr><tr><td>Ours</td><td/><td>0.25</td><td/><td>0.43</td><td>0.65</td><td>0.75</td><td>0.79</td><td>0.85</td><td>0.92</td><td>0.96</td></tr><tr><td>Methods</td><td colspan="2">Greeting</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>80ms</td><td colspan="7">160ms 320ms 400ms 560ms 640ms 720ms 1000ms</td></tr><tr><td>ERD [1]</td><td/><td>0.85</td><td/><td>1.09</td><td>1.45</td><td>1.64</td><td>1.93</td><td>1.89</td><td>1.92</td><td>1.98</td></tr><tr><td>LSTM-3LR [1]</td><td/><td>0.80</td><td/><td>0.99</td><td>1.37</td><td>1.54</td><td>1.81</td><td>1.76</td><td>1.79</td><td>1.85</td></tr><tr><td>Res-GRU [2]</td><td/><td>0.52</td><td/><td>0.86</td><td>1.30</td><td>1.47</td><td>1.78</td><td>1.75</td><td>1.82</td><td>1.96</td></tr><tr><td>Zero-velocity [2]</td><td/><td>0.54</td><td/><td>0.89</td><td>1.30</td><td>1.49</td><td>1.79</td><td>1.74</td><td>1.77</td><td>1.80</td></tr><tr><td>MHU [3]</td><td/><td>0.54</td><td/><td>0.87</td><td>1.27</td><td>1.45</td><td>1.75</td><td>1.71</td><td>1.74</td><td>1.87</td></tr><tr><td>Ours</td><td/><td>0.52</td><td/><td>0.86</td><td>1.26</td><td>1.45</td><td>1.58</td><td>1.69</td><td>1.72</td><td>1.79</td></tr><tr><td>Methods</td><td colspan="2">Taking photo</td><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>80ms</td><td colspan="7">160ms 320ms 400ms 560ms 640ms 720ms 1000ms</td></tr><tr><td>ERD [1]</td><td/><td>0.70</td><td/><td>0.78</td><td>0.97</td><td>1.09</td><td>1.20</td><td>1.23</td><td>1.27</td><td>1.37</td></tr><tr><td>LSTM-3LR [1]</td><td/><td>0.63</td><td/><td>0.64</td><td>0.86</td><td>0.98</td><td>1.09</td><td>1.13</td><td>1.17</td><td>1.30</td></tr><tr><td>Res-GRU [2]</td><td/><td>0.29</td><td/><td>0.58</td><td>0.90</td><td>1.04</td><td>1.17</td><td>1.23</td><td>1.29</td><td>1.47</td></tr><tr><td>Zero-velocity [2]</td><td/><td>0.25</td><td/><td>0.51</td><td>0.79</td><td>0.92</td><td>1.03</td><td>1.06</td><td>1.13</td><td>1.27</td></tr><tr><td>MHU [3]</td><td/><td>0.27</td><td/><td>0.54</td><td>0.84</td><td>0.96</td><td>1.04</td><td>1.08</td><td>1.14</td><td>1.35</td></tr><tr><td>Ours</td><td/><td>0.24</td><td/><td>0.50</td><td>0.76</td><td>0.89</td><td>0.95</td><td>1.08</td><td>1.15</td><td>1.24</td></tr><tr><td/><td/><td>160 ms Walking</td><td/><td>320 ms</td><td>560 ms</td><td>640 ms</td><td>720 ms</td><td/><td>1000 ms</td></tr><tr><td colspan="2">Starting Pose</td><td>160 ms Eating</td><td/><td>320 ms</td><td>560 ms</td><td>640 ms</td><td>720 ms</td><td/><td>1000 ms</td></tr><tr><td/><td/><td colspan="2">160 ms Sitting Down</td><td>320 ms</td><td>560 ms</td><td>640 ms</td><td>720 ms</td><td/><td>1000 ms</td></tr><tr><td>Figure 4:</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Peer review (iclr2020):
Summary: The paper introduces a probabilistic grammar formulation for the task of predicting labels of future timesteps. The grammar is simplified to only contain productions of the form A -> Bc with nonterminals A and B and terminal c. The current nonterminal encodes the current ‘state’ and the current terminal encodes the label of the current timestep. The production encodes the transition from one state to the next as time proceeds. The grammar is formulated using a number of neural feed-forward networks in order to make the rules, nonterminals, and terminals all differentiable and thus learnable via backpropagation. Since the grammar can generate samples, it can be (and is) trained in a generative adversarial framework where a learned discriminator D must determine if a pair of sequences of nonterminals and terminals are generated by the model or from the data. The model is evaluated against baselines on two activity-forecasting / -recognition datasets and outperforms all baselines. It is also evaluated on a pose forecasting dataset and outperforms a number of prior works here, particularly at longer time horizons. Overall: Reject. While I think there are merits to the approach, the explanation of the model was opaque and did not sufficiently compare and contrast to existing work both in the model definition and in the experimental evaluation. In particular, from my understanding of this model (made challenging by the explanation) and the limited probabilistic grammar that it encodes (only A -> Bc), it seems nearly identical to a (neural) hidden Markov model (i.e., a hidden Markov model that uses deep networks to parameterize the conditional distributions, e.g., [1]). Second, empirically, from a quick google search, other papers have published results on both the MultiTHUMOS (e.g., [2]) and Charades datasets (for the latter, there was even a competition [3]), none of which are mentioned here. Clarity: Well written but not particularly clear. Model section is somewhat confusing and needs clarification, and the experiment section is missing details on the variants of the proposed model. It was also difficult to determine the specific way that adversarial training was performed; the explanation given was of high-level adversarial training but lacking sufficient specifics to reimplement. Significance: Potentially useful but quite limited by the above-mentioned issues. I am not an expert in this application domain, however. Detailed comments: Section 1. The introduction should better clarify what is meant by ‘adversarial sampling’. I was uncertain whether this meant a GAN-like training procedure or something else entirely until later sections. Section 3. - The model, as far as I can tell, does not learn transition between continuous events in time. It is given a sequence of discrete timesteps and learns discrete labels for those. Specifying ‘continuous events in time’ is misleading. Section 3.1. - It would be helpful to include an example of a nonterminal, terminal, and production at the beginning when the grammar is defined. - The grammar defined is non-probabilistic; however, the grammar used later is probabilistic. The definition should match the actual model. - Is a nonterminal a vector in R^D? And a terminal a vector in R^|Sigma|? This should be said explicitly. - r is used to denote both a “set of rules” and later “a specific rule”. Please use different notation or at least boldface to distinguish these. - Further, r is said to be a “set of rules” but then said to be a vector whose elements specify the probability of a set of rules. It is one or the other, not both. Does the latter mean that there’s a learned matrix (or tensor) of rule vectors that the probability vector indexes? Or are the rules themselves just indicators in a vector and this is the probability of each one? Please make this more clear. - The notation G(A) = { (f_N(f_R(A)), f_T(f_R(A))) } produces a set containing one pair containing two lists, which does not match the prior notation of {(B_i, t_i)} = G(A). - This whole section would be made much more clear by explicitly defining the vectors as members of R and the mappings defined by the functions. Currently, it is quite confusing and I had to go over it many times to try to understand what exactly was being proposed. - Does the model not take in images after the encoding of the initial frames? Is it simply used to roll out future labels? This seems like a missed opportunity for a more useful model. Section 3.2. - This reads as a high-level description of applying GAN-style training to the proposed approach but does not provide the reader with enough information to know how this was actually done. For example, were sequences subsampled? What lengths of sequences were used? Was anything done to prevent overfitting, which can be quite problematic in these setups (e.g., [4])? This is unfortunate because I think that this training approach could be a useful contribution, as previous methods for training these styles of model rely on variational inference or other complex methods. Section 4. - I was glad to see some minor ablations of the grammar without the adversarial training in the experiments. However, I could not find an explanation of how it was trained if not using an adversarial method. A better description of the variants of the proposed approach should be added. [1] Structured Inference Networks for Nonlinear State Space Models. Krishnan, Shalit, and Sontag (2016). [2] https://paperswithcode.com/sota/action-detection-on-multi-thumos [3] http://vuchallenge.org/charades.html [4] Task-Relevant Adversarial Imitation Learning. Zolna et al. (2019).