Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes acLSTM to synthesize long sequences of human motion. It tackles the challenge of error accumulation of traditional techniques to predict long sequences step by step. The key idea is to combine prediction and ground truth in training. It is impressive that this architecture can predict hundreds of frames without major artifacts.

The exposition is mostly clear. My only suggestion is to use either time (seconds) or frame number consistently. In the text, the paper sometimes use time, and other time uses frame index (e.g. figure 7 and its caption). It confuses me a bit since it is not immediate clear what the frame rate is.

In evaluation, I think that it is important to analyze the effect of condition length in the main text, not in the Appendix. To me, this is the most important quantitive evaluation that give me the insight of acLSTM. It also gives a practical guidance to readers how to tune the condition length. As indicated in Appendix B, "Further experiments need to be conducted to say anything meaningful." I really hope that in the next version of this paper, a detailed analysis about condition length could be added. 

In summary, I like the method proposed in the paper. The result is impressive. I have not seen an LSTM based architecture predicting a complex motion sequence for that long. However, more detailed analysis about condition length is needed to make this paper complete and more valuable.
review_title: A useful technique to avoid prediction error accumulation

final_score: rating: 6: Marginally above acceptance threshold
review_content: Paper presents an approach for conditional human (skeleton) motion generation using a form of the LSTM, called auto-conditioned LSTM (acLSTM). The key difference of acLSTM is that in it parts of the generated sequences, at regular intervals, are conditioned on generated data (as opposed to just ground truth data). In this way, it is claimed that acLSTM can anticipate and correct wrong predictions better than traditional LSTM models that only condition generation on ground truth when training. It is shown that trained models are more accurate at long-term prediction (while being a bit less accurate in short-term prediction). 

Generally the idea is very sensible. The novelty is somewhat small, given the fact that a number of other methods have been proposed to address the explored challenge in other domains. The cited paper by Bengio et al., 2015 is among such, but by no means the only one. For example, “Professor Forcing: A New Algorithm  for Training Recurrent Nets” by Goyal et al. is a more recent variant that does away with the bias that the scheduled sampling of Bengio et al., 2015 would introduce. The lack of comparison to these different methods of training RNNs/LSTMs with generated or mixture of ground truth and generated data is the biggest shortcoming of the paper. That said, the results appear to be quite good in practice, as compared to other state-of-the-art methods that do not use such methods to train. 

Other comments and corrections:

- The discussion about the issues addressed not arising in NLP is in fact wrong. These issues are prevalent in training of any RNN/LSTM model. In particular, similar approaches have been used in the latest image captioning literature.

- In the text, when describing Figure 1, unrolling of u=v=1 is mentioned. This is incorrect; u=v=4 in the figure.

- Daniel Holden reference should not contain et. al. (page 9)
review_title: Well written paper with somewhat limited novelty but state-of-the-art results

final_score: rating: 7: Good paper, accept
review_content: The problem of learning auto-regressive (data-driven) human motion models that have long-term stability
is of ongoing interest. Steady progress is being made on this problem, and this paper adds to that.
The paper is clearly written. The specific form of training (a fixed number of self-conditioned predictions,
followed by a fixed number of ground-truth conditioned steps) is interesting for simplicity and its efficacy.
The biggest open question for me is how it would compare to the equally simple stochastic version proposed
by the scheduled sampling approach of [Bengio et al. 2015].

PROS:  The paper provides a simple solution to a problem of interest to many.
CONS:  It is not clear if it improves over something like scheduled sampling, which is a stochastic predecessor
       of the main idea introduced here. The "duration of stability" is a less interesting goal than
       actually matching the distribution of the input data.

The need to pay attention to the distribution-mismatch problem for sequence prediction problems
has been known for a while. In particular, the DAGGER (see below) and scheduled sampling algorithms (already cited) 
target this issue, in addition to the addition of progressively increasing amounts of noise during training
(Fragkiadaki et al). Also see papers below on Professor Forcing, as well as "Learning Human Motion Models
for Long-term Predictions" (concurrent work?), which uses annealing over dropout rates to achieve stable long-term predictions.

  DAGGER algorithm (2011):  http://www.jmlr.org/proceedings/papers/v15/ross11a/ross11a.pdf
  "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning"

  Professor Forcing (NIPS 2016)
  http://papers.nips.cc/paper/6099-professor-forcing-a-new-algorithm-for-training-recurrent-networks.pdf

  Learning Human Motion Models for Long-term Predictions (2017)
  https://arxiv.org/abs/1704.02827
  https://www.youtube.com/watch?v=PgJ2kZR9V5w
  
While the motions do not freeze, do the synthesized motion distributions match the actual data distributions?
This is not clear, and would be relatively simple to evaluate.  Is the motion generation fully deterministic?
It would be useful to have probabilistic transition distributions that match those seen in the data.
An interesting open issue (in motion, but also of course NLP domains) is that of how to best evaulate
sequence-prediction models.  The duration of "stable prediction" does not directly capture the motion quality. 

Figure 1:  Suggest to make u != v for the purposes of clarity, so that they can be more easily distinguished.

Data representation:
Why not factor out the facing angle, i.e., rotation about the vertical axis, as done by Holden et al, and in a variety of
previous work in general?
The representation is already made translation invariant. Relatedly, in the Training section,
data augmentation includes translating the sequence: "rotate and translate the sequence randomly".
Why bother with the translation if the representation itself is already translation invariant?

The video illustrates motions with and without "foot alignment".
However, no motivation or description of "foot alignment" is given in the paper.

The following comment need not be given much weight in terms of evaluation of the paper, given that the
current paper does not use simulation-based methods. However, it is included for completeness.
The survey of simulation-based methods for modeling human motions is not representative of the body of work in this area
over the past 25 years.  It may be more useful to reference a survey, such as 
"Interactive Character Animation Using Simulated Physics: A State‐of‐the‐Art Review" (2012)
An example of recent SOTA work for modeling dynamic motions from motion capture, including many
highly dynamic motions, is "Guided Learning of Control Graphs for Physics-Based Characters" (2016)
More recent work includes "Learning human behaviors from motion capture by adversarial imitation", 
"Robust Imitation of Diverse Behaviors", and "Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning", all of which demonstrate imitation of various motion styles to various degrees.

It is worthwhile acknowledging that the synthesized motions are still low quality, particular when rendered with more human-like looking models, and readily distinguishable from the original motions.  In this sense, they are not comparable to the quality of results demonstrated in recent works by Holden et al. or some other recent works.  However, the authors should be given credit for including some results with fully rendered characters, which much more readily exposes motion flaws.

The followup work on [Lee et al 2010 "Motion Fields"] is quite relevant:
"Continuous character control with low-dimensional embeddings"
In terms of usefulness, being able to provide some control over the motion output is a more interesting problem than
being able to generate long uncontrolled sequences.  A caveat is that the methods are not applied to large datasets.

review_title: Review:   generally positive; some reservations

Meta-review: This paper proposes a real-time method for synthesizing human motion of highly complex styles. The key concern raised by R2 was that the method did not depart greatly from a standard LSTM: parts of the generated sequences are conditioned on generated data as opposed to ground truth data. However, the reviewer thought the idea was sensible and the results were very good in practice. R1 also agreed that the results were very good and asked for a more detailed analysis of conditioning length and some clarification. R3 brought up similarities to Professor Forcing (Goyal et al. 2016) -- also noted by R2 -- and Learning Human Motion Models for Long-term Predictions (Ghosh et al. 2017) -- noting not peer-reviewed. R3 also raised the open issue of how to best evaluate sequence prediction models like these. They brought up an interesting point, which was that the synthesized motions were low quality compared to recent works by Holden et al., however, they acknowledged that by rendering the characters this exposed the motion flaws. The authors responded to all of the reviews, committing to a comparison to Scheduled Sampling, though a comparison to Professor Forcing was proving difficult in the review timeline. While this paper may not receive the highest novelty score, I agree with the reviewers that it has merit. It is well written, has clear and reasonably thorough experiments, and the results are indeed good.
Decision: Accept (Poster)