Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes two new modules to improve the motion generation capabilities of auto-regressive networks. The designing of modules lies in the idea of using a bank of learned features, which are used to reduce the noise in prediction motions. Both modules were tested in the task of dance motion generation. The experimental results in the AIST++ dataset show the benefits of using the proposed modules. It is worth mentioning that the modules seem to be very useful in other applications where motion generation is required.
strengths_and_weaknesses: - Strengths: 
    - I enjoyed reading the paper, which is well written, and structured. It is easy to read, understand, and the tables and figures are illustrative.
    - Incorporating representation banks with learned features to leverage the learning of motion generation is a very interesting and well-formulated idea. 
    - The experiments are done thoroughly. The user study, the experimental results with the baselines, and the ablation study are in great detail and defend the finding well.


- Weakness:
    - I have two minor concerns. First refers to the lack of motivation in the designing choices, such as the stage-wise strategy to train the network. Clear seems to be a good choice, but why is that in the case of dance motion generation.
    - Another possible weakness would be the limitation of requiring training the features bank for each genre. This limitation and designing decision could be better discussed in the paper since it limits the practical use of the approach.

In conclusion, the proposed method is very interesting, and I believe this is a solid paper that deserves to be communicated.

questions: 1. Have you tried to train the network without applying the stage-wise strategy? What would be the performance of the methods?
2. For the sake of reproducibility, kindly consider explicitly saying the list of audio features used to train the network.
3. After training and creating each feature bank, are the weight frozen when training the whole network?
4. Regarding the use of one bank for each genre. Have you tried to use banks trained with all genres? I think the results in this scenario could bring interesting insights to the community.
5. Another question related to the one bank for each genre. Why not condition the bank to the music style?

limitations: Yes. The limitations and broad impact are well discussed in the conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission specifically focuses on preventing the generation of frozen motions in long-term motion generation task.
To achieve this goal, the latent code (z^hat) used to generate the final motion is represented as the linear combination of codes in a learned codebook (termed as feature bank in this submission) as VQ-VAE [29]. And the weight of every code (in the codebook) is determined by the similarity between the original latent code (z) and the code.
And a TransitBank is proposed to "retrive" future motion features according to the past motion features.
strengths_and_weaknesses: **Major weaknesses**

- The description is not clear enough and I could not understand the exact operations of the proposed method.

L94, L156: cite the VQ-VAE [29] paper. What is the difference between the proposed RefineBank and VQ-VAE? This is very important and should be discussed as thorough as possible. Maybe you use the weighted sum of the codes while they use a certain code? Personally, I do not think this is a good choice. Pros and cons of these two choices should be evaluated. BTW, why do you need to do a clustering in L143-144? And what is the E_M, is it the same as E_RB?

Since there is an average operation when calculating b^i (L145), so I guess the network extract a latent code for every frame (but with a temporal receptive field), right? Then what are the Ks/Vs in the past-future bank, I did not find the T dimension but it's said that "TransitBank is a temporal extension of RefineBank" in L139? Frankly, I did understand the exact use and learning related to TransitBank (Sec. 4.2 & 4.3). Many operations need to be clarified.

BTW, try not to name both network module (RefineBank) and tensor (manifold bank) as banks. It is a little confusing. Personally, I would recommend using code (codebook) or word (vocabulary) to refer to the latent feature (the collection of the latent features).

How does the music feature (Fig. 1) affect the final output? Is it working? Some dedicated experiments should be conducted about this part.

Why does FACT results look that bad even for the first few seconds?

L119: "basic short motion segments": What are they and how do you find them?

User study: Are there any professionals conducting the user study (L220)? Better in what aspect (L222)? Normally, people will compare several specific and clearly described aspects in a user study.

**Minor weaknesses**

L175: How many sec is 240 frames of music?

L197: differences between what and what?

L212: What do you mean by "stationary poses at transition moments"?

Fig. 6:How is GT looks like?

What are prediction-based methods (L57), retrieval-based methods (L68), and bank-based auto-encoder (L249) exactly? These are not well-known concept yet.
questions: The novelty; More thorough evaluation and analysis; Much improved writing.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors look into the problem of freezing motions in dance generation frameworks.  They propose RefineBank and TransitBank modules that can be plugged on existing methods to generate high fidelity dance movements without freezing after a few frames. Refine Bank learns a low dimensional manifold representaton of motion data close to the ground truth to prevent error accumulation. TransitBank learns a motion prior for the future poses to reduce ambiguity in long-term motion prediction.The authors evaluated their method on the AIST++ dataset and outperform certain baselines. The authors also propose a new metric which measues rate of freezing sub-sequences in movements.
strengths_and_weaknesses: Strengths:
- The authors tackle a challenging problem for long-term dance motion generation and synthesize diverse and natural dance movements with respect to music. The presented qualitative results and user study results indicate the proposed approach is promising.
- Using the bank-based auto-encoder modules in the architecture to learn a compact motion manifold is technically sound.  
- The authors provide ablations for various experiments to strengthen their claims. 
- The paper is overall clearly written and easy to follow.

Weaknesses:
- I am not clear why the three-stage optimization is required. Why is the Manifold Bank trained separately from the Refine Bank encoder and decoder?
- How are the different loss terms being used in the first stage of optimization (Eq. 7)? Ablations for the use of multiple losses are missing.
questions: - How is the commitment loss different from the item loss (Eq. 7)?
- How is the Win Rate interpreted in the last column of Table 1?
limitations: - Since this is an incremental work on the existing architecture of FACT [18], it is heavily reliant on the performance of the baseline FACT model. It is unclear if or how the proposed modules can be combined with other baseline models such as [10, 17, 38].

- It is also unclear if the proposed modules can be generalized to any genre of dancing.

- Qualitative results also produce some self-intersections of the limbs, which is problematic for such synthesis problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: A common failure mode of human motion or dance generation models (which aim to synthesize parametrized human pose sequences, possibly conditioned on music), is that when attempting to predict long sequences of motion, they can freeze. Essentially, due to accumulated prediction uncertainty, the model eventually decides to predict a static, non-moving pose, as a way to minimize prediction error. This has been raised in a number of papers, and there are a few ideas on how to solve it, but no clear definition or metric exists.

The authors address this by adapting the method, dataset, and evaluation from the FACT [18] paper, and adding two modules designed to eliminate freezing, as well as proposing a new freezing metric.

The modules, RefineBank, and TransitBank, are based on attention-style lookups from learned memory banks. Specifically, RefineBank is designed to take a predicted motion segment, and clean it up by projecting it to and from a low-dimensional motion manifold. Essentially use a transformer encoder to the pose input sequence, do memory lookup for each pose from this learned pose bank, then decode the sequence. TransitBank uses a similar technique to propose extensions of an initial pose sequence based on a learned memory bank that maps motion prefixes to suffixes.

The new freezing metric is based on how often the pose delta or global translation delta between frames is less than a threshold determined by analyzing the ground truth data. The proposed model improves upon FACT in standard FID, Diversity, and beat alignment methods, while doing significantly better at presenting freezing, and is preferred in a human quality assessment study.


strengths_and_weaknesses: **Quality & Clarity:** The paper is clearly written and easy to follow. In particular, the diagrams in Figures 1-4 were well done, and very helpful for understanding sections 4.1-4.3. The explanations are sound, with two exceptions detailed below.

Line 84: SPML should be cited and explained. Also the phrase "Since shapes" should be explained more clearly, to convey that it's the shape parameters of the individual dancers you're discarding. 

Line 182: "We construct a separate manifold bank and past-future bank for each dance genre." This implies that, at test time, the model is informed which dance genre should be used for generation. This is strictly more information than is used in FACT [18], potentially making the problem easier. The need for per-genre memory banks and the quantitative advantage obtained by doing so should be addressed further.

Lines 210-213 "Interestingly, the ∆ Trans of our method is even larger than that of GT. ..." This explanation highlights a problem with the definition of the delta trans, delta pose, and Freeze metrics. The goal shouldn't be to maximize delta pose & delta trans, and minimize Freeze... instead it should be for the predicted model to hit the same values for each sequence as observed in the ground truth.

Furthermore, I cannot square the claim that "it is difficult for learning-based methods to predict stationary poses due to the lack of sufficient data" with the observation that the "GT often has stationary poses at transition moments". This suggests that there *is* sufficient stationary data, and the models are not capturing it, or specifically designed to never freeze even though occasional freezes are desirable.

**Originality & Significance:** This paper is an extension of the FACT model from [18] with a small enhancement based on relatively known techniques (transformer memory) and new evaluation metric, applied to the same AIST++ dataset. As such the originality and significance is very minor. Perhaps if the memory bank methods were demonstrated to be more widely useful across a variety of human motion tasks or models, the paper would be more significant. As is, this does not meet the bar for publication based on significance and originality.

questions: Table 1 copies the numbers from [18] Table 2, but misreports two numbers: for FACT, FIDg is 12.40 and Distg is 5.30. Using the numbers from [18], "Ours" would no longer surpass the original FACT on FIDg. Could the authors please explain?

How does the proposed method for presenting freezing interact with FACT's "future-n full-attention" scheme, also designed to prevent freezing? Do RefineBank & TransitBank eliminate the need for future-n full-attention?

limitations: yes
review_title: null

Meta-review: metareview: This paper proposes to fix common issues in motion generation by representing motion latents as combinations of discrete latent codewords learned by a VQ-VAE style approach. The idea is interesting and novel, and in the response period has been shown to potentially work for more than just dance generation, including a human trajectory prediction task.

While the idea might be quite well-motivated, reviewers all agreed that the writing does not quite do it justice in the submitted draft. I would recommend that the authors work on addressing those relevant reviewer comments for the next iteration (either camera-ready / future submission), aside from incorporating the new results into the draft.
confidence: Certain
Decision: Accept