EGGesture: Entropy-Guided Vector Quantized Variational AutoEncoder for Co-Speech Gesture Generation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 21:03:47

Abstract: Co-Speech gesture generation encounters challenges with imbalanced, long-tailed gesture distributions. While recent methods typically address this by employing Vector Quantized Variational Autoencoder (VQ-VAE), encode gestures into a codebook and classify codebook indices based on audio or text cues. However, due to the imbalanced, the codebook classification tends to bias towards majority gestures, neglecting semantically rich minority gestures. To address this, this paper proposes the Entropy-Guided Co-Speech Gesture Generation (EGGesture). EGGesture leverages an Entropy-Guided VQ-VAE to jointly optimize the distribution of codebook indices and adjust loss weights for codebook index classification, which consists of a) A differentiable approach for entropy computation using Gumbel-Softmax and cosine similarity, facilitating online codebook distribution optimization, and b) a strategy that utilizes computed codebook entropy to collaboratively guide the classification loss weighting. These designs enable the dynamic refinement of the codebook utilization, striking a balance between the quality of the learned gesture representation and the accuracy of the classification phase. Experiments on the Trinity and BEAT datasets demonstrate EGGesture’s state-of-the-art performance both qualitatively and quantitatively. The code and video are available.

Main paper: 
 related work 2.1 co-speech gesture generation: 
 Pioneering deep-learning literature on Co-Speech gesture generation primarily focused on generative model architectures. Initial efforts revolved around end-to-end architectures, such as GRUs [10, 40] , enriched by the integration of GANs [11, 36] , FLow [1, 16] , VAE [23, 29] or Diffifusion [37, 44] . Typically, these techniques decoded gestures by regressing them onto joint rotations or offsets. Recent advanced methods are based on VQ-VAE, during this phase, Rhythmic Gesticulator [2] first employed the VQ-VAE to encode the wordlevel duration-normalized gestures, and then learned the mapping from audio and text cues to clustered class indices. Concurrently, Talkshow [39] leveraged the VQ-VAE architecture to achieve the holistic gestures generation in two phases. QPGesture [38] combined the VQ-VAE and learned motion phases to guide the gesture generation. More recently, DGGesture [30] and GestureDiffuClip [3] also set their baselines using a VQ-VAE based two phase training framework. Different with the above methods, our method explicitly tackles the imbalanced distribution present in the VQ-VAE's codebook. Most close to our topic, DisCo [26] discovered that motions follow a long-tail distribution and suggested using motion words to categorize these motions into rhythm and content codes. It addressed the long-tail problem by employing motion word-based clustering using positional distance, the accuracy of clustering by motion words is compromised as it is based on positional distance, leading to clusters that consider only adjacent or identical positionlevel motion clips. Different with DisCo, our method innovatively utilizes a neural-network-based clustering algorithm within a VQ-VAE, enabling deeper and more distinct feature learning for motion categorization, enhancing the overall generated gestures quality and utilization of the codebooks.Vector Quantized Variational AutoEncoders (VQ-VAE) [35] , was first proposed in 2016. It has been gradually adopted by modern deep learning models [42] , particularly in the domain of image representation learning [5] . VQ-VAE is designed to quantize the latent space while facilitating partial gradient backward operations. Here, each codebook entry is treated as a token or cluster that represents the original features. Recent advancements in VQ-VAE encompass hierarchical coarse-to-fine codebook learning [32] , along with the amalgamation of VQ-VAE tokens with masked representation learning [18, 24] . Moreover, while existing works [33, 34] utilize offline codebook metrics for evaluating VQ-VAE performance, in contrast,Gestures 𝑮 :The first …. The first …. we propose an online optimization approach to maximize codebook utilization.Addressing imbalances in machine learning has been widely studied within image classification domains [7, 17, 22, 43] . Solutions are typically categorized into either resampling or reweighting methods. Resampling strategies, e.g., upsampling minority classes [14] or downsampling majority ones [6] , aim to achieve a balanced class distribution. However, they face challenges when multiple classes are intertwined within a single sample, e.g., distinct frames in video sequences correspond to various classes. Reweighting methods offer greater flexibility by assigning varied weights to different frames. Moving beyond mere linear weighting, adaptive strategies e.g., focal loss [25] can further mitigate the influence of imbalanced datasets. However, in the context of VQ-VAE-based co-speech gesture generation, simply reweighing the data cannot effectively optimize codebook utilization. 
entropy-guided gesture generation: 
 Our methodology builds upon the vanilla motion VQ-VAE, as shown in Figure 3 . It is divided into two modules: the codebook learning module and the indices classification module. We first present a prior of vanilla motion VQ-VAE. Following that, we detail our entropy computation techniques and the entropy-guided approach for codebook and classification learning. Overall the pipeline leverages raw audio waves A, text T, seed gestures G 𝑠𝑒𝑒𝑑 to produce full 6D gestures rotations denoted as Ĝ. To maintain a consistent frame rate with codebook tokens ẑ𝑔 , we interpolate both the audio features F 𝐴 and text features F 𝑇 .We begin by the training of vanilla gesture motion VQ-VAE. This involves a gesture encoder E 𝑔 , a codebook C 𝑔 , and a gesture decoder D 𝑔 . The codebook functions as a repository of learned parameters, composed of integer indices i 𝑔 and associated values ẑ𝑔 . The gesture encoder, E 𝑔 , first encodes the gesture into a latent vector z 𝑔 . Following this, the distance between each latent vector and its affiliated values is calculated via the distance function H . the index of the closest ẑ𝑔 is determined through the argmin of distances, which is non-differentiable,EQUATIONthen we could sample the closest ẑ𝑔 with the index i 𝑔 , and the decoder will leverage the latent ẑ𝑔 to reconstruct the gestures G.Due to the non-differentiable argmin operation, the latent code z 𝑔 do not have gradients from the gesture reconstruction, where reconstruction loss is given as,EQUATIONTo address this, VQ-VAE propagates the gradient from ẑ𝑔 to z 𝑔 by copying the gradient from the part where they coincide. Specifically,EQUATIONwhere sp denotes the stop-gradient operation. The gradient of the first element from the codebook 𝑎 ĝ will then be used to optimize the encoder.Finally, an additional loss term is introduced to minimize the discrepancy between the codebook value ẑ𝑔 and the encoded latent z 𝑔 , EQUATIONThe entropy is computed to evaluate the utilization of the codebook. We calculate the similarity between the encoded latent z 𝑔 and the codebook latent ẑ𝑔 , followed by the Gumbel softmax operation for a soft assignment of latent z 𝑔 to the corresponding ẑ𝑔 . This is represented as,EQUATIONwhere • denotes the matrix multiplication. The probability, p, is given by the Gumbel softmax function instead of the vanilla softmax to simulate the uncertainty of codebook index during training,EQUATIONwhere: p 𝑖 is the resulting probability distribution, i.e., the "soft" onehot encoded representation, n 𝑖 are i.i.d samples from the 𝐺𝑢𝑚𝑏𝑒𝑙 (0, 1) distribution, 𝜏 is the temperature parameter. As 𝜏 approaches 0, the Gumbel softmax operation becomes the standard discrete softmax, and as 𝜏 increases, the distribution becomes more uniform. Sampling from the 𝐺𝑢𝑚𝑏𝑒𝑙 (0, 1) distribution can be done using the inverse transform sampling,EQUATIONwhere u is a sample from the uniform distribution 𝑈 (0, 1).The Cosine similarity and Gumbel softmax is differentiable, allowing for gradient-based optimization. For each batch, the utilization is calculated from the average probability for each class. Inspired by batch normalization [19] , a moving average of the utilization for each class is maintained to ensure stable training. This is represented asEQUATIONsubsequently, the entropy loss is calculated as,EQUATIONThe entropy loss promotes an uniform probability indices distribution across the codebook, benefiting both the maximizing in utilization and balancing the codebook indices distribution. According to the mathematical theory, the entropy 𝐸 reaches its maximum value when all 𝑝 𝑡 are equal, i.e., 𝑝 𝑡 = 1 𝑁 for all 𝑖. The 𝐸 is calculated from a differentiable path by the cosine similarity 𝑠, the gradient could be backward to adjust the weight of motion encoder for a more balanced z 𝑔 ).We then leverage class probability statistics for jointly training the classifier. For a given training epoch 𝑒, we minimize the distance between the predicted gesture index î𝑔,𝑒 (from audio and text) and the real codebook index sp(i 𝑔,𝑒 ). In addition to the vanilla negative log likelihood (NLL) loss, the distance is reweighted based on the inverse class-level probability, where w 𝑘 = 1 p 𝑘 . The classifier loss is then given by,EQUATION)where 𝑛 is the number of samples in each batch and 𝑘 is the size of the codebook.To mitigate the impact of the codebook indices randomness in the early training epochs, we linearly combine the clustering-related and classification-related terms. Our overall training objective is:EQUATIONwhere 𝛾 are exponential set scaling factor.Our pipeline integrates transformer-based pretrained audio and text encoders, Wav2Vec2 [4] and BERT [8] . Their parameters are freezed for faster training. Additionally, we refine audio and text features through the implementation of dedicated transformer-based encoders [8] .Inspired by the state-of-the-art motion VQ-VAE architecture in TM2T [12] , we adopt a 1D CNN based ResNet [15] architecture to encode gestures into latent vectors with a quarter of the frame rate. While gesture decoding is achieved via a combination of upsampling and 1D CNN layers. As a main component, a transformer decoder is adopt as our sequential classifier. The decoder leverages the positional embedding and seed codebook indices as inputs, and employs the cross-attention operations on concatenated audio and text features for the final motion indices classification. 
experiments 4.1 settings: 
 4.1.1 Datasets. We evaluate our method on two benchmark motioncaptured gesture datasets: Trinity [9] and BEAT [27] . Trinity provides 244 minutes of motion-captured gestures from a single male actor, encompassing diverse conversational topics such as hobbies and sports. BEAT offers around 76 hours gestures from 30 speakers, we use the English subset of BEAT2, which moshed the skeleton level data into the SMPLX [31] , facilitating consistent mesh-level visualizations. We leverage the speaker-2's data from BEAT2, including 4 hours speech and conversational data.4.1.2 Baselines. We benchmark our approach against a comprehensive set of both seminal and state-of-the-art methods on the Trinity and BEAT datasets. Our comparison includes methods speech2gestures [11] , audio2gestures [23] , moglow [1] , trimodal [40] , disco [26] , camn [27] , ha2g [28] , qpgesture [38] , and talkshow [39] . Using publicly available codes, we reproduce results for camn, ha2g, qpgesture, and talkshow on Trinity. Specifically, we adapt the camn method by excluding the emotional, speaker ID, and facial encoder modules due to the absence of corresponding modalities in Trinity. We reproduce the performance of disco, ha2g, and talkshow on BEAT. For the remaining terms, we reference objective scores directly from their papers. 4.1.4 Metrics. We utilize three objective evaluation metrics: FID (Fréchet Inception Distance) [40] , BA (Beat Alignment) [28] , and L1Div (L1 Diversity) [23] . FID computes the distance between two distributions based on the discrepancy in their means and covariances,EQUATION)where 𝜇 𝑟 and Σ 𝑟 are the first and second moments of the latent features distribution 𝑧 𝑟 of real gestures g, and 𝜇 𝑝 and Σ 𝑝 are the first and second moment of the latent features distribution 𝑧 𝑝 of generated gestures ĝ. We pretrained a CNN-based gesture autoencoder on both the BEAT and Trinity datasets. BA quantifies the alignment between gesture and audio beats. Gesture beats are determined from the local minima of the gesture curve, while audio beats are discerned using onset detection,EQUATION)where 𝐺, 𝐴 is the set of gesture beat and audio beat, respectively. The final score is the average beat alignment across all joints. L1Div calculates the average L1 distance between two randomly chosen We calculate the win rate for evaluation (in percentage). Our method shows higher participants' overall preference, and also outperforms state-of-the-art methods in diversity. The error bar is calculated by standard deviation. Pre., BA, Div. are Preference, Beat Alignment and Diversity, respectively.gesture sequences of equal length within a specified group.EQUATIONwhere 𝑟 𝑡 is the rotation of joints in frame 𝑡, prior works have set this group size to 40, we follow this in our evaluations.In Table 1 and Table 2 , we present objective evaluation metrics for both the Trinity and BEAT datasets. Results show our approach outperforms pervious state-of-the-art methods in terms of FID, BA, and Diversity metrics, establishing a new state-of-the-art performance. Notably, our method demonstrates a more pronounced improvement in Diversity and FID compared to Beat Alignment. This is attributed to the enhanced recall of minority classes, which facilitates the generation of more diverse gestures and achieves a distribution closer to 1 Video results are available in supplementary materials. In Figure 5 , we present detailed visualization of the synthesized gesture sequences. Compared to previous approaches, our method produces more diversified gestures that aptly align with the textual content. Our approach could generate longer motions more than 20s (the training length of transformer), for example, to handle audio from 21 to 40 seconds, the transformer takes as input both the predicted motion in 20s as seed pose and the audio and textual features from 21 to 40s, as shown in video results.We further conduct a user study, assessing three distinct dimensions: Preference, Beat Alignment, and Diversity.• Preference: This mertic encompasses (i) the physical correctness of the results, which assesses issues such as jitters and artifacts (e.g., hands blending into incorrect angles), and (ii) the semantic relevancy between the audio content and gestures. For instance, the phrase "a huge ball" should correspond with an open-arm gesture to ensure realism. • Beat Alignment: Evaluation under this mertic focuses on the synchronization of the gesture's rhythm with the audio's rhythm. For example, as the speaker utters "that is," the corresponding hand movement should commence with the word "that" and conclude before "is" is pronounced, demonstrating effective beat alignment. • Diversity: This metric assesses the variety of gesture classes within a 10-second sequence, similar to the approach in related works such as Audio2Gestures [23] and DisCo [26] .A sequence repeating a single, semantically aligned motion is considered realistic but lacking in diversity. Conversely, a sequence incorporating various types of content-rich motions is categorized as having high diversity. Before the test, participants are required to: (i) read and watch an instructional video explaining the evaluation metrics; (ii) evaluate five test videos and submit their results; (iii) take a screening process where submissions from participants with a random-like win rate are filtered out. Then in each test, participants evaluate a pair of synthesized gestures and are expected to select the winner across the each of three metrics. Gestures sequences are 10-20s, with a total of 60 sequences sampled (10 from the Trinity-trained model and 50 from BEAT, encompassing 5 different speakers) for every model, which are totally 600 comparisons. In the user study, we report a subset of baselines based on their performance rank, specifically: DisCo [26] , HA2G [28] , TalkShow [39] , QPGesture [38] , and our EGGesture.Figure 4 shows that most methods yield similar results in terms of beat alignment. This may be due to the inherent challenge for human evaluators to accurately gauge the congruence between motion and audio beat alignment. Overall, EGGesture perform a higher preference and diversity (semantic alignment) win rates. 
ablations 5.1 comparison with reweighting baselines: 
 We start our ablation study by comparing against a straightforward baseline: retaining the vanilla VQ-VAE's clustering phase while only reweighting the classification phase. Table 3 presents results from five reweighting strategies: log, sqrt, linear, square, and exponent of the inverse class probabilities. Notices that our EGGesture actually follow a linear reweighting setting. When comparing our EGGesture with the traditional baseline, it becomes evident that linear reweighting is the optimal approach, mitigating the effects of an imbalanced codebook indices distribution.Moreover, as shown in Figure 6 , in contrast to the baseline method that maintains a stable codebook usage ratio, the implementation of entropy loss progressively converge towards a higher codebook usage rate. This approach significantly outperforms conventional clustering methods, highlighting the benefits of achieving a balanced distribution and enhanced codebook utilization.We report the ablation of the comparison between proposed EGGesture and EGGesture without classification reweighing, as shown in Table 4 , the result demonstrates that only optimizing the distribution of gesture motion codebook to a balanced distribution will increase the difficulty of classfication (audio to gesture codebook index) stage, leading a limited improvment compared with the vanilla gesture VQ-VAE (FID 10.38 We then evaluate the effects of removing the joint clustering and classification training, as shown in Table 5 . Only optimizing the clustering phase to enforce equal probability and keep the 100% codebook usage detrimentally impacts reconstruction performance, suggesting a sub-optimal learned latent representation. This in turn compromises the quality of synthesized gestures. However, when classification and reconstruction are jointly trained, the model effectively balances between these two objectives, leading to better performance.As shown in Table 5 , the utilization of Gumbel softmax assignment provides an enhancement to our model's performance. Nonetheless, there's a pronounced sensitivity in the choice of 𝛾. Using the standard softmax leads to a fast convergence of the classification loss, often culminating in less-than-ideal results. Conversely, employing the Gumbel softmax infuses noise into the true index, increasing the classification difficulty and consequently leading to a more stable convergence.As shown in Figure 7 to 2048 and further to 10240. Subjectively, when visualizing the reconstructions from vanilla VQ-VAE, our EG-VQVAE, and the ground truth, there are more detailed gestures for EG-VQVAE, e.g., more accurate spatial positioning. This implies that EG-VQVAE could capture more fine-gained gesture representations.We also experimented with a variety of pre-trained audio and text encoders. We first found the finetuned wav2vec2 performance is similar to a customized TCN [40] with a FID 5.83. This indicates that the gradients passed back to the audio encoders might be minimal, leading to only marginal adjustments to the pre-trained encoders. But in this paper, we keep the same setting, i.e., leveraging the wav2vec2, with previous baseline [39] for a healthy comparison. Besides, for the text, our experiments also did not reveal any significant performance advantages when varying the text encoder. Results in Table 6 demonstrate that finetune the encoder or not is more important than the type of pre-trained encoder, and even with a customed TCN without pretraining, we could get similar results.Compared to other methods, the improvement of performance from our proposed EGGesture is agnoistic with the selection of network architectures for the audio, text encoder and motion decoder. We report a comparison of Transformer-based and LSTM-based EGGesture, which replaces the audio, text encoder and audio to motion cross-attention to LSTM. The results are shown in Table 7 , EGgestures' performance is suboptimal when the network architecture is LSTM as the given the same training time, e.g., 7 hours for 100 epochs, the LSTM-based encodes could only be trained within 34 frames, and the Transformer-based encoders could be trained in 300 frames.  
conclusion: 
 In this paper, we address the problem of imbalanced co-speech gesture generation by introducing EGGesture, a framework that synchronously optimizes the codebook learning and classification phases. EGGesture integrates a differentiable entropy regularization, employing this entropy for reweighting during the classification phase. This approach propels us to achieve state-of-the-art results in the domain. In the future, we will explore the distribution of vector utilization across different dimensions, allowing us to delve deeper into the constraints of motion VQ-VAEs.  
: 
 Recent works mitigate the impact of imbalance by employing Vector Quantized Variational AutoEncoder (VQ-VAE) [35] , demonstrating improved gesture diversity. Approaches based on VQ-VAE [2, 3, 39] synthesize gestures in two phases: first, a gesture codebook is pretrained using vanilla motion VQ-VAE, followed by the sequential classification of codebook indices using audio and text cues. The advantage of VQ-VAE lies in its transformation of the gesture regression problem into a classification one, normalizing the penalization across various gesture classes, thereby boosting the recall of gestures from minority classes. However, as shown in Figure 1 , there are two bottlenecks limiting the performance of VQ-VAE based methods: the imbalanced distribution of codebook indices and the suboptimal codebook utilization.The imbalanced distribution of codebook indices. In the context of co-speech gestures generation, our observations show that the distribution of codebook indices remains similarly imbalanced regardless of the codebook size (as shown in Figure 2 ). This kind of distribution is expected when learning representations of imbalanced gestures. However, it adversely impacts classification during the subsequent phase.The suboptimal codebook utilization. As shown in Figure 2 , an evident upper limit exists on the effective use of the codebook, even when assigning a large codebook size, e.g., 10240, for cospeech gesture generation. This suboptimal codebook utilization results in the increase of codebook size doesn't improve the gesture representation learning and FID scores, i.e., clustering gestures into more fine-grained tokens.The above analysis raise one straightforward concept of potential solutions: Balancing and maximizing the codebook's utilization, to enable a more fine-grained representation learning and a balanced classification task. However, the implementations should be carefully considered as: i) Maximizing the codebook utilization, i.e., the number of effectively used indices, will benefit representation learning but also increase the complexity of the classification task.ii) Balancing the distribution of codebook indices, will improve the classifier but adversely affect for the representation learning, as the real-world gestures inherently follow an imbalanced distribution, it's theoretically advantageous for the VQ-VAE space to reflect this original distribution, as discussed in [21] .iii) To the best of our knowledge, there exist few methods that compel VQ-VAE to employ a broader range of the codebook during its training phase. Calculating the utilization rate presents challenges as the index operations are not differentiable.These suggest a trade-off between the performance across different phases of VQ-VAE when attempting to both maximum and balance the utilization. We hypothesize that there's an optimal balance between utilization and distribution for each co-speech gesture dataset, evaluated in terms of final generated gestures from speech and text. Based on the above analysis. we propose an implementation of the concept of balancing and maximizing, termed Entropy-Guided VQ-VAE. This implementation optimizes both codebook utilization and distribution in a differentiable, data-driven manner.The EGGesture consists of three components: a) we employ the entropy of codebook utilization probability to evaluate the utilization of codebook. b) We compute entropy in a differentiable way by combining cosine similarity with the Gumbel-Softmax function [20] . This enables us to incorporate the codebook utilization probability as a regularization loss during training. c) We adopt a joint training strategy for both codebook learning and classification. The classification loss is weighted based on the computed entropy. This strategy effectively addresses the trade-off of entropy optimization by simultaneously minimum both reconstruction and classification losses. Overall, our contributions are:• We introduce the concept to balance and maximize codebook utilization, addressing the bottleneck in VQ-VAE-based cospeech gesture generation.  
 
 Appendix:  


Decision: This paper aims to address the challenges of imbalanced distribution in Co-Speech gesture generation and proposes a differentiable approach to optimize the entropy of the codebook indices to yield a more balanced distribution. Experiments on the Trinity and BEAT datasets demonstrate the effectiveness of the proposed method. This paper received four reviews with initial ratings of two borderline reject, one borderline accept, and one weak accept. The authors provided rebuttal to address the reviewers' concerns. After discussion, the final ratings became two borderline accept and two weak accept. The reviewers are satisfied with the rebuttal and agree most of their concerns are addressed in the rebuttal. I recommend acceptance of this paper and request the authors to take the reviews and their rebuttal into consideration when preparing the camera-ready version.

Reviews: review: paper_summary: paper_summary: The paper tackles the issue of imbalanced, long-tailed gesture distributions prevalent in co-speech gesture generation by introducing an Entropy-Guided VQ-VAE. This approach enhances the distribution of codebook indices and dynamically adjusts classification loss weights, achieving both balanced and maximized codebook utilization. The method has proven its efficacy, showcasing state-of-the-art performance across the Trinity and BEAT datasets.
strength_weakness: strength_weakness: 1. The paper is well written and easy to follow.

2. The paper introduces an entropy-guided VQ-VAE approach that effectively employs a broader range of the codebook during training. This optimization enhances the latent representation from imbalanced data, which is a significant advancement over traditional methods.

3. Comprehensive experiments are conducted to validate the effectiveness of the proposed method.
limitations: limitations: 1. While the VQ-VAE is commonly applied in various domains, the paper does not explore the potential of its entropy-guided variant in other long-tailed representation tasks beyond co-speech gesture generation. 

2. The paper does not sufficiently discuss the limitations of end-to-end generative models, such as GANs, standard VAEs, and Diffusion Models, in managing unbalanced co-speech gesture datasets. Additionally, the paper does not clearly establish the necessary relevance of VQVAE to this specific task.

3. A broader comparison that includes various categories of current baseline methods, such as diffusion-based approaches, would better highlight the effectiveness of the entropy-guided VQ-VAE and provide a clearer overview of its advantages over existing techniques.
paper_summary: paper_summary: This paper introduces EGGesture, which addresses the problem of imbalanced co-speech gesture generation through differentiable entropy regularization, enhancing the potential of VQ-VAE in the A2G domain.
strength_weakness: strength_weakness: 1. The paper is well-written, and the experiments are comprehensive.
2. The use of differentiable entropy for re-weighting during the classification stage is quite novel.
limitations: limitations: The paper lacks comparison with the latest A2G methods based on VQ-VAE, given the continuous emergence of new A2G approaches recently.
paper_summary: paper_summary: This paper proposes a method to improve the vector quantized variational autoencoder (VQ-VAE) based framework to generate co-speech gestures. VQ-VAE is a popular method that encodes gestures into a discrete codebook but faces the issue of imbalanced, long-tailed gesture distributions. To solve this problem, authors employ the entropy information to guide the gesture generation. The experiments on two datasets demonstrate its capability to generate diverse and semantic-related gestures.
strength_weakness: strength_weakness: 1. The authors utilize gumble-softmax to optimize the entropy of generated gesture distribution and improve the quality of co-speech gesture generation.
2. The qualitative results and the attached video demonstrate a more diverse and semantic-related gesture.
limitations: limitations: 1. Data imbalance and long tail are common issues in data-driven tasks. Experiments compared with other baselines for long-tail and imbalance issues.
2. What's the distribution of the real-life co-speech gestures? Is it necessary to increase the ratio of gestures in the tail?
3. In paper [1], the authors discretize the co-speech gestures into a codebook in a fine-grained way. Is it helpful to alleviate the long-tail and imbalance issues?

[1] Learn to Gesture: Let Your Body Speak. https://dl.acm.org/doi/abs/10.1145/3338533.3366602
paper_summary: paper_summary: This paper aims to address the challenges of imbalanced distribution in Co-Speech gesture generation. The authors find two issues that leads to suboptimal performance when using codebook: (1) the imbalanced distribution of codebook indices (2) the suboptimal codebook utilization. The authors propose a differentiable approach to optimize the entropy of the codebook indices to yield a more balanced distribution. Experiments on the Trinity and BEAT datasets demonstrate the strong performance of the proposed losses.
strength_weakness: strength_weakness: 1) Writing: The paper is clearly written and well-presented.

2) Motivation: The authors highlight the overlooked issues of long-tailedness and suboptimal utilization in co-speech gesture generation. The motivation is clear and convincing.

3) Approach: The authors propose a differentiable method to optimize the entropy of the codebook, which balances the distribution of indices. The proposed method is straightforward, intuitive, and easy to implement.

4) Experiments: The experiments conducted are comprehensive, and the visualizations provided are convincing.
limitations: limitations: I have some comments regarding Section 2.3, "Imbalance Problem in Machine Learning." In addition to importance sampling (resampling and reweighting), typical solutions in machine learning also include logit adjustment [1] and knowledge distillation [2]. Please consider discussing these methods and including them in the ablation studies, for example, in Table 3 or Table 4.

[1] Long-tail learning via logit adjustment.

[2] Cross Domain Empirical Risk Minimization for Unbiased Long-tailed Classification.