Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training

Authors: haoran  he
chenjia  bai
ling  pan
weinan  zhang
bin  zhao
xuelong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:57:40

Abstract: Learning a generalist embodied agent capable of completing multiple tasks poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. In contrast, a vast amount of human videos exist, capturing intricate tasks and interactions with the physical world. Promising prospects arise for utilizing actionless human videos for pre-training and transferring the knowledge to facilitate robot policy learning through limited robot demonstrations. However, it remains a challenge due to the domain gap between humans and robots. Moreover, it is difficult to extract useful information representing the dynamic world from human videos, because of its noisy and multimodal data structure. In this paper, we introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. We start by compressing both human and robot videos into unified video tokens. In the pre-training stage, we employ a discrete diffusion model with a mask-and-replace diffusion strategy to predict future video tokens in the latent space. In the fine-tuning stage, we harness the imagined future videos to guide low-level action learning with a limited set of robot data. Experiments demonstrate that our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches with superior performance.

Main paper: 
 introduction: 
 How do we derive a general-purpose robot agent that can complete a wide variety of tasks? We believe that recent advances in vision and language give us a clue, which delves into pre-training foundation models on extremely large and diverse datasets, followed by fine-tuning on specific domains. For instance, through pre-training on internet-scale datasets [65] , large language models [81, 82, 63] and vision models [9, 69, 72] showcase impressive performance on various downstream tasks such as question answering, coding, and image generation. However, unlike general visual language tasks that can exploit copious amounts of data available on the Internet, embodied tasks necessitate high-quality egocentric data in robotics domains for precise control. Collecting such data can be expensive or time-consuming due to the reliance on robot interactions through teleoperation or kinematic solvers [31] , and significant gaps in embodiments and dynamics persist when applying them to different robots.In contrast to the limited availability of robot data, there is a wealth of human interaction videos capturing intricate tasks and varied interactions with the physical world [27] . These videos inherently encapsulate rich semantic information regarding objects, environmental backgrounds, and handobject interactions across diverse scenarios, making them potentially valuable for acquiring shareable knowledge relevant to embodied tasks. Motivated by this, many works have emerged to learn various objectives pre-trained on human actionless videos, aiming to capture useful knowledge that can be beneficial for embodied tasks. These approaches involve learning pre-trained image representations [61, 57, 68, 91] , trajectory representations [4, 85, 74] , reward functions [13, 56] and world models [59, 89] . However, they are still limited to comprehending the dynamic rules of the world or reasoning based on long-term behavior rather than relying solely on step-by-step transitions. We summarize three main challenges that bottleneck their performance: (i) The domain gap between humans and robots which hinders knowledge transfer; (ii) Complex, diverse and noisy behavior patterns hidden in human videos which are difficult to learn; (iii) Large-scale data from different modalities (e.g., videos, actions, texts) which requires a scalable and high-expressive model architecture to process.To address these challenges, we propose a Video-based Policy learning framework via Discrete Diffusion (VPDD). VPDD bridges the visual gap between the human and robot domains by representing these two data types as unified latent representations. Then, VPDD performs video prediction as a pre-training stage with actionless videos, which acquires the commonsense knowledge shared between human and robot interactions, including dynamic rules and behavior patterns (e.g., pick, place, push) to understand and complete tasks. Then, VPDD performs policy learning via a fine-tuning stage with action-labeled robot videos, where VPDD learns to predict actions with foresight from future video predictions. The pre-training stage learns extensive knowledge from human video prediction, and the fine-tuning stage concentrates on training parameters specifically associated with actions.To tackle the challenge of modeling the noisy and complex distribution of large-scale videos while enabling the multi-modal generation of both videos and actions, we leverage the generative capability and flexible architecture offered by discrete diffusion models [2, 29] . We provide an overview of our method in Fig. 1 .In summary, we highlight our contributions as follows. (i) We propose VPDD, a novel pretrainingfinetuning paradigm for learning an actionable policy with limited robot data accessible. This paradigm demonstrates superior ability in transferring valuable knowledge from large-scale actionless human videos to downstream embodied tasks. (ii) We formulate both video prediction and action learning processes as unified discrete denoising problems, showing the supreme effectiveness in handling high-dimensional, multi-modal data. (iii) We conduct thorough experiments using human videos from Ego4D [27] , as well as embodied datasets from Meta-World [98] and RLBench [43] ,showcasing its ability to predict dynamic-consistent future videos. Our actionable discrete diffusion policy also exhibits superior performance compared to previous state-of-the-art approaches [32, 61, 24, 75, 15] , encompassing both seen and unseen scenes for multi-task robotic problems.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The contributions and scope are introduced in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries 2.1 multi-task pomdp: 
 In this work, we consider a generalist vision-based agent that is capable of addressing multi-task predicaments, where the landscape is characterized by the inherent challenge of acquiring different skills across tasks and partial observability when dealing with image inputs. Given a specific task T ∼ p(T ), we further approach the problem as a task-specified Partially Observable Markov Decision Process (POMDP), defined as (S T , O, A, P T , R T , µ T , γ). Here, O is a shared observation space as we use image observations for all tasks. We also assume all tasks share the same action space with the same embodiment.In order to unify the feature space of both human videos and robot videos, we leverage the Vector Quantized Variational Auto Encoder (VQ-VAE) [83] to compress high-dimensional data points into information-rich discretized latent codes. Given a high-dimensional video segment x ∈ R T ×H×W ×C , the encoder E first converts it to the temporal-spatial features z = E(x) = {z m,i,l } ∈ R t×h×w×d , where t × h × w represents the encoded sequence length and is much smaller than T × H × W . Then we transfer the continuous features into discrete space by performing a nearest neighbors lookup in a codebook of embeddings Z = {e j } J j=1 ∈ R J×d to obtain the tokensEQUATIONwhere the video tokens z q ∈ R t×h×w×d can be faithfully reconstructed via a decoder, i.e., x = G(z q ).The encoder E, decoder G, and codebook Z can be trained end-to-end via the following loss functionL = ∥x -x∥ 1 + ∥sg[E(x)] -z q ∥ 2 2 + β∥sg[z q ] -E(x)∥ 2 2, where sg denotes stop gradient.The discrete diffusion model was first proposed to deal with discrete state space with transitions converging to a binomial distribution [76] , and then extended to multinomial diffusion with more options for transition matrices [36, 2] . In this work, we utilize discrete diffusion with the absorbing state for sequence prediction of discrete tokens. Besides J tokens from a codebook, an additional [MASK] token is introduced. We denote x k as a one-hot vector identifying the token index. The forward process fromEQUATIONwhere ) is the Markov transition matrix from k -1 to k, which is formulated as:[Q k ] m,n = q(x k = m|x k-1 = n) ∈ R (J+1)×(J+1EQUATIONwhere α k ∈ [0, 1] is the probability of retaining the token, and each ordinary token has a probability of γ k to be replaced by [MASK] token, leaving a chance ofβ k = (1 -α k -γ k )/J to be diffused.Importantly, due to the property of the Markov chain, we can derive the probability of x k at arbitrary timestep directly from x 0 asEQUATIONBesides, the posterior of this diffusion process is tractable asq(x k-1 |x k , x 0 ) = q(x k |x k-1 ,x0)q(x k-1 |x0) q(x k |x0) = (x T k Q k x k-1 )(x T k-1 Q k-1 x0) x T k Q k x0. In the reverse process, rather than explicitly predicting the posterior through a denoising neural network, the x 0 -parameterisation enhances stability and allows for fast inference (by skipping ∆k steps per iteration). The reverse transition with reparameterisation is formulated asEQUATIONwhere the neural network predicts the logits of the target data q(x 0 ).Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: It has been discussed in Appendix D.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 We commence with the pre-training of our model through future video prediction, enabling the learning of a general dynamic pattern across diverse domains. Subsequently, we fine-tune the model using a limited dataset of robot data for policy learning, leveraging foresight from predicted videos. Our framework is illustrated in Figure 2 . In each diffusion step of the fine-tuning stage, we leverage p θ1 to provide hidden representations z xv 0 to benefit downstream action learning with video foresight. This integration of video prediction and action learning is achieved through our unified discrete diffusion.Robot Data Collection. We use the rule-based script policy to rollout 20 expert demonstrations for each task in Meta-World [98] . We also run VPDD on 16 tasks from RLBench [43] , a more challenging benchmark involving 6-Dof manipulation that necessitates multi-view images from a 3D scene to select actions. Following the multi-view manipulation frameworks [24, 75] , we utilize the script motion-planner to collect 10 demonstrations for each task. Each demonstration from robot-data is formulated as Human Data Collection. As for human data collection, we obtain untrimmed videos from the open-sourced Ego4D dataset [27] , which contains massive-scale human-object interactions of various durations ranging from 5 seconds to 7 hours. We filter out videos without human-object interaction and segment each video into short clips with 8-frame intervals [60] . Thus each video is represented asτ i = {v 1 , a 1 , • • • , v t , a t , • • • , v T , a T } with v t = [o t-I+1 , • • • , o t-1 , o t ],τ i = {v 1 , v 2 , • • • , v n }, where v t denotes a clip containing 4 frames. This approach yields a total of 996,177 clips of human videos, comprising approximately 4M frames. More details on data collection and processing are given in §C.1.VQ-VAE Encoding. To extract useful features from raw videos in both human and robot domains, a conventional approach is to directly encode them into an embedding space using pre-trained vision models like ViT. However, these models are usually specifically trained on image dataset [71] , posing a significant challenge due to the domain gap with our interaction videos. Thus, we leverage VQ-VAE to compress the diverse and noisy videos into discrete latent codes, which provide a unified codebook for mixed videos and alleviate the domain gaps between human and robot videos. Formally, we adopt the VQ-VAE architecture introduced by VideoGPT [94] for encoding videos into a discrete latent space. The codebook comprises 2048 codes, each represented by 256-dimensional embeddings. The encoder architecture consists of a series of 3D convolutions that downsample by a factor of 4 over space-time (resulting in a 64× total reduction), followed by 6 attention residual blocks. Consequently, each video clip v t ∈ {τ i } is embedded into latent codes e t . The architecture for the decoder is the reverse of the encoder, featuring attention residual blocks followed by an equivalent number of 3D transposed convolutions for upsampling over space-time. The VQ-VAE is pre-trained on large-scale videos and remains fixed in the subsequent processes, providing flexibility for various downstream utilization methods.Action Discretizing. For subsequent pre-training and fine-tuning, we process the collected continuous actions via uniform action discretization [46, 11] . In the case of Meta-World, the action space is a 2-tuple consisting of the change in the 3D space of the end-effector followed by a normalized torque that the gripper fingers should apply. Here all the continuous dimensions are discretized into 48 bins uniformly. Thus, the robot action can be represented using ordinals of the discrete bins as a 4 integer number. For RLBench, an action consists of the gripper open state and 6-DoF pose including position and rotation. The position is discretized into 360 bins, and rotation is discretized into Euler angles as 1-degree bins for each of the 3 rotation axes [75] . Gripper open state is a binary value.Extracting general patterns useful for downstream decision-making from large-scale in-the-wild human videos is challenging, primarily because of the absence of labeled actions and the complexity of the underlying structure of human interactions. Different from previous ways of learning a visual representation, we propose a novel objective to further unleash the representation and temporal modeling ability of diffusion models. Specifically, after obtaining discrete tokens from VQ-VAE encoding, we train a unified discrete diffusion model on the latent space via a self-supervised objective. This objective involves predicting future videos based on observed historical videos for both humans and robots, while masking action tokens. Benefiting from the proposed objective and the x 0 -parameterisation of discrete diffusion, the diffusion model is incentivized to capture both the high-level temporal dynamics and the low-level visual commonalities between historical and future videos at each diffusion step. Then the acquired knowledge can be leveraged to guide action denoising at each step.Unified Transition Matrix. The presence of a transition matrix determines the nature of the discrete diffusion model [2] . While the original discrete diffusion is limited to one modality, drawing inspiration from UniD3 [37] , which enhances the transition matrix to encompass both images and text, we construct a unified transition matrix to capture global connections between the two modalities-videos and actions. The matrix [Q k ] m,n below illustrates the unified transition process:Qk =                 αk + βk βk • • • βk 0 0 • • • 0 0 βk αk + βk • • • βk 0 0 • • • 0 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . βk βk • • • αk + βk 0 0 • • • 0 0 0 0 • • • 0 αk + β * k β * k • • • β * k 0 0 0 • • • 0 β * k αk + β * k • • • β * k 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 0 0 • • • 0 β * k β * k • • • αk + β * k 0 γk γk • • • γk γk γk • • • γk 1                , where β k and β * k are the probabilities of a token to be replaced by any other accessible tokens in different modalities. The dimension of Q k is (J + J * + 1) × (J + J * + 1), where J and J * are the number of tokens in different modalities, i.e., J is the size of codebook in VQ-VAE and J * is the number of action classes in discretization. The sum of each column in this transition matrix is one to preserve probability mass. Mathematically, we haveβ k = (1 -α k -γ k )/J and β * k = (1 -α k -γ k )/J * .All the mass of the stationary distribution falls on the [MASK] token, which satisfies the prerequisite for a discrete diffusion model transition matrix [2] . The details of the diffusion process are provided in §A.1.Unified Objective. We cast both video prediction and action learning as a conditional generative problem, and the goal is to maximize E τ ∼∪iτi log p θ (x 0 (τ ) y(τ ), l . Here x = [x v , x a ], where x v = [e t+h+1 , • • • , e t+h+M ] represents future video segments with M clips, andx a = [a t , • • • , a t+H-1 ]denotes action sequences of H steps. y = e t serves as the condition containing historical video tokens. l is the language instructions describing current tasks. In practice, we train two separate denoising networks, namely p θ1 (x v k-1 |x k , y, l) and p θ2 (x a k-1 |x k , z xv 0 , l), to learn videos and actions, respectively. Here, z xv 0 represents the hidden representation of predicted future videos given by p θ1 at each diffusion step, which is utilized to guide action learning. Formally, xv 0 = Softmax(MLP(z xv 0 )).As the actions are absent during the pre-training stage, we mask x a and freeze p θ2 , as illustrated in Figure 2 . The network is trained to minimize the following variational lower bound (VLB) [76, 29] :EQUATIONEQUATIONEQUATIONEQUATION)L K is a constant number that can be ignored in the training, as the prior distribution p(x K ) is fixed:EQUATION)Model Architecture. As in Eq. ( 8), the neural network p θ1 receives x k , history y, language l and the diffusion timestep k as inputs. These inputs are individually embedded into embeddings h of size d via separate MLPs f , depicted as:h l = f l (CLIP(l)), h T i = f T i (k), h x k = f x k (x k ), h y = f y (y),where language instructions l is encoded with CLIP's language encoder [66] . Afterwards, the embeddings are formulated as input tokens ash tokens = LN(h T i × [h l , h T i , h x k , h y ] + E pos ), where E pos is the positional embedding, and LN denotes layer normalization [3] for stabilizing training.The input sequence that represents a video can be extremely long so a standard Transformer with O(n 2 ) complexity is hard to fit. We adopt Perceiver Transformer [41] to tackle this problem, as it has been widely utilized for modeling long sequences [75, 24] . Perceiver is a latent-space Transformer, where instead of attending to the entire input, it computes cross-attention between the input and a much smaller set of latent vectors (which are randomly initialized and trained). These latents are encoded with self-attention layers, and are cross-attended with the input to match the size for the final outputs. More details about the Perceiver Transformer are referred to §A.2.During the fine-tuning stage, we leverage a limited dataset of robot data, including both videos and actions, for rapid adaptation. Both x v and x a attend to training the diffusion model. Given that p θ1 has been trained sufficiently to capture fruitful information to predict future videos from history, we freeze p θ1 and solely tune parameters of p θ2 to minimize L vlb . As expressed in Eq. ( 8), the input of p θ2 consists of x k , language l, hidden representation z xv 0 , and diffusion timestep k. In this case, we are tasked with predicting a sequence of action tokens x a 0 , considerably shorter than video-token sequence x v 0 , so we employ GPT2 [67] Transformer to process tokens embedded with MLPs. GPT2 has demonstrated an impressive ability to solve multi-task problems and model multimodal distributions. The model architecture of p θ2 closely resembles that of MTDIFF-P [32] . More details of our method can be found in Appendix A.3.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: We provide the code and sufficient instructions of our approach in the supplementary material.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Robot Learning from Human Videos. Leveraging human videos [26, 23, 18, 27] for policy learning is promising to extract commonsense knowledge from human activities, which can be shared to embodied scenarios that suffer from scarce robot data [17, 22] . Since the human data is actionless and the domain gap between humans and robots exists, a main branch of research employs human video to learn shareable visual representations [12, 47] via time-contrastive [57] , video-language alignment [61, 48] , value function [8] , and perceptual skills [40, 52] . Visual affordance like human-object interaction hotspots [54, 25] and the post-grasp trajectory [5] are also helpful for embodied agents in goal-conditioned imitation. Alternative methods involve extracting hand trajectories [4, 85] or keypoints [93] to transfer plans to robots. Different from the above methods, we eliminate the domain gaps by learning video tokens and representations for video prediction, which implicitly captures visual features, affordances, and long-term plans. Other works attempt to infer actions from videos via inverse kinematics [6, 50] , whereas we learn action prediction through policy fine-tuning without external models.Pretraining for Generalized Policy Learning. Early works of policy adaptation emerged in meta-RL [30, 99] , while the pre-training and fine-tuning environments are assumed to be similar. Leveraging the transformer architecture, works perform pre-training in multi-task datasets by optimizing the multi-task policy [51, 70, 79, 80] or self-supervised objectives [78, 73, 59] . In tasks involving visual observations, methods adopt visual tokens for transformer-based multi-task policy learning [10, 11] and adaptation [53] . Additionally, some studies pre-train a reward function through video-language correspondence [13, 56] or diffusion models [21, 38] for downstream RL training. Different from the previous Transformer and continuous diffusion frameworks, our work first integrates visual tokens with discrete diffusion to predict consistent videos and actions simultaneously. Concurrently, GR-1 [88] utilizes human data to pre-train a GPT-style architecture for predicting future observations. In contrast, we perform video prediction instead of step-by-step image prediction using a unified discrete diffusion architecture.Diffusion Models for RL. Diffusion models are a powerful family of generative models [72, 69] that can be categorized into continuous Gaussian diffusion models and discrete diffusion models that handle discrete visual tokens or symbols [29] . Continuous diffusion models have found extensive applications as multi-modal policies [87, 64, 16] , environmental dynamics [95] , and planners to generate action [45, 90] or state sequences [1, 14] , guided by desired properties. Several methods also extend the diffusion models for multi-task learning [32, 62, 19] with low-dimensional states, while we specifically address the more challenging image-based setting. UniPi [20] and its following work [100] are related to our method by performing video generation via continuous diffusion, while we adopt a more flexible architecture with discrete diffusion to seamlessly connect the pretraining and fine-tuning stages, without relying on a task-specific inverse-dynamic model for acting. Additionally, we train the model on video tokens that maintain temporal consistency, while UniPi relies on super-resolution to improve the time consistency of generated frames. 
bnechmarks and baselines: 
 After the video pertaining in Ego4D [27] , we use the following robotic benchmarks to evaluate our method.Meta-World. The Meta-World benchmark [98] contains 50 distinct manipulation tasks that require a Sawyer robot to interact with various objects with different shapes, joints, and connectivity. The action is the 3D position movements of the robot's end effector and the gripper openness. We follow recent works [96, 77] to extend the original environment to a more challenging setting with random goals, and refer to it as MT50-rand. We train the policy with 20 demonstrations per task, and report the average success rates on 50 evaluation episodes per task.RLBench. RLBench [43] is a more challenging 3D manipulation benchmark with diverse tasks concerning interactions with a wide range of objects. We select 16 tasks from RLBench to evaluate our method, where each task has several possible variations, such as the shapes, colors, sizes and positions of objects. The input observations are captured from four RGB cameras positioned at the front, left shoulder, right shoulder, and wrist. The action is an 8-dimensional vector including 3-dimensional transitions, 4-dimensional quaternion, and a binary value about gripper openness. We follow the convention by using macro steps [42] , which are key turning points in the action trajectory where the gripper changes its state (open/close) or the joint velocities approach to zero. We train the policy with 10 demonstrations per task and report average success rates on 25 evaluation episodes per task.Baselines for Meta-World. We compare the proposed method VPDD with the following baselines: (i) R3M-Diffusion is a discrete diffusion model sharing identical architecture with p θ2 , leveraging the R3M [61] ResNet50 encoder to encode images as input. R3M is also trained on Ego4D videos via a contrastive learning objective and stands as the state-of-the-art (SOTA) visual representation specifically designed for manipulation tasks; (ii) VC-1-Diffusion utilizes VC-1 [58] encoder (ViT-L) to extract image representations, which is also trained on large-scale egocentric videos [27] and ImageNet [71] using Masked Auto-Encoding [33] . (iii) MTDIFF-P [32] is the SOTA method for multi-task RL, which employs a continuous diffusion model with a Transformer architecture. Since it is designed to handle state-based input, we employ the R3M encoder to extract hidden embeddings from images, which are then fed into MTDIFF-P; (iii) Video-MTDT is an extension of Decision Transformer (MT) [15] , learning from multi-task data with video tokens as input; (v) VPDD-w/o.- human excludes human videos during pre-training, remaining other parts unchanged. This baseline helps to ablate the effect of pre-train on large-scale human videos; (vi) SODA [39] is a recently proposed diffusion-based representation method that employs an encoder to generate representations z from input images to support the denoising process. We pre-train the encoder by employing the video prediction objective on the same dataset and subsequently feed the learned z into p θ2 during fine-tuning. See more details in §B.Baselines for RLBench. Learning policies for RLBench is more challenging as it requires understanding the 3D scene structure for predicting the 6D poses of end-effectors. The baselines used in Meta-World all fail in the benchmark since they are disadvantaged with single-view observations. In contrast, VPDD can predict multi-view images, implicitly recovering the 3D geometry in manipulation. Thus, we use the following SOTA imitation architectures designed for 3D manipulation: (i) RVT [24] stands as the SOTA method, initially re-rendering visual observations into orthographic projections of cube views and subsequently predicting the next move based on these multi-view projections.; (ii) PERACT [75] encodes RGB-D images into voxel grid patches for 3D representation and predicts the action using the Perceiver Transformer. Although our primary contribution is not about video generation, it remains crucial to predict consistent future videos to aid in policy fine-tuning.The predicted raw videos are visually depicted in Fig. 3 , with more video samples including real robots (trained by RoboSet data [7] ) accessible at https://video-diff.github.io.UniPi [20] 264.66 VPDD (Ours) 235.81 These videos are reconstructed from predicted video tokens by using the decoder of VQ-VAE. After pre-training, the video-prediction model p θ1 demonstrates the capability to generate dynamically consistent single-view videos for Meta-World and multi-view videos for RLBench. Furthermore, we computed the Frechet Video Distance (FVD) score [34, 86] averaged across frames on 32 generated video samples from the Meta-World dataset. From the results in Table 5 .2, we observe that the FVD score of VPDD is considered acceptable and even lower than the hierarchical video synthesizing method UniPi [20] (score reported from its original paper). We attribute the capability of generating high-quality videos to the well-trained video codebook and the proposed discrete diffusion model learned from large-scale human data. To evaluate the learned policy after fine-tuning, we take the first action generated by p θ2 to interact with the environment. The results on Meta-World and RLBench are referred to Fig. 4 where the red curve is evaluated using the same experimental setting as in Fig. 4 , and the blue curve corresponds to the setting in Fig. 5 .Question 4. Can VPDD maintain satisfactory performance when provided with fewer robotic demonstrations?Leveraging the large-scale video pre-training, VPDD can learn the policy using only a small number of demonstrations. To validate the sample efficiency of our method, we conduct an ablation study on the number of demonstrations used in the fine-tuning stage. The results, depicted in Fig. 6 , reveal that the performance of VPDD exhibits linear growth after training on 5 or more demonstrations, indicating the potential for VPDD to achieve better performance with increased demonstration data. Moreover, VPDD maintains a comparable success rate even when only 1 demonstration is used in the fine-tuning process.The superior generalizability of VPDD, which is validated in Fig. 5 , stems from large-scale human-data pertaining, which effectively extracts commonsense knowledge and representations that can be shared between human and unseen robot scenarios. To further investigate the effects of human-data pre-training, we ablate the number of human videos used during pre-training and present the performance changes in Fig. 7 . Our findings indicate that an increased number of human videos enhances success rates, particularly in improving generalizability by a large margin.To verify the representation capability inherent in our unified discrete diffusion framework, we reproduce SODA [39] , which serves as a strong diffusion representation method. As shown in Fig. 4 , VPDD outperforms SODA in the context of policy fine-tuning. We hypothesize that VPDD provides coarse-to-fine representations (i.e., z xv 0 ) throughout the action-denoising steps from K → 1, exactly encapsulating useful information the denoising network focuses on at each step [35] . In contrast, SODA produces representation z that remains constant across all steps during action denoising. 
conclusion: 
 We propose VPDD, a video-based policy learning framework via discrete diffusion. With a VQ-VAE tokenizer, we bridge the gap between human and robot videos by a discrete latent codebook. We leverage a unified discrete diffusion for pre-training on large-scale actionless mixture videos and subsequent fine-tuning the policy on a limited number of robot demonstrations. Experiments demonstrate that VPDD achieves superior performance on a variety of challenging manipulation tasks and showcases impressive generalization ability benefited from human video prediction. More Discussions of our work are given in §D.• We set batch size as 20 for pre-training and 40 for fine-tuning. We train our model using Adam optimizer [49] with 2e -4 learning rate for 2e 6 training steps. • For pre-training, we represent our denoising network p θ1 as a Perceiver Transformer described in §A.2. MLP f t , which processes the language embeddings given by the CLIP encoder, MLP f y , which processes the historical video tokens, and MLP f x k are 2-layered MLPs (prepended by a layer norm [3] and with Mish activation). MLP f T i , which processes diffusion timestep k, is a 2-layered MLP (prepended by a Sinusoidal embedding and with Mish activation). Finally, we use a linear layer to project the encodings given by the Perceiver Transformer into the original dimension space of the input. • For fine-tuning, we represent our denoising network p θ2 as a GPT2 Transformer. Similar to the architecture of p θ1 , we first use MLPs with Mish activation to project different inputs into the same hidden dimension space, and then recover the encodings given by the GPT2 Transformer into the original dimension space via a linear layer. • We choose the sequence length H = 4 and M = 1 for future actions and videos prediction.• We set h = 20 which means that we predict future videos after 20 steps.• We set diffusion timesteps K = 100.• We run all the experiments on a single RTX 3090 machine.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide them in Appendix B and Appendix A.3.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the error bars for evaluation under different random seeds.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide these details in Appendix A.3.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts of our method in Appendix E.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite them throughout our paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We describe the implementation details of baselines used in our experiments as follows:• R3M-Diffusion. We borrow the pre-trained R3M encoder from https://github.com/ facebookresearch/r3m, and leverage the encoder to encode sing-view RGB images in Meta-World and multi-view RGB images in RLBench. Thus, we skip the pre-training process and directly train our discrete diffusion model on the robot data. The model architecture and hyper-parameters of R3M-Diffusion are almost the same as p θ2 . The encoded hidden representation encoded by R3M is denoted as z R3M , then the denoising network can be written as p θ (x a k-1 |x k , z R3M , l). • MTDIFF-P. We borrow the official codes of MTDIFF-P from https://github.com/ tinnerhrhe/MTDiff. In order to process high-dimensional images instead of low-dimensional states, we leverage the R3M encoder in R3M-Diffusion to obtain the visual representations. • VIDEO-MTDT. We use the language l to indicate tasks, which are encoded as a vector with size 2048 by the same CLIP encoder used in VPDD. We take the video tokens used in VPDD as the states. Then we follow the implementation from https://github.com/kzl/ decision-transformer/ to train Video-MTDT on the limited robot data. • VPDD-w/o-human. During the pre-training stage of VPDD, we remove the human videos and only use the robot videos for training. • SODA. Following the SODA paper [39] and open-sourced codes from https://github.com/ FutureXiang/soda, we first maintain an encoder E SODA equipped with the same Perceiver Transformer in p θ1 . Then during the pre-training stage, a representation z is first encoded by E SODA conditioning on e t and l, i.e., z = E SODA (e t , l). Then x k-1 is obtained via the following process:EQUATION)where Attn is an attention operation [84, 55] where the queries are formed from x, the keys and values from y. The encoder is trained end-to-end and to minimize the same loss term L vlb . During the fine-tuning stage, z xv 0 is replaced of z output by E SODA . The model architecture and other hyper-parameters during fine-tuning remain the same. Sample a batch B = (x v , x a , y, l) from human and robot videos, where x a ← ϕ 6:Sample diffusion timestep k from [1, K] with importance sampling strategy 7:x k ← q(x k |x 0 ), where x 0 = [x v , x a ] ▷ Eq. ( 11) and ( 4)8: L vlb = L 0 if k = 1, L k-1 otherwise.▷ Eq. ( 6)9: Sample a batch B = (x v , x a , y, l) from videos and discretized actions dataset.θ 1 ← θ 1 -η∇ θ1 L ▷Sample diffusion timestep k from [1, K] with a importance sampling strategy 7:x k ← q(x k |x 0 ), where x 0 = [x v , x a ] ▷ Eq. ( 11) and ( 4)8: L vlb = L 0 if k = 1, L k-1 otherwise.▷ Eq. ( 6)9:θ 2 ← θ 2 -η∇ θ2 L ▷ Adam optimizer 10: end for # Evaluation Process 1: Given a task, reset the environment 2: Obtain the initial video clips v 0 , language instructions l 3: for t = 0 to t max do 4:EQUATIONConstruct y ← e t = VQVAE-Encoder(v t )6:for k = K to 1 do 7: Sample x v k-1 ∼ p θ1 (x v k-1 |x k , y, l) and obtain z xv 0 from p θ1 encoding ▷ Eq. (SampleEQUATIONxk-1 = [x v k-1 , x a k-1 ] 10:end for 11:Obtain predicted videos via VQVAE-Decoder(x v 0 )12:Reconstruct executable action sequence[a t , • • • , a t+H-1 ] from x a 0 13:Execute the first action a t as the current action to interact with the environment RLBench. We use the official codes from https://github.com/peract/peract to generate a dataset for RLBench via a motion planner. Each demonstration consists of multi-view RGB images (i.e., front, left shoulder, right shoulder and wrist) and 8-dimensional actions including 3-dimensional transitions, 4-dimensional quaternion and a binary value about gripper openness. Following prior works [42, 75] , we extract macro actions (keyframe actions) from collected demonstrations and leverage networks to predict the keyframe actions instead of consistent continuous actions. Specifically, a set of keyframe actions {k 1 , k 2 , • • • , k m } ⊂ A is captured with a simple heuristic: an action is a keyframe if (1) the joint velocities are near zero and ( 2) the gripper open state has not changed. Each data point in the demonstration τ can then be cast as a "predict the next (best) keyframe action" task [44] . In this way, the sequence length of actions that need to be predicted is significantly reduced from hundreds of small steps to typically less than 10 macro steps. "screw in the light bulb" put in safe placement 3 "put the money away in the safe on the shelf" place wine placement 3 "stack the wine bottle to the of the rack" put in cupboard category 9 "put the in the cupboard" push buttons color 50 "push the button, [then the button]" insert peg color 20 "put the ring on the spoke" stack cups color 20 "stack the other cups on top of the cup" place cups count 3 "place cups on the cup holder" Table 3 : Language-Conditioned Tasks in RLBench [43] with various variations.We take Meta-World as a main benchmark to evaluate our method and baselines, which consists of 50 diverse manipulation tasks. The poses and positions of goals are randomly generated during evaluation. These tasks require an agent to identify the observed sing-view RGB images and reach the goals with the correct behavior. See Fig. 9 for a sample visualization of the tasks.We select 16 tasks out of 100 tasks from RLBench [43] that involve at least two or more variations to evaluate the multi-task and generalization capabilities of agents. Task variations include randomly sampled colors, sizes, counts, placements, and categories of objects. The set of colors include 20 instances: colors = {red, maroon, lime, green, blue, navy, yellow, cyan, magenta, silver, gray, orange, olive, purple, teal, azure, violet, rose, black, white}. The set of sizes include 2 instances: sizes = {short, tall}. The set of counts include 3 instances: counts = {1, 2, 3}. The placements and object categories are specific to each task. For instance, put in cupboard includes 9 YCB objects. In addition to these semantic variations, objects are placed on the tabletop at random poses. The tasks in RLBench require an agent to process multi-view RGB images properly and generalize to different variations that could be unseen in training. The details of variations for each task are referred to Table 3 .Generalize to unseen scenes. To evaluate the generalization ability of our method, we change the camera position and table texture in the Meta-World benchmark to generate out-of-distribution observations. Borrowing the official codes from https://github.com/RLAgent/factor-world, hand-insert-v2 32% 36% bin-picking-v2 78% 84% door-unlock-v2 100% 100%Table 4 : Average success rate of VPDD on unseen tasks.we set the camera at another corner position and generate unseen table texture randomly while rendering. See Fig. 10 for visualization.Generalize to unseen tasks. In the following, we show that VPDD can generalize on different tasks from the same domain. In experiments, during stage 2, we train the VPDD on 47 tasks on MetaWorld and leave 3 unseen tasks to test the generalizability. In stage 3, we fine-tune the pretrained model on these 3 tasks. We report the success rate of the final model trained over 0.2M gradient steps.The following experimental results in Table C .3 demonstrate the promising generalization ability of our model, as the performance gap compared with the oracle is very small. We present illustrative samples of robot videos generated using discrete diffusion model p θ1 in Fig. 3 and Fig. 11 . More samples are available at https://video-diff.github.io. It is noteworthy that VPDD is able to generate dynamic consistent future videos, incorporating information beneficial for low-level control while maintaining coherence across distinct perspectives. However, it is imperative to acknowledge that our primary contribution is not on generating high-quality videos with exceptional resolution and meticulous semantic details. Consequently, some blurriness may be observed in our predicted videos, exemplified by deviations in the gripper's pose. See Fig. 12 for a failure example.For future work, we could consider enhancing the coherence of videos across diverse views by leveraging the recently released Ego-exo4d data [28] . This extension encompasses considerations beyond solely temporal dynamics. Moreover, for the augmentation of video quality and the optimization of decision-making performance, it is worth exploring to scale up both the training dataset and model capacity.This paper presents work whose goal is to advance the field of Machine Learning. Specifically, we propose a novel pretraining-finetuning framework to make better use of a copious amount of human actionless videos. Since this method is easy to reproduce (as we will release our code soon) and exhibits the SOTA performance, it encourages future research to further advance this field. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. NeurIPS Paper Checklist† Correspondence to: Chenjia Bai (baicj@chinatelecom.cn). 


Decision: The paper proposes an approach to leverage action-less human videos to train robot policies. The paper received unanimous acceptance recommendation from the reviewers. The paper is very well motivated and presents a relatively simple approach with strong performance compared to the baselines. The AC follows the recommendation of the reviewers and recommends acceptance. However, reporting results beyond table-top manipulation would make the paper stronger. The authors are encouraged to incorporate the comments of the reviewers and revise the paper.

Reviews: review: paper_summary: paper_summary: The paper targets at transferring the knowledge from human videos to robotic manipulation policy. The paper proposes VPDD, a method that first pre-trains on video generation on both human and robot videos and then fine-tunes on robotic manipulation data with action labels. The intuition is, by learning to generate human and robot videos, the model obtains a "prior" (e.g., task planning, trajectory planning, etc.) of robotic manipulation which makes it easier to learn diverse manipulation tasks with even few-shot data. The authors compare VPDD to several robotic pre-training baselines and show non-trivial improvements over several benchmarks.
strength_weakness: strength_weakness: 1. The pre-training stage is trained jointly on human and robot data. The amount of human data is pretty abundant here but how many robot videos are we using here? If the robot videos are too few, will it affect the resulting pre-trained model from this stage? If the pre-training stage still needs a lot of robot videos, then it still faces the problem of lacking enough robot data. On the other hand, if the pre-trained model can still generalize pretty well without robot data in the pre-trianing stage, i.e., pre-training on human data only and fine-tuning on a small amount of robot data can work pretty well, it would be very helpful.

2. Since the main message of the paper is training on human data can help learning robotic manipulation policies, it would be good to see how the performance changes when trained on different amounts of human data, for example, a scaling curve with x-axis being the number of human videos and y-axis being the performance. It would be interesting to see if the performance can keep going up by feeding more and more human data, or is there somewhere it saturates.
questions: questions: 1. In Figure 4, VPDD w/o human videos is still better than other baselines. What's the reason here? Assuming VPDD-w/o-human has seen the same robot data as other baselines, what's the main factor that causes such a difference?

2. The authors tried VPDD on unseen scenes such as varying background and camera poses. What about unseen tasks? Assuming there are some tasks presented in human videos but not in robot videos, will the model learn to do these tasks, at least to some extent?

3. As a bonus, it would be good to evaluate some off-the-shelf pre-trained video generation models such as Stable Video Diffusion for future exploration, but I don't think it affects the main conclusions of this paper.
limitations: limitations: See above.
paper_summary: paper_summary: The authors propose utilizing large scale internet videos along with robot data to train a diffusion model which predicts video future frames conditioned on past video frames and language description of videos. This diffusion model operated on the vector quantized embeddings of the video frames. Subsequently, this diffusion model is adapted for action generation which is trained using a smaller amount of robot video data with action labels. The analysis is performed on the Metaworld suite and RLBench.
strength_weakness: strength_weakness: 1. **Missing Baselines:** How does the method compare against other generalist imitation learning methods like Octo, RT-1 etc? It's unclear how such methods compare in the finetuning setting. 
2. **Missing Ablations on amount of robot data used during pretraining:** While no action labels have been used during pretraining, the model still has access to robot trajectories. How does the method perform when no robot data is present during pretraining?
3. **Lack of robot experiments:** Does the model extend to any real robot setups?
questions: questions: 1. What are the computational costs of training the backbone?
2. Are the results on RLBench and Metaworld finetuned for every task or are in multi-task setting, i.e., do you have one model that yields the aggregate result on Metaworld or do you have one model per task?
limitations: limitations: 1. The method is currently limited to simulation and it remains to be seen if the results will transfer to real robots. 
2. The method still requires 20 demonstrations per task during pretraining. It’s not clear if this is a kosher assumption. What if behavior cloning was used on these 20 demonstrations? Would this yield higher results? If so, its unclear why one would use VPDD?
paper_summary: paper_summary: In this paper, the authors focus on the problem of policy learning via leveraging large video data without action labels. To this end, they employ a discrete diffusion framework which is first employed to predict future quantized video frames. Afterwards, the model is fine-tuned on robot data to learn the final policy. The authors evaluate their approach on the Meta-World and RLBench and demonstrate promising quantitative results over other approaches.
strength_weakness: strength_weakness: 1. Technical novelty is limited. The approach is similar to a lot of other approaches: quantize videos using VQVAE, predict future tokens using transformer, somehow account for missing action tokens.

2. The scope of domains for evaluation domains is somewhat limited. It would be interesting to see if the results hold on other robotic datasets.
questions: questions: 1. While I believe the current results in the paper are convincing, it still may strengthen the paper to demonstrate results on even more datasets.
limitations: limitations: Yes, the authors have properly addressed limitations.
paper_summary: paper_summary: The authors propose an approach that takes advantage of human-object interaction videos to improve policies. Their proposal involves three steps: 1) learn a video tokenizer common to both human and robot videos. 2) Learn a discrete-diffusion model that performs video token denoising on both human and robot videos. 3) Learn a discrete-diffusion model that performs action denoising on only robot data. They show this technique results in better performance as compared to baselines.
strength_weakness: strength_weakness: 1. The paper claims that pre-training on a large dataset of human-object interaction videos without actions is useful for learning actions for robots. However from the experiments it is not clear how much the human videos are adding to the policy performance? There is a significant amount of data difference between the models trained with or without human videos. Yet the performance between the two models is close. How does the model without human data perform given some more robot trajectories? In Fig. 1, for comparison without human videos the drop in performance in marginal. In Fig. 5 seems there is a bigger drop in performance which might be compensated by having multiple camera views in the robot training data. 

2. Ego4D videos are very different visually from the tasks and environments shown. It is not clear which tasks from Ego4D are helping in transferring good features for the robotics tasks. How much of the video tokenizer codebook is shared between the human and robot videos. Because the dataset statistics look so different I wonder if enough of the codebook is being shared by the two datasets?

3. Generating egocentric videos is challenging. If the model is good at generating egocentric videos then that in itself is a big contribution. What do the generated videos look like for the Ego4D dataset? Does performance on this task correlate with better downstream policy performance or is the fine-tuning step capable of recovering good policies on video prediction models that are not that good?

4. Another baseline to compare against would be a VideoMAE[1] kind of method to learn the video features on just robot dataset and then do diffusion on that. This will be a good way to make the case for a tokenizer based diffusion model to learn video features. 

[1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training. Zhan Tong, Yibing Song, Jue Wang, Limin Wang
questions: questions: See above weakness section.
limitations: limitations: Yes.

Comments: title: Response to Rebuttal
Respectful Reminder
Respectful Reminder
Thank you for rasing your score!
Thank you for the response
Thank you for raising the score!
comment: Thank you again for your time and efforts in reviewing our paper! We are glad to see you recognize our contribution. We will incorporate the reviewers' valuable suggestions and the added experiments into the next version of our paper.
I would like to thank you for addressing my concerns. I stand by my rating. I believe this paper is an interesting contribution to the conference and would like to see it accepted.
Dear Reviewer,

We hope this message finds you well. Since the discussion phase will end in approximately two days, we want to kindly follow up on the response we provided to your valuable comments and questions. Your feedback has been crucial in refining our work, and we greatly appreciate the time and effort you have invested in reviewing our paper. If you have any additional thoughts or questions about our responses, we would be more than happy to address them.

Thank you once again for your insightful review! We look forward to your further feedback.
Dear Reviewer,

We hope this message finds you well. Since the discussion phase will end in approximately two days, we want to kindly follow up on the response we provided to your valuable comments and questions. Your feedback has been crucial in refining our work, and we greatly appreciate the time and effort you have invested in reviewing our paper. If you have any additional thoughts or questions about our responses, we would be more than happy to address them. 

Thank you once again for your insightful review! We look forward to your further feedback.
Thank you for increasing your score! We sincerely appreciate your thoughtful and detailed feedback for improving the quality of our paper. We are grateful for your recognition of our efforts and honored by your positive evaluation!
I would like to thank the authors for their thoughtful response. My concerns are all addressed and I increase my score to 7.
Thank you for raising your score and for supporting our work! We will incorporate the reviewers' valuable suggestions and the added experiments into the next version of our paper.
Thank you for answering the questions and conducting the new experiments. I will be raising my score to weak accept as my concerns are addressed. Please add these results in the paper to provide the readers more context.