Leveraging RGB-Pressure for Whole-body Human-to-Humanoid Motion Imitation

Authors: humanoid  key points
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 05:31:05

Abstract: Whole-body motion imitation has gained wide attention in recent years as it can enhance the locomotive capabilities of humanoid robots. 
In this task, non-intrusive human motion capturing with RGB cameras is commonly used for its low-cost, efficiency, portability and user-friendliness. However, RGB based methods always faces the problem of depth ambiguity, leading to inaccurate and unstable imitation.
Accordingly, we propose to introduce pressure sensor into the non-intrusive humanoid motion imitation system for two considerations: first, pressure can be used to estimate the contact relationship and interaction force between human and the ground, which play a key role in the balancing and stabilizing motion; second, pressure can be measured in the manner of almost non-intrusive approach, which can keep the experience of human demonstrator. In this paper, we establish a RGB-Pressure (RGB-P) based humanoid imitation system, achieving accurate and stable end-to-end mapping from human body models to robot control parameters. Specifically, we use RGB camera to capture human posture and pressure insoles to measure the underfoot pressure during the movements of human demonstrator. Then, a constraint relationship between pressure and pose is studied to refine the estimated pose according to the support modes and balance mechanism, thereby enhancing consistency between human and robot motions. Experimental results demonstrate that fusing RGB and pressure can enhance overall robot motion execution performance by improving stability while maintaining imitation similarity.

Main paper: 
 introduction: 
 Humanoid have long been a focal point of robotics research, embodying engineering challenges related to human biology, cognition, and motor abilities [54] . Despite their human-like appearance often suggests higher interactivity and approachability compared to other forms of robots, traditional control methods relying predefined action sequences limits the adaptability and autonomy of robots in real-world environments [27, 47] . To address this limitation and broaden the range of actions achievable by humanoid robots, researchers have introduced motion imitation as a means to When performing an action where the CoM projection on the ground exceeds the foot support region, the robot falls. However, when the action, corrected by the pressure data, is input to the robot, the action is executed stably.Unstable Stable Pressureenhance their capabilities, facilitating humanoid interaction with the physical world in a manner similar to humans [11] . Nonetheless, there is the issue of creating a precise, efficient, and user-friendly demonstration format for human demonstrators during imitation [29, 34] . Previous methods have employed manipulators [20, 21] , force feedback devices (exoskeleton) [2, 19] , and high-precision motion capture equipment such as inertial motion capture [12, 24, 43] and optical motion capture systems [10, 18] . However, these tools are often expensive, operationally complex, poor portability, and cumbersome to wear. In contrast, low-cost, non-intrusive devices like RGB or RGB-D cameras offer significant advantages [5, 16, 45, 59, 67] .Regrettably, Methods relying solely on RGB often encounter challenges in accurately capturing 3D representation of human movements due to depth ambiguity and uncertainties in the footground contact relationship, resulting in motion that is unstable and potentially hazardous for robots. Fig. 1 depicts a classic movement within Tai Chi, known as the single-leg stance. The pose estimated from the RGB image perceptually describes the tilted state of the human body, but inaccurately determines the center of mass (CoM), leading to an unreasonable reference. When introducing pressure for guidance, the reference pose can be adjusted to incorporate a more reasonable CoM. This correction will greatly enhance the ability of humanoid robots to imitate human motion effectively.In this paper, we propose a novel non-intrusive method for wholebody human-to-humanoid motion imitation by integrating RGB and pressure information. Initially, we establish a systematic baseline comprising three modules: pose estimation, motion retargeting, and whole-body control. This system not only captures human poses in the real world but also retargets them into the new pose space of a humanoid robot, utilizing them to drive the robot in reality. Additionally, foot pressure data is obtained through pressure sensors embedded in insoles. Subsequently, the estimated pose undergoes refinement by determining the support mode and correcting the center of mass (CoM). To validate the efficacy of our proposed method, we compare the motion imitation performance of RGB and RGB-P based methods using similarity and stability metrics. Furthermore, we showcase motion imitation results not only in simulation but also in real-world scenarios. Through our endeavors, we demonstrate that integrating RGB and pressure information can facilitate human-to-humanoid motion imitation efficiently, accurately, and safely while maintaining user-friendliness.The contributions of our work can be summarized as follows:• We explore the feasibility and superiority of the pressure modality for non-intrusive robot motion imitation through both theoretical analysis and experimental validation. • We develop a non-intrusive human-to-humanoid motion imitation system with RGB and pressure. • We evaluate the performance not only in simulation but also in real environments, demonstrating that the proposed method enhances the completeness and stability of robot motion imitation tasks while ensuring motion similarity. 
related work 2.1 physics-based and multi-modal human motion capture: 
 The rapid development of monocular motion capture has attracted more and more widespread attention [6, 23, 26, 30, 41, 55] . However, due to the depth ambiguity of monocular images, the estimated human motion does not meet the real physical constraints. In particular, the principle of balance is not satisfied.To enhance the realism of virtual human movements, some researchers have incorporated physical constraints and corrections [31, 44, 49, 50, 60, 61] . These endeavors provide valuable priors for estimating accurate human body poses.On the basis of monocular motion capture, adding multi-modal information has also become a method to improve the plausibility and stability of virtual human. Von et al. [56] uses offline optimization method to estimate human body pose by fusing RGB and IMUs. While, Liang et al. [32] and Pan et al. [40] combine IMUs and human body 2D keypoints to obtain pose and translation. Recently researchers have paid attention to the importance of pressure for motion capture and tried to introduce pressure as supervision or constraint information in monocular motion capture. Scott et al. [46] curated a dataset containing real human action images, poses, and foot pressure data. However, their emphasis was primarily on estimating pressure from human body images. Tripathi et al. [55] achieves more precise motion capture by inferring physical characteristics of the human body, such as center of mass (CoM), center of pressure (CoP) and contact pressure, and constructs a human body dataset containing pressure for evaluation. Zhang et al. [63] introduced pressure as supervision to enhance the estimation of virtual human contacts, thereby obtaining more accurate human pose and translation. Pressure has emerged as our preferred non-invasive tool, offering a wealth of physical information.The teleoperation of humanoids through imitation has been prevalent for a considerable period [3, 11] . He et al. [16] categorized it into three types: task space teleoperation [2, 8, 9, 48, 65] , upperbody teleoperation [4, 14, 58, 62, 66] , and whole-body teleoperation [5, 12, 13, 16-18, 24, 38, 52, 53, 67] . We focus on the third type to explore the scalability of humanoid robots in whole-body motion capabilities.When transferring human motion to humanoid robots, the selection of motion capture modalities and motion representation is crucial for bridging the gap between humans and robots. The challenge is to identify the most relevant features that capture the essence of human actions. Various approaches have been explored in this area, including using CoM [2, 8, 13, 38] , joint rotations or positions [4, 12, 14, 67] , force [19, 37] , and other information for representation and mapping. He et al. [16] utilized image-based motion capture for teleoperation of humanoids, representing a commendable effort towards user-friendly teleoperation. However, their tracking of the lower body of the robot was not sufficiently precise. These uni-modal methods didn't integrate information from multiple dimensions. This leads to the loss of human motion features, and sometimes can impose redundant information unsuitable for imitation onto the robot. Whole-body teleoperation based on multimodal motion capture is relatively rare. It is worth mentioning that Dafarra et al. [9] integrated IMUs, pressure insoles, and optical sensors into their iCub3 avatar system, achieving effective teleoperation in task space. However, the complex wearing devices are not user-friendly for operators, and they do not emphasize the consistency and similarity of lower-body motion between humanoid robots and humans.In whole-body motion imitation, a crucial issue is how to balance the similarity and stability of the lower body. There are two approaches for mapping lower-body motions: (1) Robot-centric [12, 18, 42, 59 ]: This approach emphasizes robot control, where the human posture is transferred to the robot. Here, the robot autonomously selects or predefines the appropriate support based on the actual posture situation. (2) Human-centric [24, 28, 67] : In contrast, this approach prioritizes human support, where the support is obtained from the human and then mapped to the robot. Subsequently, the robot adjusts its posture based on the acquired support mode. These two approaches give dominance to one side without unifying the consistency of human-robot actions. Either humans make stiff leg movements to accommodate the robot's structure, or robots experience delays to adapt to human actions, significantly increasing the risk of instability, especially when the posture conflicts with the support mode.Therefore, we believe that Image-based motion capture can provide a good user experience, while integrating pressure sensing can ensure consistency between support and posture, thereby unifying human and humanoid motion. 
overview: 
 The ultimate goal of human-to-humanoid motion imitation is that humanoid robots can perform exactly the same motion as humans, i.e., P𝑅𝑜𝑏𝑜𝑡 P𝐻𝑢𝑚𝑎𝑛 . However, achieving end-to-end humanto-humanoid motion imitation is not a straightforward task and typically involves three main steps, as illustrated in Fig. 2 (a) . Pose Estimation serves as the initial step in capturing and representing the human pose in the real world P𝐻𝑢𝑚𝑎𝑛 , into an interpretable and executable format P 𝐻𝑢𝑚𝑎𝑛 . Since there are still differences in size, degree of freedom, and structure between human and humanoid robot, Motion Retargeting is necessary to transfer the estimated human body pose P 𝐻𝑢𝑚𝑎𝑛 into humanoid robot pose P 𝑅𝑜𝑏𝑜𝑡 . Considering the control strategy and balance constraints of the robot, it is crucial to determine precise drive parameters 𝑸 within the Whole-Body Control module to represent the final pose in the real world P𝑅𝑜𝑏𝑜𝑡 . In this paper, we develop a humanto-humanoid motion imitation system with non-intrusive sensors (e.g., RGB cameras and pressure insoles). The details are elaborated as follows.Pose Refinement PressurePose estimation utilizing monocular RGB camera have emerged as the predominant approach, owing to their non-intrusive nature, cost-effectiveness, and convenience. However, due to the depth ambiguity of RGB images, estimating 3D joint points from RGB images is ill-posed. To solve this problem, we introduce the parametric human model SMPL [33] to offer a robust human structure prior in natural human poses. We obtain the SMPL model parameters 𝚯 of the human body by leveraging the off-the-shelf monocular human pose estimation method CLIFF [30] . In this case, the sub-problem of pose estimation can be formulated as:EQUATIONwhere the estimated human pose P 𝐻𝑢𝑚𝑎𝑛 (𝚯) can be represented by {𝑱 𝐻 (𝚯), 𝑴 𝐻 (𝚯)}. 𝑱 𝐻 (𝚯) ∈ R 24×3 represents the positions of 24 joints, including head, hands, elbows, and feet [33] . 𝑴 𝐻 (𝚯) ∈ R 1×3 is the position of human body's CoM [55] . Both of them are calculated by the SMPL model parameters 𝚯. It is expected that P 𝐻𝑢𝑚𝑎𝑛 (𝚯) is as similar as possible to P𝐻𝑢𝑚𝑎𝑛 , which is measured by Euclidean distance 𝐿 𝑒 (•).As shown in Fig. 3 , there are large differences between human and humanoid robot in terms of topology, quantity, size and structure, etc. Motion retargeting is a essential process in Human-to-Humanoid motion imitation. The estimated human pose P 𝐻𝑢𝑚𝑎𝑛 (𝚯) undergoes a series of transformations, including reduction, scaling, and coordinate system alignment [24, 39] , to yield the humanoid robot pose P 𝑅𝑜𝑏𝑜𝑡 = {𝑱 𝑅 , 𝑹 𝑅 , 𝑴 𝑅 }. Here, 𝑱 𝑅 ∈ R 16×3 denotes the positions of the 16 robot joints, 𝑹 𝑅 ∈ R 16×3×3 denotes the rotation of the feet, while 𝑴 𝑅 ∈ R 1×3 represents the position of the humanoid robot's CoM.While the retargeted pose P 𝑅𝑜𝑏𝑜𝑡 exhibits a high degree of morphological similarity to the human pose, it cannot be directly applied to the humanoid entity due to the inability to meet real physical constraints, thus leading to potential issues such as falling or imbalance. To solve this problem, whole-body control is employed following the classical approaches [36] . Specifically, P 𝑅𝑜𝑏𝑜𝑡 is input into a differential inverse kinematics (IK) solver to calculate the joint control parameters 𝑸 by utilizing the robot's Jacobian matrix under the constraints of robot's kinematic balance. This can be formulated as min 𝑸 𝐿 𝑒 (P 𝑅𝑜𝑏𝑜𝑡 , P𝑅𝑜𝑏𝑜𝑡 (𝑸))EQUATIONwhere 𝑪 𝑟𝑜𝑏𝑜𝑡 represents a set of stable and safe motion. 
motion retargeting using pressure: 
 As shown in Fig. 2 (a), the pose estimated from RGB encounters challenges related to depth ambiguity and an unclear foot-ground contact relationship. Little attention is paid to verifying whether the RGB-estimated human body key points align with reality, which significantly impacts the stability of robot imitation. Combining Eq. 1 and 2, we introduce an additional constraint, where 𝝓 denotes the pressure between the human feet and the ground. It serves as a link within the function 𝛿 (•) to establish the mapping between the human body model and the humanoid. Building upon the analysis, we elaborate on our motion retargeting method using pressure. As shown in Fig. 2 To incorporate pressure into human pose representation, let's begin by analyzing the stability of human and humanoid. Both humans and humanoids have two support areas (feet), each bearing a portion of the body's mass. The pressure insoles measure the pressure distribution on the soles of each foot, which is presented in pixels.From the pressure and positions of the pixels, we can obtain CoP of each foot 𝑷 = {𝑷 𝑙 𝑓 , 𝑷 𝑟 𝑓 }. As shown in Fig. 4 , (•) 𝑙 𝑓 and (•) 𝑟 𝑓 means left and right foot. Taking the left foot as an example:EQUATIONEQUATIONHere, 𝜙 𝑖 represents the pressure value of each pixel, and 𝑿 𝑖 is its position. 𝑷 𝑙 𝑓 is the CoP of the left foot. The calculation method for the right foot is similar. In theory, we can also obtain the CoP of the whole body, written asEQUATIONwhere 𝑜 𝑝 is a pressure offset denoted as follows:EQUATIONThis value ranges from 0 to 1, with a value of 0 when all the pressure falls on the left foot and a value of 1 when all the pressure falls on the right foot.Next, we follow [24, 38, 42] to extend the concept of CoM offset and define it asEQUATIONHere, 𝑆 represents the support mode, 𝐿, 𝑅, and 𝐷 respectively denote left leg support, right leg support, and dual legs support.𝑜 𝑚 ∈ [0, 1].When the body is fully supported by the left leg, this value is 0, and when it is fully supported by the right leg, the value is 1.Under quasi-static conditions, the CoP can be considered equivalent to the projection of the CoM onto the ground [7, 22, 57] , so we have the following relationship when 𝑆 = 𝐷:EQUATIONEq. 9 indicates that under quasi-static conditions, the pressure offset 𝑜 𝑝 is equal to the CoM offset 𝑜 𝑚 . Hence, we can utilize the pressure offset 𝑜 𝑝 to correct the CoM offset 𝑜 𝑚 , ensuring that the human key points align with the pressure distribution.For a human demonstrator, we can derive an estimated 𝑜 𝑚 from RGB image and a real 𝑜 𝑝 from pressure data. However, the estimated 𝑜 𝑚 and the 𝑜 𝑝 are usually not equal. Following the analysis presented in Sec. 4.1, we apply a geometric method, as described in [24] , to refine estimated pose, ensuring consistency between 𝑜 𝑚 and 𝑜 𝑝 . Specifically, the pose of human P 𝐻𝑢𝑚𝑎𝑛 and humanoid P 𝑅𝑜𝑏𝑜𝑡 are refined respectively according to the support mode (i.e., Dual support and single support).Dual support. As illustrated in Fig. 4 , the ultimate target of pose refinement is to find a new offset ′ 𝑜 𝑚 satisfies ′ 𝑜 𝑚 = 𝑜 𝑝 . Assuming the right foot remains fixed, our goal is to locate a new left foot CoP ′ 𝑷 𝑙 𝑓 , which is constrained along the line connecting 𝑴 and 𝑷 𝑙 𝑓 . According to Eq. 8, we have:EQUATION)After solving the Eq. 10, we can obtain the refined CoP ′ 𝑷 𝑙 𝑓 . Then, the normal vector of the foot plane can be obtained as followEQUATIONFrom the normal vector, we can obtain the orientation of the foot plane byEQUATION)where 𝑰 is the identity matrix, [𝒏 𝑙 𝑓 ] × represents the skew-symmetric matrix of the normal vector, 𝜃 is the angle between foot normal vector and ground normal vector.Algorithm 1: Support Mode Discriminator Pseudo-codeInput :𝑆, 𝑜 𝑝 Output :𝑆 1 if S == L then 2 if 𝑜 𝑝 > 𝑡ℎ then 3 𝑆 ← D 4 else if S == R then 5 if 𝑜 𝑝 < 1 -𝑡ℎ then 6 𝑆 ← D 7 else if S == D then 8 if 𝑜 𝑝 ≤ 𝑡ℎ then 9 𝑆 ← L 10 else if 𝑜 𝑝 ≥ 1 -𝑡ℎ then 11 𝑆 ← RThe refinement process for the right foot follows the same principle. When 𝑜 𝑚 > 𝑜 𝑝 , we perform left foot refinement; otherwise, we perform right foot refinement. Single support. When the support mode 𝑆 = 𝐿 or 𝑅, there is no need to refine the position. It only needs to calculate the orientation of feet by current 𝑷 𝑙 𝑓 or 𝑷 𝑟 𝑓 according to Eq. 11 and 12. Note that, orientation information is not crucial for refining the human body, as we do not focus on human joint orientation. However, it is vital for refining the robot key points, as the robot's foot plane must align with the normal vector to ensure balance.Subsequently, the refined pose ′ P 𝑅𝑜𝑏𝑜𝑡 is fed into whole-body control module to drive the humanoid robot, facilitating the achievement of balanced and stable motion.There is little research that can precisely capture foot-ground contact using RGB, RGB-D, or even IMU data. However, pressure sensing presents notable advantages in this context, as it accurately captures changes in the body's CoP, thereby determining the human support mode. Our designed discriminator is illustrated in Algorithm. 1, where 𝑡ℎ is the threshold for the pressure distribution. We employ the concept outlined in Eq. 7. When 𝑜 𝑝 surpasses 𝑡ℎ, a switch in the support mode is activated.We believe that in the process of action imitation, accurate mapping of leg support modes is crucial, as it constitutes the essence of imitation. Otherwise, in cases where only the overall location is considered, the leg structure of the humanoid robot would be meaningless. 
experiments: 
 To assess the efficacy of our proposed human-to-humanoid motion imitation method, we utilize the PSU Taiji MultiModal (PSU-TMM100) Dataset [46] as the human motion demonstrator and the NAO humanoid robot [15, 25, 51] as the motion executor. We quantitatively and subjectively compare the similarity and stability of methods based on RGB and RGB-P modalities. Dataset. The PSU Taiji MultiModal (PSU-TMM100) Dataset [46] comprises 100 Taiji motion sequences performed by 10 human subjects, providing RGB video and foot pressure. Additionally, a wearable optical motion capture system (Vicon motion capture system (V-MoCap) [1] ) is used to obtain precise and accurate 3D markers on human body. We select this dataset for the following reasons: (1) It contains RGB and pressure modal data, aligning with the requirements of our method; (2) It offers accurate human body 3D marker data, serving as a good benchmark for non-intrusive pose estimation; (3) Tai Chi encompasses numerous balancing motions, posing significant challenges for humanoid robot. Platform. NAO humanoid robot [15, 25, 51] has 25 degrees of freedom (DOF). Its motion model is based on generalized inverse kinematics and performs well in tasks involving Cartesian and joint control, balance, and other functions. Our simulation environment utilizes Webots and qiBullet. We evaluate the the similarity of humanoid in qiBullet and stability in Webots. Metrics. We employ the Mean Per Joint Position Error (MPJPE) 𝔈 𝑚𝑝 𝑗𝑝𝑒 and the Frechet Distance 𝔈 𝑓 𝑟𝑒𝑐ℎ𝑒𝑡 for similarity evaluation, while for stability evaluation, we utilize Imitating Duration 𝔈 𝑙𝑒𝑛𝑔𝑡ℎ , CoM Deviation 𝔈 𝑐𝑜𝑚 , and CoP Deviation 𝔈 𝑐𝑜𝑝 .1) MPJPE 𝔈 𝑚𝑝 𝑗𝑝𝑒 : After aligning the estimated and groundtruth 3D joints at the root, we calculate the MPJPE 𝔈 𝑚𝑝 𝑗𝑝𝑒 (mm) to measure the accuracy of the estimated pose.2) Frechet Distance 𝔈 𝑓 𝑟𝑒𝑐ℎ𝑒𝑡 : Due to the mismatch in the joint numbers and link sizes between human and humanoid, we use the root-aligned mean per-joint Frechet distance 𝔈 𝑓 𝑟𝑒𝑐ℎ𝑒𝑡 (mm) to evaluate the similarity of all motion joints of the robot and the corresponding human groundtruth [62] , including the head, elbows, hands, and feet.3) Imitating Duration 𝔈 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒 : We evaluate the completeness of humanoid imitation by summing the total lengths of stable action sequences (𝔈 𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒 ) executed by the robot without experiencing falls. Throughout the experiment, we terminate the process whenever the absolute height of the head drops below 250 millimeters, indicating a fall by the robot.4) CoM Deviation 𝔈 𝑐𝑜𝑚 : To evaluate the stability of the humanoid during execution, we measure the mean global deviation 𝔈 𝑐𝑜𝑚 (mm) between the CoM projection and the ideal support region. Specifically, when the humanoid stands on dual legs, we calculate the distance from the CoM projection to the line between the ankle joints [39] . When the robot stands on a single leg, the distance is from the CoM projection to the ankle joint projection of the supporting leg.5) CoP Deviation 𝔈 𝑐𝑜𝑝 : We also use the whole-body CoP computed from the foot sensors. To calculate the global deviation of the CoP and ideal support region 𝔈 𝑐𝑜𝑝 (mm).Considering that V-MoCap can provide precision and accurate 3D markers on human body, we follow [35] to obtain the ground-truth of human body key points P𝐻𝑢𝑚𝑎𝑛 through SMPL model [33] . So that, we can compare the motion imitation similarity of our proposed RGB-P based method with the RGB based method through 𝔈 𝑚𝑝 𝑗𝑝𝑒 and 𝔈 𝑓 𝑟𝑒𝑐ℎ𝑒𝑡 . The results are demonstrated in Tab. 1. Given that robots cannot execute all human actions, the motion imitation always terminate early with falling down. For a fair and comprehensive comparison, we provide test results for two cases. The term of "All sequences" in the table indicates that both our method and the comparative method use their respective metrics at their highest completion. The term of "Normalized sequences" indicates normalizing the sequence lengths to a uniform length. For instance, if our RGB-P method achieves a completion of 2000 frames and the RGB method achieves a completion of 1000 frames, the metrics are calculated across all frames (i.e., 2000 for RGB-P method and 1000 for RGB method) in terms of "All sequences". Meanwhile, the metrics are calculated across the minimum number of frames (i.e., 1000 for both RGB-P and RGB methods) in terms of "Normalized sequences".The 𝔈 𝑚𝑝 𝑗𝑝𝑒,𝐻 in the first column primarily presents the error between the estimated human pose (i.e., P 𝐻𝑢𝑚𝑎𝑛 for RGB method and ′ P 𝐻𝑢𝑚𝑎𝑛 for RGB-P method) and the ground-truth P𝐻𝑢𝑚𝑎𝑛 . 𝔈 𝑚𝑝 𝑗𝑝𝑒,𝑅 represents the error between the target pose of the robot (i.e., P 𝑅𝑜𝑏𝑜𝑡 for RGB method and ′ P 𝑅𝑜𝑏𝑜𝑡 for RGB-P method) and the actual executed pose P𝑅𝑜𝑏𝑜𝑡 . For RGB method, there is no human key points refinement module, and support mode discriminator relies on a method from the estimated pose [28, 64] . It can be observed that our method does not improve the accuracy of pose estimation or the execution accuracy of the robot before and after adding pressure. This is understandable because the accuracy of human pose estimation depends on the precision of the RGB-based pose estimation algorithm we employ, and our pose refinement only corrects the CoPs of the human feet, which does not contribute to improving accuracy. It is worth noting that, the performance degradation of RGB-P method is less than 1 mm, which is negligible.The metric 𝔈 𝐹𝑟𝑒𝑐ℎ𝑒𝑡 indicates the similarity between the robot pose P𝑅𝑜𝑏𝑜𝑡 and the human ground-truth P𝐻𝑢𝑚𝑎𝑛 . From the experimental data, it can be observed that the similarity remains almost unchanged before and after the addition of pressure. This suggests that although our refinement leads to a decrease in the accuracy of human pose estimation, this decrease does not significantly affect the similarity of robot execution actions. This also implies that in the task of robot motion imitation, higher accuracy in pose estimation does not necessarily lead to better performance. The motion remapping module plays a more significant role in performance improvement. As shown in Table 2 , we evaluate the stability of our system. The metrics 𝔈 𝑐𝑜𝑚 and 𝔈 𝑐𝑜𝑝 suggest that integrating pressure data can notably enhance stability. This indicates the effectiveness of our pressure-based pose refinement and support mode discriminator. We also test the stability using human ground-truth (V-MoCap) as input, and the results show that our RGB-P method performed the best. This suggests that the accuracy of human pose estimation does not necessarily lead to improved stability. Instead, it is more crucial to find a mapping relationship that can reasonably establish human and humanoid pose and balance.To intuitively demonstrate the effectiveness of human-to-humanoid motion imitation using our proposed method, we conduct tests both in simulation and real-world scenarios.In Figure 5 , we present a motion sequence in PSU-TMM100 alongside corresponding pressure distribution. The imitation results obtained from the pose captured by V-MoCap, RGB, and RGB-P are compared. It is evident that pressure data provides accurate foot-ground contact information, leading to more precise mode recognition. Interestingly, our method even surpasses the V-MoCap approach in terms of action completion, showcasing the advantages of our RGB-P integration. Upon observing instances of falls in both V-MoCap and RGB methods, we note that both of them are caused by misjudgments of the support mode.The motion imitation performed by a real physical humanoid robot (i.e., NAO) is depicted in Figure 6 . For illustrative purposes, only five key frames are selected as examples. It can be observed that our human-to-humanoid motion imitation system can conducted on real robots and achieve good balance between motion similarity and stability. In detail, the methods using RGB-P and RGB are similar in terms of upper-body similarity. But regarding the whole body, the RGB-P method sometimes adjusts leg posture to achieve a support mode more similar to that of humans, resulting in a better imitation effect. As highlighted by the red circles, the human primarily supports her weight on the right foot, whereas the RGB-based method results in a humanoid pose distributing weight across both feet. Conversely, our method achieves a support pattern consistent to that of the human. 
conclusion and discussion: 
 In this study, we establish a multi-modal motion mapping system to explore the importance of pressure in humanoid robot imitation. By utilizing RGB and pressure data for humanoid robot motion imitation, we introduce a low-cost and non-intrusive method that enhances stability and balance. Leveraging precise pressure data, we refine the posture of both humans and robots, thereby enhancing their physical consistency. Through experiments in both simulated and real environments, our method demonstrates a significant improvement in stability while maintaining the imitation similarity. However, we must acknowledge that humanoid robot motion imitation still has a long way to go. Making humanoid robots perform routine actions like humans remains highly challenging. We suppose there are several key aspects to consider: Robot motion dataset. Current robot motion imitation faces a challenge due to the limited availability of comprehensive datasets. Most existing datasets are based on human motions, which often include actions beyond robots' capabilities. We believe that in the future, generating synthetic data using computer graphics and physics simulations could broaden the dataset's scope. Transfer learning from existing human motion datasets could also aid in adapting motions to new robot tasks, accelerating learning and improving performance. High dynamic motion imitation. In high-speed dynamic motion imitation, robots face increased complexity in dynamics and kinematics, requiring precise modeling and prediction. This involves understanding interactions between different body parts and the impact of the external environment. Hence, advanced perception and cognition systems are vital. We suppose that integrating various sensors like vision, sound, and touch for comprehensive environmental data is crucial. Additionally, efficient control algorithms leveraging deep learning and reinforcement learning are essential for real-time monitoring and swift adjustments to maintain stability and balance. Whole-body motion imitation. In humanoid whole-body motion imitation, a frequently discussed issue is how to enhance motion stability while ensuring similarity in lower-body actions. For bipedal structures, future focus should also be on maintaining accurate global location control while achieving precise leg imitation. These challenges underscore the need for more advanced sensor and feedback systems, as well as dynamic control algorithms. Real-time motion imitation. Robot teleoperation imposes higher demands on the real-time performance of motion capture and motion control algorithms. We believe that it is highly necessary to develop a low-latency real-time imitation system in the future, which should effectively integrate multiple processes including perception, learning, decision-making, action, and feedback to enhance the capabilities of robot teleoperation.  
 
 Appendix:  


Decision: This paper received five borderline acceptances from the reviewers, indicating a high level of agreement among them. The area chair supports the acceptance of this paper, as it introduces fresh ideas and goes beyond the common focus on combining vision and language. Instead, it explores the integration of vision with other modalities, specifically pressure sensing in this case. The paper is well-written, and its relevance to multimodal research is strong.

However, as noted by the reviewers, the authors should address the limitations of their work. Specifically, they should discuss the constraints of the scenarios considered, the data used, and potential overfitting issues. Additionally, they should consider comparing their approach with physics-aware methods.

Reviews: review: paper_summary: paper_summary: The paper is tackling the whole-body motion imitation challenge. The authors propose system that combining a RGB images with underfoot pressure to improve the posture balance when retargeting the motion to a humanoid robot. The system uses the insole pressure to determine the support mode such as single or dual sides. In the experiment, a PSU-TMM dataset is applied to simulate and compare approaches using different sensing modality quantitatively. In addition to the simulation, they further retarget to a physical robot to validate robot’s balance.
strength_weakness: strength_weakness: The paper is well structured and easy to follow. The flow chat is clear and informative. Their method details the calculation makes the work seem to be reproducible. In the experiment, the pressure does not improve the overall similarity, but their insight on the potential trade-off between balance and similarity due to the target subject’s capacity is rather interesting.
limitations: limitations: Not sure the second contribution that claims “human-to-humanoid motion imitation system with RGB and pressure” is adequate. This work seems does not include details on data collection approach such as camera setup, pressure sensor, but merely use a dataset available to implement the retargeting system. 

Minor: 
- What’s the initial input S for the first frame in Algorithm 1? Or how to initialize?
- No full name for acronym: IMU
paper_summary: paper_summary: This paper tackles the problem of imitation of human body motions under a monocular camera constraint. The hypothesis is that using a pressure sensor on the foot helps to resolve certain corner cases with monocular-only motion imitation. This is relevant for maintaining stable humanoid control / balancing motions as pressure readings can update the estimates for center of mass and meet constraints from physics. Results from both simuation and real world experiments are shown where adding the pressure modality to the estimation framework seems to improve baseline of RGB-only retargeting.
strength_weakness: strength_weakness: - Leveraging pressure for stablizing and correcting retargeted motions is definitely an interesting idea! Especially considering balancing humanoids and their associated control issues is still being figured out in the community.

- The experiments shown consider a broad range of pose estimation metrics and show improvement along all of them. Using the support mode discrimantor seems to be a simple but effective way to infer leg mappings and make the overall method more robust.

- The paper seems easy to follow and read. Supplementary materials and video also show the key results without much difficulty to the reader.
limitations: limitations: - In terms of experiment metrics, it would be worthwhile to consider for how many trials the control fails (referred to as "unstable" pose in figures/video). Over a sequence of motions to perform, I suppose if RGB-P helps correct the balance, this metric could showcase the success rate across the 100 different sequences in the dataset. 

- Adding retargeting of same human motions to different humanoid models (in simulation?) could be also be shown. The current results do show usefulness of RGB-P over RGB but don't seem to motivate why pressure might be a good modality to pursue across generalized humanoid motion retargeting. Another possible drawback is existing large scale human datasets don't seem to include this info.

- Additionally, a fair comparison to other imitation schemes (e.g. RGB-Depth) with the exisiting dataset could be added if possible. Pressure sensors are advocated as "almost non-intrusive", but a depth camera with modern RGB-D based pose estimation methods could also show similar results. It might be beyond the scope in terms of more experiments, so some smaller scale experiment/justification as to why a rgb-depth approach might not work could also help the paper.
paper_summary: paper_summary: The paper proposes a novel RGB-Pressure method for enhancing whole-body human-to-humanoid motion imitation, addressing the limitations of depth ambiguity in RGB-only approaches. It integrates RGB cameras for posture capture with pressure insoles to measure underfoot pressure during demonstrations, improving motion stability. A constraint relationship between pressure and pose is analyzed to refine estimations based on support modes and balance, increasing consistency between human and robot motions. The paper outlines a system comprised of pose estimation, motion retargeting, and whole-body control modules. Experiments validate that combining RGB and pressure data enhances robot motion stability and similarity to real human actions, advancing humanoid imitation capabilities.
strength_weakness: strength_weakness: Integrated Sensing Approach: It innovatively combines RGB cameras and pressure sensors (insoles) to overcome depth ambiguity in human motion capture, enhancing the accuracy and stability of humanoid robot imitations.

Consistency Enhancement: The proposed constraint analysis between pressure and pose refines estimated poses for better support and balance, significantly improving the consistency between human and robot motion sequences.

Systematic Framework: Establishes a comprehensive system with clear modules (pose estimation, motion retargeting, and whole-body control) that not only captures real-world poses but also adapts them to humanoid robots' operational parameters.

Performance Validation: Empirical results demonstrate the method's effectiveness in enhancing robot motion execution stability and maintaining imitation similarity to human actions, validated on benchmarks like HumanML3D.

Efficiency & Non-Intrusivity: Offers a low-cost, non-intrusive alternative to expensive or cumbersome motion capture methods, making it practical for broader adoption and enhancing human-robot interaction research.

Application Breadth: Its applicability spans multiple domains like animation, robotics, virtual reality, and rehabilitation, where natural human-like motion imitation is vital for engagement or functional tasks.
limitations: limitations: Depth Data Limitations: The paper should clarify the scope of the dataset used for training and validation, including diversity in activities, age groups, and body types. Limited or biased datasets might affect the generalizability of the model.

Quantitative Metrics for Pressure Integration: While qualitative improvements are mentioned, the paper could benefit from more detailed quantitative metrics evaluating the direct impact of pressure data on motion quality, such as balance, stability improvements in specific poses, or fall-risk scenarios.

Long-term Coherence and Fluidity: The paper focuses on momentary stability but lacks a detailed analysis of long-term motion coherence. How well does the system maintain fluidity and natural transitions over extended sequences, considering the influence of pressure input?

Pressure Sensor Dependence and Calibration: The reliability and calibration of pressure sensors could influence results. The paper should discuss sensor accuracy, drift over time, sensitivity, and how calibration procedures, and potential impact on the system's robustness.

User Study Details: The user study is mentioned but lacks depth. Sample size, demographics, user feedback variability, and a more subjective ratings on perceived naturalness or uncanniness would strengthen conclusions.

Comparison with State-of-the-art: The paper should thoroughly compare against the latest methods integrating additional sensory modalities like force plates or IMU data, not just RGB. Are there any that combine vision-pressure approaches to contrast?

Technical Depth of Analysis: The paper could provide a deeper technical dive into the algorithmic choices, explaining how pressure constraints are modeled, the interaction network architectures refined, and how these decisions were motivated.

Ethical and Safety Considerations: With robots imitating human motion, safety implications arise. Does the paper discuss potential hazards, risk mitigation strategies, or ethical concerns in creating realistic humanoid behaviors?
paper_summary: paper_summary: The paper presents a novel approach for whole-body human-to-humanoid motion imitation, leveraging both RGB camera data and pressure sensor information. The method aims to enhance the stability and accuracy of humanoid robots in mimicking human motions by integrating pressure data to refine pose estimation and ensure a more realistic interaction with the environment. The approach is tested in both simulated and real-world scenarios, demonstrating improvements in motion execution performance.
strength_weakness: strength_weakness: Innovative Multi-modal Approach: The integration of RGB and pressure sensors represents a significant advancement in humanoid robotics, providing a more robust solution for motion imitation by addressing depth ambiguity and enhancing pose accuracy.

Enhanced Stability and Accuracy: The use of pressure data to refine pose estimations and support modes significantly improves the stability and fidelity of the humanoid's movements, making this method particularly valuable for applications requiring high precision.

Comprehensive Evaluation: The paper provides a thorough evaluation of the system, including comparisons with RGB-only methods and testing in both simulated and real-world environments, which illustrates the practical benefits and robustness of the approach.

Potential for Broader Applications: The improved stability and accuracy in humanoid motion could greatly benefit fields such as assistive robotics, entertainment, and complex interaction environments where reliable and safe human-robot interaction is crucial.
limitations: limitations: Complexity and Cost Implications: Integrating pressure sensors, while innovative, introduces additional complexity and potentially higher costs in terms of hardware and maintenance, which might limit widespread adoption.

Limited Novelty in Concept: The concept of enhancing humanoid motion imitation by integrating sensory data is explored in previous research, such as the work presented on Ipman.is.tue.mpg.de, which also discusses the use of intuitive physics for motion understanding. Although the specific application of pressure data alongside RGB data is a valuable contribution, the foundational idea has precedents in the field.

Scalability Issues: Customizing this method to different humanoid platforms and varying environmental conditions could be challenging, potentially affecting the scalability and broad applicability of the proposed approach.

Potential Overfitting to Specific Scenarios: While the system is evaluated comprehensively within controlled environments and specific motion sets, extending its efficacy to a broader range of human activities and less controlled environments may require additional development and validation. This could limit the generalizability of the proposed method.
paper_summary: paper_summary: The paper introduces a novel method for enhancing whole-body motion imitation between humans and humanoid robots by integrating RGB cameras and pressure sensors. The paper addresses the limitations of RGB-based methods in accurately capturing human movements due to depth ambiguity, leading to unstable imitation. By incorporating pressure sensors to estimate contact relationships and interaction forces, the proposed RGB-Pressure (RGB-P) system achieves accurate and stable mapping from human body models to robot control parameters.
strength_weakness: strength_weakness: - The paper presents a novel approach of integrating RGB cameras and pressure sensors to enhance whole-body motion imitation between humans and humanoid robots, addressing the limitations of traditional RGB-based methods.
- The proposed RGB-Pressure (RGB-P) system demonstrates improved completeness and stability of robot motion imitation tasks while ensuring motion similarity, as evaluated in both simulation and real-world environments.
- The clarity of the methodology, experimental setup, and results in the paper contributes to its strength, providing a clear understanding of the proposed approach and its implications 3.
limitations: limitations: - The idea of adding a pressure sensor is essentially simple and incremental.
- Let humans wear a pressure sensor may be burdensome. Also, it cannot utilize a large amount of the existing videos on the web.
- Lack of comparison between Physics-aware vision-based motion imitation/learning. For example:
[1] Learning Physically Simulated Tennis Skills from Broadcast Videos, SIGGRAPH 2023
[2] Motion Inbetweening for Physically Simulated Characters, SIGGRAPH Asia 2022
- The paper lacks a comprehensive discussion on the limitations and challenges of the proposed method, particularly in addressing high-speed dynamic motion imitation and whole-body motion stability while ensuring similarity in lower-body actions.
- There is a need for a more detailed comparison with existing methods and approaches in the field of humanoid robot motion imitation, especially in terms of technical correctness and evaluation metrics.