Title: CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Physics-based Character Animation

Human-Object Interactions

Multi-Agent Cooperation
Abstract: Enabling humanoid robots to clean rooms has long been a pursued dream within humanoid research communities. However, many tasks require multi-humanoid collaboration, such as carrying large and heavy furniture together. Given the scarcity of motion capture data on multi-humanoid collaboration and the efficiency challenges associated with multi-agent learning, these tasks cannot be straightforwardly addressed using training paradigms designed for single-agent scenarios. In this paper, we introduce **Coo**perative **H**uman-**O**bject **I**nteraction (**CooHOI**), a framework designed to tackle the challenge of multi-humanoid object transportation problem through a two-phase learning paradigm: individual skill learning and subsequent policy transfer. First, a single humanoid character learns to interact with objects through imitation learning from human motion priors. Then, the humanoid learns to collaborate with others by considering the shared dynamics of the manipulated object using centralized training and decentralized execution (CTDE) multi-agent RL algorithms. When one agent interacts with the object, resulting in specific object dynamics changes, the other agents learn to respond appropriately, thereby achieving implicit communication and coordination between teammates. Unlike previous approaches that relied on tracking-based methods for multi-humanoid HOI, CooHOI is inherently efficient, does not depend on motion capture data of multi-humanoid interactions, and can be seamlessly extended to include more participants and a wide range of object types.

CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics

Abstract

Enabling humanoid robots to clean rooms has long been a pursued dream within humanoid research communities. However, many tasks require multi-humanoid collaboration, such as carrying large and heavy furniture together. Given the scarcity of motion capture data on multi-humanoid collaboration and the efficiency challenges associated with multi-agent learning, these tasks cannot be straightforwardly addressed using training paradigms designed for single-agent scenarios. In this paper, we introduce Coo perative H umanO bject I nteraction ( CooHOI ), a framework designed to tackle the challenge of multi-humanoid object transportation problem through a two-phase learning paradigm: individual skill learning and subsequent policy transfer. First, a single humanoid character learns to interact with objects through imitation learning from human motion priors. Then, the humanoid learns to collaborate with others by considering the shared dynamics of the manipulated object using centralized training and decentralized execution (CTDE) multi-agent RL algorithms. When one agent interacts with the object, resulting in specific object dynamics changes, the other agents learn to respond appropriately, thereby achieving implicit communication and coordination between teammates. Unlike previous approaches that relied on tracking-based methods for multi-humanoid HOI, CooHOI is inherently efficient, does not depend on motion capture data of multi-humanoid interactions, and can be seamlessly extended to include more participants and a wide range of object types.

1 Introduction

Imagine the scenario where you're moving home and seeking help from humanoid robots. However, certain items, like beds and sofas, are too large and heavy for a single robot to manage effectively. Despite significant advancements in learning and control of physics-based humanobject interactions [6, 43, 21, 37, 20, 45, 7] and humanoid robots performing complex operations [29, 19, 26, 2, 42, 41, 10, 9, 39], the area of multiple humanoids collaboratively transporting objects -particularly when a single individual may find it challenging to handle heavy or long items-remains relatively under-investigated. Some previous efforts [44] tried to address these challenges using tracking-based methods, capturing interactions between two characters during tasks like box carrying, and training policies to mimic these actions. However, obtaining comprehensive motion data on interactions involving multiple participants is costly, and tracking-based methods struggle to adapt to scenarios with varying object sizes and an increased number of agents. Another straightforward idea might be to directly train a multi-agent policy from scratch, using an approach

CooHOl

Multi-Agent

Transportation

and

Cooperation

Interaction with

In

Diverse

A Natural Style

similar to that for single agents. However, the expanded action sampling space significantly slows down the overall training process, preventing the policy from converging properly.

Inspired by how humans learn cooperation skills, i.e. beginning with mastering tasks independently and then developing strategies for collaborative efforts in coordination tasks, we propose a twostage framework for training multi-agent cooperation strategies, named Coo perative H umanO bject I nteraction, CooHOI . In the initial stage, we train a single-agent object carrying policy utilizing the AMP framework [25, 7]. To ensure agents focus significantly on the dynamics of the objects they are tasked with transporting, we include the state and velocity information of the object's bounding box in the agent's observation space. The second stage aims to transfer these individual carrying skills into collaborative strategies. When two humans collaborate to carry a long object, they typically hold it at opposite ends. Moreover, when one individual takes action, it affects the dynamics of the object, enabling the other individual to perceive this change and adjust their actions accordingly for effective cooperation. Therefore, we adjust the observation for both agents to focus on the bounding boxes located at the ends of the long objects. This refinement aligns with the observation space used during the single-agent training phase, effectively leveraging the previously developed single carrying skills. Additionally, the inherent rigid body characteristic of long objects facilitates implicit communication between agents. This setup allows an agent to adeptly adjust to their teammate's actions by observing changes in the object's dynamics , thereby enhancing coordination and cooperation in carrying tasks. The overall framework is illustrated in Figure 2.

To validate the effectiveness of our framework, we conducted experiments where we trained control policies for two humanoid characters to carry various long objects, such as boxes and sofas. Our results demonstrate that our framework enables these characters to exhibit natural-looking behaviors while successfully completing cooperative tasks, utilizing only motion capture data from one single agent. We compared our approach against the baseline method of training from scratch and performed detailed ablation studies to evaluate the impact of our design decisions, also testing the limitations of our framework. In summary, our main contributions are as follows: 1)We have developed an efficient and robust framework, CooHOI , for training physically simulated characters in cooperative object transporting tasks, demonstrating significant effectiveness. 2)We have established that utilizing object dynamics for communication proves to be an effective strategy in learning cooperative object transporting tasks.

2 Related Work

Physics-based Human-Object Interaction Motion Synthesis. Synthesizing natural and physically plausible human-scene interactions, such as humanoid characters sitting on chairs, lying on beds, and carrying boxes, is crucial for advancements in character animation and robotics. Physics-based methods leverage physics simulators [18, 35] to control characters modeled as interconnected rigid bodies through joint torques and deep reinforcement learning methods. To facilitate the training process, some strategies employ tracking-based techniques [16, 23, 1, 38, 20, 44], which rely on the availability of high-quality reference motions. This reliance often limits their application in humanscene interactions due to the scarcity of suitable data and affects their versatility across different scenarios. Recently, the Adversarial Motion Priors(AMP) framework [25] introduced the use of a discriminator to ensure that generated motions align with the distribution of reference motions. This approach has shown success in various downstream tasks [11, 24, 34], including human-scene interactions [7, 21, 37]. Despite these advancements, there has been limited focus on synthesizing cooperative behaviors among multiple characters interacting with objects-a gap that our work aims to address.

Multi-Character Control. While significant advancements have been made in synthesizing motions for single agents, the realm of multi-character animation remains relatively unexplored. Existing approaches predominantly rely on kinematic-based methods [36, 14, 32, 30, 5] and datasets of multicharacter interactions [13, 31, 4]. These methods, however, require high-quality interaction data and often fall short in ensuring physical plausibility or adequately handling multi-character cooperative interactions with objects. Physics-based techniques for character animation have primarily focused on aspects like crowd navigation [8, 27], limiting themselves to behaviors such as pedestrian movement and collision avoidance. Although [44] showcases interactions among multiple characters and object manipulation, such as carrying long boxes, these tracking-based approaches necessitate motion capture data of human interactions or human-object interactions and struggle to scale to an arbitrary number of agents or different types of objects. In contrast, our framework requires only single agent motion capture data for multi-character object transporting tasks and can easily extend to different types of objects and different numbers of agents.

3 Methodology

Figure 2 shows the overall framework of our approach. Our method utilizes motion data from individual agents and involves a two-stage learning process to create cooperative control policies. First, we train a policy for single-agent carrying tasks, using the dynamics of the manipulated object

Multi Agent Cooperation Learning

Large Objects

Small Objects

Shared

!

"

#

$

!

#

$

"

"

!

$

#

!

#

#

Policy

Motion Data

Policy

Discriminator

Discriminator

Policy

as feedback, as described in Section 3.2. Then, we use parallel training to develop cooperative strategies, with changes in the object's dynamics acting as a form of implicit communication. This is further explained in Section 3.3.

3.1 Preliminary

Physics-based Character Control. We formulate physics-based character control as a goalconditioned reinforcement learning task. At each time step t , the agent samples an action from its policy π ( a t | s t , g t ) based on the current state s t and the task-specific goal feature g t . When this action is applied to the character, the environment transitions to the next state s t +1 , and the agent receives a task reward r G ( s t , g t , s t +1 ) . To train control policies that enable characters to achieve high-level tasks in a natural and life-like manner, we adopt the AMP framework [25]. While this framework aims to optimize the expected cumulative task reward J ( π ) , it introduces a discriminator to encourage the character to produce behaviors similar to those in the dataset by providing a style reward r S ( s t , s t +1 ) . The agent's reward r t at each time step t is defined by r t = w G r G ( s t , g t , s t +1 ) + w S r S ( s t , s t +1 ) . More details can be found in our appendix and the original AMP paper [25].

Multi-Agent Reinforcement Learning. We formulate our cooperative task as a Partially Observable Markov Decision Process (POMDP) [15]. A POMDP with n agents is defined by {S , A 1 , · · · , A n , O 1 , · · · , O n , R , P , T } , where S is the state space, A i is the action space for agent i , O i is the local observation for agent i , and R is the shared reward function. Each agent uses its own policy π θ ( a i | o i ) to take action a i ∈ A i based on its local observation o i ∈ O i . The environment transitions according to the function P ( s t +1 | s t , a 1 , · · · , a n ) , where s t , s t +1 are states of time step t and t +1 , respectively. The agents then get a reward r t based on the states s t and s t +1 . The goal of multi-agent reinforcement learning algorithms is to jointly optimize the discounted accumulated reward J ( θ ) = E a t 1 , ··· , a t n ,s t [ Σ T t =0 γ t r t ] .

3.2 Single Agent Carrying Skills Training

In developing our approach for single-agent object manipulation, we introduce several advancements based on previous methods. We integrate the dynamics of the manipulated object into the observation space and introduce a reward function framework for object manipulation tasks. These enhancements allow the trained policy to easily adapt to multi-agent settings.

3.2.1 Enriched Goal Feature with Manipulated Object Dynamics as Feedback.

For successful object carrying tasks, we emphasize the critical role of using the dynamics of the manipulated object as feedback. These dynamics are captured through the eight vertices of the object o 's bounding box b ver t , its rotation angle b facing t , its velocity b v t and its angular velocity b w t , as described in Equation (1).

By incorporating these dynamics information into the observation, we establish a feedback mechanism that keeps agents continually informed about the outcomes of their actions. This also equips agents with the capability to react appropriately, whether engaged in single-agent tasks or collaborative multi-agent environments. Along with the state of the agent s t and the position of the target d pos t , we formulate the dynamics-aware task observation as:

3.2.2 Enriched Task Design facilitating Efficient Skill Transfer.

To facilitate the transition from single-agent to multi-agent object carrying tasks, we decompose the carrying process into three sub-tasks: walking towards the objects, lifting the object from the ground, and carrying the object to its intended destination. Consequently, the reward system is structured into three components: r walk , r held and r target .

To encourage the agents to choose the face of the object they will face while carrying, we introduce an additional goal feature called stand points x stand t . During training, these stand points are randomly allocated to positions directly in front of the various faces of the object at time step t . This strategy is

designed to facilitate multi-agent cooperative training, helping the agents learn to avoid walking to the long side of the object where it is difficult to grasp and carry. We then define r walk as the distance between the agent and the stand point, as specified in Equation (3).

Additionally, in scenarios where multiple agents are transporting an object, the object's considerable size often makes it difficult for agents to find an appropriate grip point for lifting. To address this, we introduce a novel concept called held points . Specifically, we select the geometric center h t of each end of the object as the held point and encourage the agents to interact at these points. The reward function r held, as defined in Equation (4), uses x hand t to represent the mean position of the agent's two hands. This design aids the transition from individual to collective task environments by encouraging agents to identify the held points at the geometric center of each end of the long object.

We then define r target to encourage agents to carry the object to the destination:

3.3 Cooperation Strategy Training

Upon mastering single-agent tasks involving object carrying, it is crucial to effectively transfer and further enhance these abilities to boost the cooperative learning process. Initially, we facilitate the skill transfer by guiding each agent to lift and transport one end of a large object. Subsequently, we refine the control policy through the application of the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm [40], leveraging the dynamics of manipulated objects for feedback and implicit communication.

3.3.1 Efficient Skill Transfer Using Dynamics Information.

Initially, we replicate the single-agent object-carrying policy for all agents and then fine-tune it within a cooperative framework. However, handling a long object poses a challenge due to its size, which complicates the agents' ability to identify suitable lifting points, thereby hindering the efficient application of their object-carrying skills. To address this issue, we encourage agents to observe the dynamics information-specifically, the state and velocity data of the bounding box at each end of the long object, as outlined in Section 3.2.1. This approach allows agents to simulate carrying a smaller box positioned at the long object's ends, using the dynamics of this 'smaller box" as feedback , similar to the single-agent skill training process. Additionally, we incorporate previous methodologies, such as stand points and held points described in Section 3.2.2, to enhance the smooth transition of skills.

Moreover, this design of dynamics as observation acts as implicit communication channel between agents. When an agent takes an action, the object dynamics will change, and because we use the manipulated object dynamics as feedback in single-agent training, the change in object dynamics will result in a corresponding change in strategy for the other agents. This method of implicit communication presents a straightforward yet effective approach for enhancing teamwork in multiagent settings. Furthermore, this framework proves to be adaptable with variations in the number of participating agents.

3.3.2 Cooperation Training using CTDE Scheme.

The non-stationary environment of multi-agent reinforcement learning, coupled with sample efficiency issues, presents a significant challenge in training agents to collaborate effectively. During the cooperation training phase, we continue to use a reward function similar to that used in the singleagent training phase and employ the centralized training and decentralized execution (CTDE) [15] scheme for coordination training. During the training phase, we utilize the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm [40] to develop cooperative strategies among agents. This approach involves updating the value function network V ϕ according to Equation (6), using the trajectories D that are accumulated and shared among all agents. More details can be found in our

appendix and the original MAPPO paper [40].

In addition, given the homogeneous roles of agents in the collaborative task of carrying long objects, we adopt the strategy of parameter sharing, a method proven to enhance performance across various cooperative tasks [3, 33, 40]. Specifically, we share the parameters of both the policy and value networks among all agents to improve the training of cooperative behaviors.

4 Experiments

We conducted extensive experiments to test the effectiveness and also the boundary of capabilities of our framework. The basic experiment setups are explained in Section 4.1 and in Appendix. We evaluate our framework on various object-carrying tasks in Section 4.2. To better understand the importance of different design decisions in our framework, we performed extensive ablation studies in Section 4.3. Since our method primarily focuses on interactions between characters and objects, we also provide extensive visual analysis and presentations to demonstrate our framework.

4.1 Experiments Setup

Datasets and Initialization. Our primary source of motion data is the AMASS dataset [17], which provides motions encoded in SMPL [22] parameters. We collected a total of four types of basic reference motion data, including 9 motions related to walking, 5 related to picking up, 4 related to carrying, and 5 related to putting down. To enhance the robustness of the carrying process, we randomly initialized the weight and size of the object, as well as the distance from the person to the destination. Specifically, for a single individual, the object's weight ranged from 5KG to 25KG, its size varied between 0.5 to 1.5 times its original scale, and the distances between the agent and the object, as well as between the object and the destination, ranged from 1 m to 20 m. To enhance the robustness of the carrying process, we randomly varied the weight, size, and of the object, as well as the distance from the person to the endpoint. Furthermore, considering that the process of multiple people carrying might involve situations where someone walks backward, we ensured that the single agent mastered the basic movements for multi-person collaboration by introducing additional motion data involving backward and side walking. More training details can be found in the appendix.

Get close to the sofa

Pick up the sofa

Put Down sofa to destination

Get close to the box

Pick up the box

'

Carry

'

box to destination

Metrics. Following InterPhys [7], we utilize Success Rate and Precision as the primary metrics. Specifically, a task is considered successful if the distance between the object and its destination is less than 0.2m. Precision measures the distance between the object and the target point across all completed tasks. Moreover, all evaluations are based on average results across 4096 environments and 4 random seeds.

4.2 Evaluation on Object Carrying Tasks

Comprehensive Evaluation on Box Carrying. We evaluate our policy in both single-agent and two-agent scenarios. Table 1 presents the performance statistics for the carrying task. Consistent with the settings described in Interphys [7], we randomize the weight and distance as outlined in Section4.1. It is important to note that InterPhys [7] includes the weight information of objects as part of the observation, enabling it to achieve a broader weight range. To ensure result comparability, as shown in Line 2, we also incorporated this information into individual training. The results indicate that, with a similar success rate, we increased the transportation distance and improved transportation accuracy, even when using discrete motion data rather than the costly whole-body motion capture data used in InterPhys [7]. As shown in Line 3, although removing the weight information reduces the weight range the agent can handle, it represents a more realistic setting and provides comparable baseline results for future research. Lines 4-5 show the results in multi-agent scenarios. Without using our CooHOI framework and simply employing parallel training for multi-agent tasks, the training fails. More analysis and visualizations will be provided in the following sections.

Extend to Diverse Daily Life Objects. Moreover, we are not satisfied with merely transporting simple boxes. Therefore, we have sampled nearly 40 common everyday objects from [6], including three types of single-person objects-Table, Armchair, and HighStool-as well as one type of twoperson object, the Sofa, as illustrated in Figure 3 and Line 1 in Figure 4. Additional visualizations of these objects can be found in appendix. Additionally, we present the results of quantitative

Success Rate and Steps of One-Agent-Policy and Two-Agent-Policy

120

20.0

20.0

20.0

One-Agent-PolicySuccessRate

Two-Agent-PolicySuccess Rate

17.5

120

Two-Agent-PolicySuccessRate

One-Agent-PolicySuccessRate

17.5

120

One-Agent-PolicySuccessRate

Two-Agent-PolicySuccess Rate

17.5

100

15.0

100

15.0

15.0

(%)

(%)

(%)

100

SuccessRate

80

12.5

Rate

80

12.5

Rate

12.5

Step

step

Step

10.0

S

Success

60

10.0

Success

80

10.0

60

7.5

7.5

7.5

40

60

5.0

5.0

5.0

40

20

20

Two-Agent-Policy step

One-Agent-Policy step

2.5

0.0

0

Two-Agent-Policystep

One-Agent-Policystep

2.5

0.0

40

Two-Agent-Policystep

One-Agent-Policystep

2.5

0.0

0

10

20

30

40

50

60

0.6

0.8

1.0

1.2

1.4

2.5

5.0

7.5 10.012.5 15.017.5 20.0

Weight of Box (KG)

Agent Walk Distance (m)

experiments in Table 2, which indicate that our policy can effectively handle most daily life objects with a high success rate. Due to the greater variability in the shape and weight of sofas, the accuracy in Line 4 shows a slight drop compared to the result in Line 4 of Table 1. It is important to note that the significant variations in object shapes make it challenging to learn a unified object representation with a limited amount of data, especially since our input is merely a simple bounding box. Therefore, for single-person transport, different types of objects need to be trained separately from scratch, similar to how we handle boxes. For multi-person transport, objects like sofas, similar to boxes, require fine-tuning based on the weights obtained from single-person box transport.

4.3 Ablation Studies

To evaluate and understand the importance of different design choices in CooHOI, we conduct a detailed analysis of scenarios involving single and multiple agents. This includes boundary analysis, which explores factors that lead to a decrease in the success rate during object carrying. Furthermore, to fully validate the scalability of our method, we also tested and included the results in a four-person scenario. To ensure the reliability of our results, our experiments continue to utilize the average of 4 random seeds as previously mentioned.

Single-Agent-Based Box Carrying. We have demonstrated our framework's effectiveness in Table 1. Moreover, we aim to further investigate the maximum potential of our approach. We examined the robustness of the single-agent policy from various perspectives, including object shape, scale, weight, and trajectory length, as shown in Figure 5. First, we observed that a single agent can handle approximately 13 kg of weight. Continuously increasing the weight makes it difficult for the agent to lift the object. As mentioned in 4.2, since our policy input does not include object density, it is challenging for the agents to generalize across different weights. When the weight reaches 20 kg, the accuracy drops to just 30%. Additionally, due to the limited reach of human arms and the lack of dexterous hands in our agent, it is challenging for the agent to lift boxes that are either too large or too small. To validate this hypothesis, we restricted the object's width to 1x while scaling the length and height to 1.5x. As shown by the green circle in the second figure of Figure 5, the success rate could reach 97.8% . Furthermore, our strategy is quite robust to distance variations and is generally unaffected by them.

Multi-Agent-Based Box Carrying. We conducted upper-limit testing on the multi-agent policy using a similar evaluation method as the single-agent case. As shown in Figure 5, the conclusions for two agents are similar to those for a single agent. The curves in the first figure demonstrate that with an increase in the number of agents, we can easily lift larger and heavier objects that a single person cannot handle, highlighting the necessity of having multiple agents. However, due to the increased complexity of coordinating two agents, boundary conditions have a more pronounced impact on the policy. For instance, objects that are excessively large or small can cause the policy to fail. As shown by the purple circle in the second figure of Figure 5, we conducted a similar experiment to the single-agent scenario. By restricting the object's width to 1x and scaling the length and height to 1.5x, the success rate increased from 0 to 88.67% .

Analysis of CooHOI Framework. To thoroughly investigate the contribution of CooHOI, we analyzed the results of each method mentioned in Section 3 separately, as shown by the training curves in Figure 8. The first factor is the influence of the Stand Point , which refers to whether an extra point is introduced in front of the object to encourage the agent to walk toward it. During the experiments, we discovered that without this, the agent sometimes fails to approach the shortest edge of the object, resulting in a lower hold reward. This leads to an incomplete lift and the subsequent inability to carry the object. Dynamic Observation is the second factor, indicating whether we use dynamic information as observation for each agent. Without it, the observation for each agent is limited to the state information of the long object. We found that without the dynamics information, the agent just stands in front of the object, seemingly unsure of what to do. Reverse Walk indicates whether the training process includes motion data for walking backward and a reward function focused on learning this movement. We found that if the policy for a single agent is restricted to only forward movement, training with two agents then leads to a deadlock state. As shown in Figure 8, the agents might be able to contact the box, but they cannot carry it to the destination. Initialization refers to whether the two-agent policy is initialized using the single-agent policy and then be fine-tuned. In our experiment, even with extended training duration, training the two-agent policy from scratch still failed to achieve successful carrying, as shown in Figure 8. Based on the results above, the absence of any of the aforementioned methods causes our policy to fall into a locally suboptimal solution, preventing the completion of the transportation task. Moreover, you can also find more interesting visual examples of the above failure cases while extending agent numbers from one to two in the appendix.

5 Conclusion and Limitation

In this paper, we present Cooperative Human-Object Interaction (CooHOI), a framework designed to address cooperative object transporting tasks through a two-phase learning approach. By initially focusing on individual skill mastery via the Adversarial Motion Priors (AMP) method, followed by a strategic transition to multi-agent collaboration using Multi-Agent Proximal Policy Optimization (MAPPO), our approach facilitates a sophisticated interplay of shared dynamics and implicit communication among agents, resulting in an efficient and generalized performance.

However, within the scope of this paper, our huamnoid characters lack dexterous hands, which limit their ability to manipulate slippery objects or perform more precise actions. We also utilize object bounding box information as the goal feature for our task, which limits our framework's capacity to generalize to a diverse range of object shapes. Additionally, our experiments are limited to humanoid characters in simulation, without any real-world deployment. In future research, we aim to incorporate dexterous hands to enable the manipulation of a wider variety of objects, as well as integrate active perception and navigational abilities to make our framework more generalizable.

Acknowledgments

This work is supported by Shanghai Artificial Intelligence Laboratory. Ziqin Wang is supported in part by Key R&D Program (2022ZD0115502), Natural Science Foundation (NO. 62122010, U23B2010), Zhejiang Provincial Natural Science Foundation under Grant No. LDT23F02022F02, Key Research and Development Program of Zhejiang Province under Grant 2022C01082, 'Pioneer" and 'Leading Goose" R&D Program (No. 2024C01161).

References

Yu-Wei Chao, Jimei Yang, Weifeng Chen, and Jia Deng. Learning to sit: Synthesizing humanchair interactions via hierarchical control. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 5887-5895, 2021.

Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 , 2024.

Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning,

ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 1989-1998. PMLR, 2021.

Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Three-dimensional reconstruction of human interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7214-7223, 2020.

Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Remos: Reactive 3d motion synthesis for two-person interactions. arXiv preprint arXiv:2311.17057 , 2023.

Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11374-11384, 2021.

Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scene interactions. arXiv preprint arXiv:2302.00883 , 2023.

Brandon Haworth, Glen Berseth, Seonghyeon Moon, Petros Faloutsos, and Mubbasir Kapadia. Deep integration of physical humanoid control and crowd navigation. In Proceedings of the 13th ACM SIGGRAPH Conference on Motion, Interaction and Games , pages 1-10, 2020.

Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858 , 2024.

Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436 , 2024.

Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-based character control. In SIGGRAPH Asia 2022 Conference Papers , pages 1-9, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Jogendra Nath Kundu, Himanshu Buckchash, Priyanka Mandikal, Anirudh Jamkhandi, Venkatesh Babu Radhakrishnan, et al. Cross-conditioned recurrent networks for long-term synthesis of inter-person human motion interactions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages 2724-2733, 2020.

Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.05684 , 2023.

Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In William W. Cohen and Haym Hirsh, editors, Machine Learning, Proceedings of the Eleventh International Conference, Rutgers University, New Brunswick, NJ, USA, July 10-13, 1994 , pages 157-163. Morgan Kaufmann, 1994.

Libin Liu, KangKang Yin, and Baining Guo. Improving sampling-based motion control. In Computer Graphics Forum , volume 34, pages 415-423. Wiley Online Library, 2015.

Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5442-5451, 2019.

Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470 , 2021.

Yutaro Matsuura, Kento Kawaharazuka, Naoki Hiraoka, Kunio Kojima, Kei Okada, and Masayuki Inaba. Development of a whole-body work imitation learning system by a biped and bi-armed humanoid. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 10374-10381. IEEE, 2023.

Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG) , 39(4):39-1, 2020.

Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Haofan Wang, Xu Tang, and Yangang Wang. Synthesizing physically plausible human motions in 3d scenes. arXiv preprint arXiv:2308.09036 , 2023.

Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10975-10985, 2019.

Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Exampleguided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) , 37(4):1-14, 2018.

Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG) , 41(4):1-17, 2022.

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG) , 40(4):1-20, 2021.

Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Learning humanoid locomotion with transformers. CoRR , abs/2303.03381, 2023.

Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, and Or Litany. Trace and pace: Controllable pedestrian animation via guided trajectory diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13756-13766, 2023.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. Deep imitation learning for humanoid loco-manipulation through human teleoperation. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids) , pages 1-8. IEEE, 2023.

Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418 , 2023.

Yijun Shen, Longzhi Yang, Edmond SL Ho, and Hubert PH Shum. Interaction-based human activity comparison. IEEE Transactions on Visualization and Computer Graphics , 26(8):26202633, 2019.

Mikihiro Tanaka and Kent Fujiwara. Role-aware interaction generation from textual description. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1599916009, 2023.

Justin K Terry, Nathaniel Grammel, Sanghyun Son, Benjamin Black, and Aakriti Agrawal. Revisiting parameter sharing in multi-agent deep reinforcement learning. arXiv preprint arXiv:2005.13625 , 2020.

Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-9, 2023.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Jungdam Won, Kyungho Lee, Carol O'Sullivan, Jessica K Hodgins, and Jehee Lee. Generating and ranking diverse multi-character interactions. ACM Transactions on Graphics (TOG) , 33(6):1-12, 2014.

Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. arXiv preprint arXiv:2309.07918 , 2023.

Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hierarchical planning and control for box loco-manipulation. arXiv preprint arXiv:2306.09532 , 2023.

Haoru Xue, Chaoyi Pan, Zeji Yi, Guannan Qu, and Guanya Shi. Full-order samplingbased mpc for torque-level locomotion control via diffusion-style annealing. arXiv preprint arXiv:2409.15610 , 2024.

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , 2022.

Chong Zhang, Wenli Xiao, Tairan He, and Guanya Shi. Wococo: Learning whole-body humanoid control with sequential contacts. arXiv preprint arXiv:2406.06005 , 2024.

Qiang Zhang, Peter Cui, David Yan, Jingkai Sun, Yiqun Duan, Arthur Zhang, and Renjing Xu. Whole-body humanoid robot locomotion with human reference. arXiv preprint arXiv:2402.18294 , 2024.

Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard PonsMoll. Couch: Towards controllable human-chair interactions. In European Conference on Computer Vision , pages 518-535. Springer, 2022.

Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. Simulation and retargeting of complex multi-character interactions. arXiv preprint arXiv:2305.20041 , 2023.

Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. arXiv preprint arXiv:2305.12411 , 2023.

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5745-5753, 2019.

Appendix

In this section, we categorize our discussion into three main parts. Initially, we delve into the sources and processing methods for motion data used in training. Following that, we explore how observations are constructed and how reward functions are established. Finally, we describe the implementation details including physics simulation and hyperparameters in network training.

A Sources and Processing of Motion Data

We collected a total of four types of basic reference motion data, including 9 motions related to walking, 5 related to picking up, 4 related to carrying, and 5 related to putting down. All these data are in SMPL format and recorded at 30 fps over 139 frames. They all originate from the ACCAD subset of the AMASS [17] dataset. Additionally, to ensure the stability of cooperative tasks involving multiple individuals, we included data for sidewalk and reverse carry motions. The sidewalk data comes from the CMU subset within AMASS, while reverse carry data was scarce. Therefore, we created reverse carry data by reversing the process of the carry data. In total, we used 26 motion data as references. Additionally, we performed a simple visualization of the extended objects as in Figure 6, which sampled from dataset [6].

ArmChair

HighStool

Table

Sofa

B Task Formulation

We formulate our approach as goal-conditioned reinforcement learning. At each discrete step t , the policy π ( a t | s t , g t ) generates an action a t , based on the current state s t and a goal-specific feature g t . Following this action, the environment transitions into a subsequent state, and the agent receives a reward r t . An episode concludes either after reaching a predetermined length or if conditions for early termination (ET) are met. Further details are provided below.

B.1 Task Observation

The observational for the task is divided into two primary elements: the state feature s , which encapsulates the character's bodily configuration, and the goal feature g , which pertains to tasks involving object manipulation.

The state feature s is constituted by a 225-dimensional vector, encompassing:

Height of the root: 1 dimension.

Rotation of the root: 6 dimensions.

Linear and angular velocity of the root: 6 dimensions.

Position of local joints: 42 dimensions.

Rotations of local joints: 84 dimensions.

Linear and angular velocity of local joints: 84 dimensions.

While the root height is measured in the global reference frame, all other components are defined in the frame local to the character. Rotations follow a 6-dimensional representation for continuity [46]. The simulated character aligns with [25, 24, 7, 21], featuring 12 internally movable joints and a total of 28 degrees of freedom.

The goal feature g comprises a 75-dimensional vector, including:

Position of the object: 3 dimensions.

Rotation of the object: 6 dimensions.

Dynamics of the object, which cover the bounding box position, along with linear and angular velocities: 33 dimensions.

Target location: 3 dimensions.

Target orientation: 6 dimensions.

Dimensions of the target's bounding box: 24 dimensions.

These are measured in the frame local to the character.

B.2 Reward Functions

The agent's reward r t at each time step t is defined by

Follow the formulation of the AMP framework [25], the style reward r S is calculated according to the discriminator:

And the discriminator is trained by the following objective:

The task reward function r G is generally segmented into three components, as in Equation (10): 1) r G walk , which encourages the agent to approach the object intended for manipulation. 2) r G held , which encourages the agent to align the center of its hands with the center of the box. 3) r G target , which encourages the agent to transport the object to the specified destination.

The walk reward r G walk is formulated as Equation (11), where x standing t denotes the position of the standing point near the object, v ∗ denotes the target velocity, and d ∗ denotes the desired direction from root to the object.

The held reward r G held is formulated in Equation (12), where x hand t denotes the center of the agent's two hands and h t is the position of the object helding point.

The target reward r G target consist of two parts, r carry and r face, as described in Equation (13).

The face reward r face guides the agent to walk either forwards or backward. As shown in Equation (14), this is achieved by comparing the agent's velocity direction with its orientation relative to the endpoint's location, thereby cultivating the agent's proficiency in bidirectional locomotion.

The carry reward r carry, is designed to guarantee that the object is delivered to the precise location at a specific angle. As outlined in Eq. 15, we constrain the agent's movement direction, alongside the proximity to the end destination and the intended angle. Within this context, x ∗ t signifies the 3D coordinates of the destination, while p ∗ t represents the 2D destination coordinates. Similarly, p root t indicates the 3D position of the agent's root. Furthermore, rot ∗ designates the object's desired orientation.

where

B.3 Reset and early termination condition

An episode ends either after reaching a predetermined duration or upon the activation of early termination (ET) conditions. During our experiments, we observed that lower object heights could lead to kicking actions, where the agent tend to kick the object to destination, significantly slowing down the training process. To address this, we assess the object's velocity and height to determine the presence of kicking phenomena. If the height of the object is lower than 0.3m and its velocity in x-y plane is greater than 1m/s, the kicking early termination (KET) condition is triggered. Experimental results show that this strategy significantly stabilize the training process.

C Implementation Details

C.1 Training Details.

Adopting the methodology of AMP [25], we develop a low-level controller encompassing both policy and discriminator networks. The policy network is bifurcated into a critic and an actor-network, each initiating with a CNN layer and proceeding to two MLP layers configured with [1024, 1024, 512] units. The discriminator network is similarly structured, featuring two MLP layers with [1024, 1024, 512] units. We select PPO [28] as the primary reinforcement learning algorithm, coupled with the Adam optimizer [12] at a learning rate of 2e-5. The only difference between the multi-agent setting and the single-agent setting during training is whether a pre-trained weight is loaded. Our experiments are conducted on the IsaacGym simulator [18] using a single Nvidia GTX 3090Ti GPU. We run 4096 parallel environments across 15,000 epochs, which takes approximately 15 hours to complete.

C.2 Hyperparameters

Following previous work[25, 7, 21], we use the Isaac Gym simulator [18]. The simulation runs at 60Hz and the control policy runs at 30Hz.

Besides, the hyperparameters we used in the training process is detailed below:

D More Ablation Studies and Visulizations.

Failure case visualization. Here, we conducted a visual analysis of the fail cases. First, for the case lacking a stand point, we can clearly see that the agent moves towards the nearest face, even though it is not the shortest edge, which leads to the agent's inability to carry the object. In the second image, in the absence of dynamic input, we observe that the agent stands still, unable even to squat. In the third image, which depicts the scenario without reverse walking, the agent is able to lift the box, but because it cannot learn the backward gait, the two agents end up pushing the box against each other, causing a deadlock.

wrong face

stand still

deadlock

叮

,

⼀

8

[

乃

⽚

䯧

π

ㄩ

⼀

s

셅

]

⼀

l

!

-

A

No stand point

No Dynamic Observation

No Reverse Walk

Figure 7: Some visualization on failure cases. "Stand point" means a leading point behind the object to encourage the agent to walk to the object. "Dynamic Observation" means that each agent has its unique input. "Reverse Walk" indicates whether a single agent possesses the skill to walk backward. Without any of the methods we propose, the policy cannot be successfully trained.

Ablation on performance of CooHOI under noisy scenarios. In CooHOI, all state information is provided using ground truth data from simulators. However, in real-world settings, input data is often noisy and prone to errors. To evaluate the robustness of our framework in such conditions, we introduce random noise into the observation space of our policy and assess its performance under observation noise, as shown in Table 4.

Training reward curves. To enhance visualization for ablation studies, we plot the training curves for both the carry reward and held reward in the two-agent training setup. The results, shown in Fig 8, illustrate the effectiveness of our CooHOI framework.

HeldRewardCurve

CarryRewardCurve

0.30

0.20

0.25

0.15

0.20

Reward

Reward

0.15

w/oDynamicObservation

0.10

w/oDynamicObservation

w/oStandPoint

w/oStandPoint

0.10

w/oInitialization(Scale4)

w/oInitialization(Scale 4)

w/oReverseWalk

0.05

w/oReverseWalk

0.05

CooHOI

CooHOI

0.00

0.00

0

2000

4000

6000

8000

0

2000

4000

6000

8000

Epoch

Epoch

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly state our assumptions and contributions in the abstract and introduction. These claims match the experimental results shown in our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations of our methods at the last section of our main paper. We also test the influence of different factors in our work in ablation studies.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We propose a novel framework for cooperative object transporting tasks. Our work dows not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We fully described our method's details in our paper and also provided the code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The motion capture data used in our work is a public dataset, and our code is provided in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We fully described the training and test details of our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We reported error bars and random seeds of our experiments in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We described type of computing resources, time of execution of our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: We mainly focus on addressing cooperative object transporting tasks of humanoid characters, there may not be any societal impact of our work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: Our work does not have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credited and followed the licence of the code and data we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We are not releasing new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on learning multi-human cooperative object manipulation, specifically the collaborative carrying of objects.\nThis paper proposes a two-stage method to learn collaborative object carrying. In the first stage, the agent learns single-person object carrying from motion capture data and heuristic-based task rewards following the Adversarial Motion Priors (AMP) framework. In the second stage, the agent policy is fine-tuned using multi-agent reinforcement learning to learn multi-agent collaborative object carrying.\nExperiments show that the trained policy can successfully control the agents to complete collaborative objects carrying tasks given objects of varying shapes and weights.\n\nSTRENGTHS:\n1. The proposed cooperative interaction learning framework can learn multi-human collaborative behaviors requiring only single-human motion capture data. In addition, the learned policy has better generalization compared to tracking-based methods.\n\n2. The implicit communication via object dynamics unifies the observation in the single-human and multi-human stages, reducing the gap between the two stages and facilitating the learning of collaborative object carrying.\n\n3. This paper presented extensive ablation study and boundary analysis to show how and to what extent the proposed method works.\n\nWEAKNESSES:\n1. This paper simulates an oversimplified humanoid without hand modeling considering the fact that it works on object manipulation problems, for which the hands play essential roles. The authors also recognize this issue leads to capacity limitations.\n\n2. The agent observation features may not be sufficient for more complex manipulations. In the presented paper, the agent observation mainly consists of the self-motion and cropped object bounding box features. However, more complex collaborative tasks like carrying articulated objects and assembling a sofa require a global overview of the object instead of only a local observation. In addition, the agent also needs to observe the collaborators to avoid interference such as grasping at the same point.\n\n3. The proposed method needs to train a separate policy for a category of objects with similar shapes (L246-247), which indicates a generalization problem.\n\nQUESTIONS:\n1. How the object weights can affect the humanoid in the simulation? Is there an upper bound for the joint force of the simulated humanoid? How does the humanoid apply forces to the object in the simulation?\n\n2. Are the standing point and held point also part of the goal inputs? If not, during the training, the same motion sequence can have different rewards given differently sampled standing points? How are the standing points sampled given an arbitrary object mesh?\n\nLIMITATIONS:\nThe author discussed the limitation of lacking dexterous hands, which limits the agent interaction capacity and is left as future work. \nPlease also refer to the weakness discussion above.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work address the problem of multi-character collaboration for object transporting tasks. Different from previous works approaching the multi-character HOI task with tracking-based method, this work learns a physics-based multi-agent policy with reinforcement learning. Instead of directly training a multi-agent policy from scratch, the authors suggest using a two-phase learning approach: CooHOI uses a two-phase learning approach: individual skill acquisition through Adversarial Motion Priors (AMP) and then collaborative learning with Multi-Agent Proximal Policy Optimization (MAPPO). Agents learn to coordinate implicitly by responding to changes in object dynamics caused by others.\n\nSTRENGTHS:\n* The manuscript is well-written, the methodology is clearly presented and easy to follow.\n    * This paper is addressing an under-studied and important application - the physics-based multi-agent cooperation HOI, where most previous works explores in the single-agent setup.\n    * The proposed two-state learning pipeline is reasonable and few important designs including the stand point, held point shows to be effective in addressing this multi-agent cooperation tasks.\n    * The ablation study and analysis provide comprehensive analysis and capacity boundary test for the proposed pipeline and these provide useful insights for the community.\n\nWEAKNESSES:\n* This paper build a framework for multi-agent cooperation policy learning with AMP for single-agent policy pertaining and MAPPO for cooperation skill learning. The technical contribution to the community is minor.\n* The method include few heuristic designs to simplify the problem, for example, the manually defined held point, and object bounding box observation, making it not easy to generalize and scale to more diverse objects.\n* The current framework can be hardly generalise to objects with different size and different shape, and the human motion styles are quite limited.\n        * Regarding generalization to objects weights and scales, are the results presented in Figure 5  the transferring results to different weights and scales or the per-object training results.\n        * Regarding the motion styles, in the training dataset, limited motion sequences are involved with only walking, lifting up and putting down, would the proposed method build upon AMP be able to cover more diverse skills, and this could be helpful to generate more natural and diverse HOI motions.\n* It is presented that when increasing the number of characters, the policy has problem in putting the object down due to the absence of dexterous hands and limited friction. Could the author elaborate more on the major bottlenecks encountered when dealing with a larger number of characters? Assuming that the friction parameter can be adjusted in the simulator and should not be a significant issue for motion generation tasks purely within the simulation, what specific contributions could dexterous hands make to this scenario? Additionally, while the introduction of dexterous hands would undoubtedly enhance the agent's capabilities, what other factors should be considered?\n\nQUESTIONS:\nSee more in the weakness section :-)\n\nLIMITATIONS:\nYes, the authors addressed the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a novel framework, Cooperative Human-Object Interaction (CooHOI), aimed at tackling the problem of multi-agent object transportation. The framework consists of two phases: initially, a single agent learns to perform tasks, followed by multiple agents learning to collaborate through shared dynamics of manipulated objects. The utilization of shared object dynamics has proven effective in learning cooperative tasks. The authors conduct comprehensive experiments to validate the efficacy of their approach and explore its capability boundaries.\n\nSTRENGTHS:\n* The idea of sharing manipulated object dynamics among multiple agents as implicit communication to facilitate multi-agent collaboration, is intuitively appealing and has proven effective. This approach aligns with object-centric concept, designed to leverage object dynamics to guide agent actions and promote collaboration. It provides valuable insights for future research.\n* The authors conduct extensive experiments to validate the effectiveness of their approach, including many ablation studies to assess the impact of different design choices, as well as exploration into the framework's capability boundaries.\n* The paper is well-organized and easy to understand.\n* The inclusion of diverse visualizations enhances qualitative comprehension of the method's performance.\n\nWEAKNESSES:\n* It appears that the shapes and categories of the manipulated objects are not very diverse, potentially constraining the method's ability to generalize to novel objects. I recommend that the authors summarize the counts of shapes and categories used in the experiments.  Additionally, if the shapes of the manipulated objects are limited and the grasp actions are abstracted in the simulation, the authors are encouraged to compare CooHOI with some heuristic methods. An intuitive heuristic baseline could involve having two agents grasp each end of a long object, and then move at the same speed to the target position.\n* While the authors provide detailed explanations in the appendix, it would enhance comprehension if they provide simple explaination when introducing a new concept in the main paper. Here are some examples:\n  * In Section 3.1, the authors introduce the task-specific goal feature $g_t$ without explanation.\n  * While it is understandable that the proposed style reward can help the method produce behaviors similar to those in the dataset, it would be better to provide a simple explanation of how to evaluate such similarity.\n  * It would be beneficial to provide a brief introduction to the baseline InterPhys in one or a few sentences.\n\nQUESTIONS:\n* While acknowledging the idea of utilizing the object’s dynamics as implicit communication between agents, such communication is not synchronous. Specifically, agents adjust their actions only after one agent changes the object dynamics, resulting in a delay between the change in object dynamics and the response actions taken. I wonder if this delay could lead to jitter of the manipulated object. If not, could the reviewers discuss why such jitter would not occur?\n* The held point is described as the geometric center of each end of the object. However, in Figure 1, when cooperatively carrying a large box, the characters grasp the edges of the box instead of the center of its face. I am unsure if I misunderstood the definition of the held point, or if the current definition is inadequate, leading agents to choose other positions they find more suitable for completing the task.\n\n* In the second row of Figure 4, I am curious about why the four agents fail to put down the box. The authors attribute this failure to the absence of dexterous hands and limited friction, however, similar limitations exist in two-agent manipulation tasks. Besides, the multi-agent put-down operation closely resembles the multi-agent pick-up operation (but in reverse), while the agents can successfully pick up the box, they struggle to put it down. Could the authors provide more discussion or visualizations for better understanding?\n\nLIMITATIONS:\nThe authors have provided a detailed discussion of limitation and failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for multi-agent cooperative manipulation in the context of humanoids carrying and transporting large furniture. This task is decomposed into several steps. First, they train a single humanoid to learn how to hold and carry relatively small objects. The agent is trained with ground-truth object information and AMP for natural humanoid behavior. Next, they train a multi-agent policy initialized from the single-agent policy for transporting larger objects. The authors present several ablations and report high performance compared to baseline methods in simulation.\n\nSTRENGTHS:\nThis paper studies a practical task and solves it with a well-designed engineering system. It is also well-written, making it easy to read.\n\nThe paper presents comprehensive ablation experiments on both the single-agent and multi-agent policy parts, revealing important information for the framework.\n\nThe framework can learn natural humanoid behavior.\n\nWEAKNESSES:\nI’m curious about the robustness of the proposed system. If I understand correctly, the bounding box information is given as ground-truth parameters. However, in reality, these quantities are far from perfect.\n\nThe task is restricted to moving and transporting large furniture-type objects.\n\nQUESTIONS:\nIn Table 1, what does the “w/o CooHOI” mean? Does that mean training the entire policy from scratch? Is that policy trained with AMP and the same rewards? And what is the definition of CooHOI? What is included in CooHOI?\n\nRegarding the robustness of the system: How would the policy perform in noisy situations? For example, when the object bounding boxes are not accurate (in real scenarios, object perception will have errors).\n\nIn Figure 6, can you elaborate more on why policies, except for CooHOI, fail to carry objects even when they can still hold objects? Specifically, what are the failure cases? Do these policies fail to maintain a stable holding motion, or do they fail to accurately move the objects to the target location?\n\nIn addition, r_carry is only defined in the supplementary material. I suggest including it in Section 3.2.2.\n\nIt would be better to plot the standard deviations in Figure 6, given that you evaluate four seeds.\n\nWhat is the action space of the policy? Is it the target joint position for each joint?\n\nLIMITATIONS:\nThe authors provide a comprehensive limitation section of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}