Discovering Robotic Interaction Modes with Discrete Representation Learning

Authors: liquan  wang
ankit  goyal
haoping  xu
animesh  garg
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 06:05:28

Abstract: Abstract: Human actions manipulating articulated objects, such as opening and closing a drawer, can be categorized into multiple modalities we define as interaction modes. Traditional robot learning approaches lack discrete representations of these modes, which are crucial for empirical sampling and grounding. In this paper, we present ActAIM2, which learns a discrete representation of robot manipulation interaction modes in a purely unsupervised fashion, without the use of expert labels or simulator-based privileged information. Utilizing novel data collection methods involving simulator rollouts, ActAIM2 consists of an interaction mode selector and a low-level action predictor. The selector generates discrete representations of potential interaction modes with self-supervision, while the predictor outputs
corresponding action trajectories. Our method is validated through its success rate in manipulating articulated objects and its robustness in sampling meaningful actions from the discrete representation. Extensive experiments demonstrate ActAIM2’s effectiveness in enhancing manipulability and generalizability over baselines and ablation studies. For videos and additional results, see our website: https://actaim2.github.io/.

Main paper: 
 introduction: 
 Humans exhibit an exceptional aptitude for manipulating articulated objects by utilizing prior knowledge and learning through imitation. Generally, the outcome after manipulating the articulated objects is categorical such as opening or closing the door. Motivated by this cognitive process, our study seeks to develop a discrete representation of object affordance by merging behavior cloning with interaction mode identification. We focus on objects with multiple moving parts to explore a variety of distinct and meaningful outcomes, which we define as interaction modes. These interaction modes can be represented as a discrete set of options from which the agent can sample during inference to determine the appropriate interaction mode for the object. Notably, we characterize interaction mode as an affordance property of the object itself which can be learned from observation data. To transfer such discrete interaction modes into robotic action, an action predictor is learned from play data containing these diverse modes. Therefore, we argue that the manipulation policy is a joint distribution over both interaction mode selector and action predictor.The robotics field is replete with studies addressing such policy decomposed into modes (or skills) and action distribution. However, most behavior cloning approaches [1, 2, 3, 4] require expert data for task representation supervision, especially using language description as the task representation such as RLBench [5] , Calvin [6] , SayCan [7] , etc. This assumes that the distribution of the interaction mode is known and can be represented as language prompting. Other works learn the interaction mode (skill) distribution using unsupervised reinforcement learning [8, 9, 10, 11] with self-supervised intrinsic reward as weak supervision for task representation. Moreover, there are other interaction mode learning approaches [12, 13, 14] which learn a distinguishable interaction mode (skill) prior. However, the distribution of interaction modes (skills) learned in these works does not specifically map to correspondent outcomes, indicating that the agent cannot sample skills with reasonable and limited options. Therefore, these works fail to find a structural and disentangled space of skill for an agent to sample the interaction mode discretely based on observation. To solve the issues above, we require that the policy 1) captures various interaction modes without necessitating expert labels or privileged information and 2) allows the agent to make discrete choices within this space, reflecting the finite interaction modes available in a given scene.We introduce ActAIM2, which splits the policy into a discrete mode selector using a Gaussian Mixture Model for discrete sampling and an action predictor that processes sampled task embeddings to predict corresponding action sequences, trained via behavior cloning. Both components utilize selfsupervised play data from a dataset collected through adaptive data collection and heuristic grasping, ensuring a balanced representation of interaction modes. Building upon ActAIM's method [12] , our data collection does not use privileged information like reward functions or part segmentation and enhances realism by employing a complete robot instead of a floating gripper. To summarize, our contributions are threefold: First, we introduce ActAIM2, a self-supervised learning approach that enables discrete sampling across different interaction modes. Second, We devise a novel data collection methodology and have constructed a dataset with diverse interaction modes for model training. Finally, we thoroughly evaluate our model against a spectrum of generative models and behavior cloning agents, demonstrating ActAIM2's superior performance over existing baselines and the capability of performing the discrete sampling. 
related work: 
 Articulated object manipulation -The manipulation of articulated objects is challenging due to their complex geometries and kinematics, often understood through partially observed data like images. Common methods deduce the object's kinematic structure via passive observation [15, 16, 17] or interactive perception [18, 19] . These techniques are crucial for modeling articulated objects in robotics and incorporating these models into planning. While imitation learning relies on expert demonstrations [20, 21] , it is limited by the need for extensive and costly data collection. Conversely, recent research explores obtaining actionable visual priors through direct interaction with objects [22, 23, 24] , focusing on their geometric and semantic properties. Additionally, learningbased methods utilizing simulation supervised visual learning, and visual affordance learning are emerging [23, 24, 25] . ActAIM2 addresses these challenges by demonstrating the stability of our model in the experiments section.Transformer-based policies -Many recent works have proposed Transformer-based policies for robotic control. For behavior cloning, Transformers model conditional action distributions [26, 3] and encode observations into latent vectors [2] . They also process language commands to specify and generalize tasks, demonstrating flexibility and robustness [27, 28, 29] . Additionally, Transformers are utilized to learn latent 3D representations from scenes, building on their success in computer vision [30, 31, 32] , and to process voxelized scenes from RGB images and point clouds [2, 33] . However, challenges remain, such as high memory demands for high-resolution voxelizations that slow training, and susceptibility to noisy data when re-rendering views from point clouds [3] . Transformers also map language and visual inputs directly to robot actions, ranging from basic attention layers to complex pre-trained models [27, 34] . ActAIM2 builds on this foundation using the RVT model but innovates by integrating a novel discrete interaction mode representation into the skill language prompting.Unsupervised Skill Learning -Unsupervised skill discovery enables agents to learn distinct behaviors without a reward function but often struggles with inadequate state space exploration when using variational Mutual Information maximization [35, 36] . This limitation hinders its effectiveness for complex tasks. Some strategies counteract this by focusing the learning on smaller, Euclidean spaces to diversify movements [37] , although this often restricts the learning to navigation and simple coordinate-based tasks. Alternative approaches propose auxiliary exploration mechanisms and novel training methods to enhance state space exploration [36, 38] . ActAIM2 addresses these issues by introducing the term "interaction mode," defined by significant visual changes from changes in the degrees of freedom (DoF) of an articulated object. By maximizing the mutual information loss and using contrastive evaluation of visual changes, ActAIM2 encourages the model to identify discrete interaction modes and develop a disentangled latent space for effective sampling. 
problem formulation: 
 The problem we are solving is how to use a parallel-jaw gripper robot to manipulate various articulated objects and generalize such skills among all types of articulated objects. We adopt a two-phase methodology to enhance a robot's ability to manipulate articulated objects: data collection and model training, inspired by [12] . We employ a structured predefined action primitive, executing a series of actions across four heuristic phases (initiation, reaching, grasping, manipulation) to gather a comprehensive dataset of observations O i , including RGBD images and multi-view camera positions, without relying on predefined inputs. Within each action phrase, we define the action a as the keypose (p, R, q) of the parallel-jaw gripper similar to [5] . The key pose is defined as the concatenation of the 3 terms, which are p ∈ R 3 as the gripper position, R ∈ SO(3) as the gripper rotation quaternion, and q ∈ {0, 1} as the binary parameter indicating whether the gripper is open or close.Our model training aims to uncover the policy's distribution P(a|o), with o representing the observation and a = (p, R, q) the action, through a decomposition strategy that reconfigures the action distribution as:EQUATIONThe mode selector P(ϵ|o), contrasting with ActAIM [12] 's Gaussian space, utilizes a mixture of Gaussian distributions to define distinct interaction modes, facilitated by a discrete, latent space ϵ ∈ Z for enhanced action prediction and mode selection.Sampling -To foster a wide array of interaction modes within our dataset, we implement GMM-based adaptive sampling inspired by the methodology outlined in [12] . Following the acquisition of M trajectory datasets {T j |j = 1, 2, ..., M } through random and heuristic grasping sampling from previous interactions, we compute the task embeddings {z j |j = 1, 2, ..., M } based on Equation 9. A Gaussian Mixture Model (GMM) prior is constructed from these task embeddings, denoted as P(z|θ) = K k=1 κ k p(a|θ k ), where θ k represents the parameters of each Gaussian component within the mixture. The choice of K, the number of clusters, is a hyperparameter that reflects the presumed number of interaction modes inherent to the object. Subsequently, we cluster the task embeddings z j , assigning a unique cluster label to each corresponding trajectory. We found that the task embeddings from different trajectories grouped within the same cluster indicate a similar interaction mode, as they share proximate visual characteristics from initial and final observation. Upon clustering, a new GMM is formulated for each cluster, based on the action sequences, represented as P k (a|ϕ) = L l=1 β l p k (a|ϕ l ). We then aim to sample an equal number of actions from each cluster, ensuring that the representation of actions-and, by extension, interaction modes-within the dataset are as diverse as possible, thus facilitating a comprehensive exploration of the articulated object's potential interactions.Utilizing these sampling methodologies, we concurrently collect data across all articulated objects within our dataset, culminating in a dataset denoted as:EQUATIONAfter data collection, we enrich each trajectory within our dataset by associating the respective task embedding with the data tuple (O, a), thereby forming atomic training data instances represented as (O, a, ϵ) j . 
actaim2: robotic interaction mode discovery: 
 We aim to derive the policy P(a|o) from Equation 1, employing the robot as the agent within an environment populated by various articulated objects. Motivated by [12] , our preliminary step was to gather offline, self-supervised data via simulation, which served as the foundation for training the policy P(a|o). Guided by Equation 1, we dissected the target policy into the action predictor P(a|o, ϵ) and the mode selector P(ϵ|o). For training efficacy, the action predictor P(a|o, ϵ) and mode selector P(ϵ|o) were pre-trained individually before collectively fine-tuning the overarching pipeline P(a|o).We collect trajectory data T j = {(a i , O i )|i = 0, 1, 2, 3} j where O i are RGBD observations from a configuration of five cameras encircling the articulated object, and a i = (p, R, q) i represents the key pose and state of the gripper. Inspired by ActAIM [12] , we use a similar GMM adaptive method to collect diverse interaction modes. To collect the data with self-supervision, for each trajectory, characterized by the initial observation O init j = O 0j and final observation O f inal j = O 3j , we utilize a pre-trained image encoder E O to transform the image observations into a latent vector v. The task embedding z j for each trajectory T j is defined as follows:EQUATIONTo determine the success or failure of a manipulation, we introduce a threshold z, defining a trajectory T j as successful if z j > z. Details of the dataset are presented in Appendix 7.4.Learning Generative Model using GMM Prior -Based on Equation 1, we aim to identify an effective mode selector to generate a relevant task latent space ϵ ∈ Z. Inspired by the concept of visual affordances in [39] , we learn such latent space encapsulates all conceivable future states using the conditional Gaussian Mixture Variational Autoencoder (GMVAE) generative model.To learn the mode selector as prior P(ϵ|o), we use the data from our collected trajectory and select data (O init , O f inal ) j for mode selector training. Knowing that z from Equation 2 contains the complete information of final states given initial observation, We pre-compute the ground-truth label z as our learning target and regard the initial observation O init as the conditional variable to generate the task embeddings. We aim to learn a generative model which predicts all possible task embeddings z from the conditional initial observation O init . Taking inspiration from [40] , we construct the generative model as a Transformer-based Conditional GMVAE, as shown in Figure 2a . We defined our inference process as O init → (c, y) → x → ϵ where c is a categorical variable with p(c) = Multi(π) and y is a Gaussian distribution with p(y) = N (0, I) which jointly forms a Gaussian Mixture distribution as the prior. We expect the output learned ϵ would have a similar distribution as the ground-truth task embedding z since z can be represented as a Mixture of Gaussian distribution fixing the initial observation (fixing the initial object and object state). Formally, we write the distribution of the variable under the following process; for simplification, we denote O init as O i ,EQUATIONEQUATIONEQUATIONwhere K represents the number of mixture components, a hyper-parameter within the training regime, and µ c k , Σ c k , µ β , Σ β denote networks to be trained where ξ and β are the parameters of these networks. In our implementation, µ c k , Σ c k are instantiated as multi-layer ResNet [41] architectures, and µ β , Σ β as a transformer with 4 self-attention layers to enhance the stability of reconstruction.To improve the training stability, we model the categorical distribution p(c) by the Gumbel-Softmax distribution [42] .Mode Selector Training Loss -The conditional-GMVAE generative model is optimized using the variational inference objective, combining reconstruction loss with the log-evidence lower bound (ELBO) loss. The ELBO loss is expressed as:EQUATIONwhere the proxy posterior q(x, y, c|ϵ, O i ) is approximated as q(x, y, c|ϵ, O i ) =i q ψx (x|ϵ, O i )q ψy (y|ϵ, O i )q ψc (c|x, y, O i ), with ψ x , ψ y representing the parameters of the q ψx and q ψc networks. Based on the decomposition, We refer to the terms in the lower bound as the reconstruction term, conditional prior term, y-prior term and x-prior term respectively. The prior term for variables c, y, x is computed as the Kullback-Leibler (KL) divergence, which penalizes the difference between the learned latent variable distribution and the prior distribution. We formalize the reconstruction term as the L2 loss between task embedding predictions ϵ and ground-truth z, written as L reconstruct = ||ϵ -z|| 2 in practice. Mathematical details of the conditional GMVAE are presented in Appendix 7.5.1.Our final objective is to infer a sequence of low-level actions a = (p, R, q) from the current observation O given the predicted task representation ϵ, ensuring the action sequence effectively accomplishes the articulated object manipulation task while aligning the specific given interaction mode ϵ. As shown in Figure 2b , The model inputs RGB-D images from encompass multi-view cameras, the present state of the robot gripper, and the task latent embedding ϵ. From the data collection phase, we collect successful manipulation trajectories T j . We decomposed T j to individual key-frame actions and observations (a i , O i ) j ∈ T j ∈ D as data points for training. To this end, we propose a multi-view transformer architecture as the basis for our behaviour cloning agent, aimed at learning the action distribution P(a|o, ϵ).Novel View Rendering and Multiview Transformer -Building upon the RVT [3] a cross-entropy loss, updates the gripper jaw's open-close state q. Accordingly, the comprehensive training loss for the action predictor is articulated as:L action = L p + L R + L q = CE(p, p) + CE(R, R) + CE(q, q) (7)In our training process, we jointly train the policy P(a|o) as formulation:EQUATIONIn this context, sg[•] stands for the stop-gradient operator, which stops the flow of partial derivatives to the next network layers. During the training phase, we include the combined total loss from both the action predictor (behavior cloning loss) and the mode selector (ELBO loss), represented asL total = L action + L ELBO . 
experiments: 
 Our experimental setup is designed to assess how well the proposed method, ActAIM2, performs in several important areas: 1) ActAIM2 proficiently handles various interaction modes, adapting to differences across object instances and categories. 2) Utilizing Gaussian Mixture Model (GMM) based priors facilitates a more structured latent space, enabling targeted searches for specific samples within the interaction modes. To evaluate ActAIM2, we report the average success rate, sample success rate, and the average reward achieved during interaction mode grounding iterations.Leveraging the success of previous studies [43, 2, 12] , we use articulated objects from the SAPIEN dataset [44] for our experiments. Our training dataset includes six categories: faucets, tables, storage furniture, doors, refrigerators, and switches, while the testing dataset features three new categories: windows, boxes, and safes, with each category comprising 8 unique objects. We employ IsaacGym as our simulation platform [45] , where we have designed a custom environment featuring a Franka Emika robot adjacent to an articulated object with 5 cameras circling the object. In our evaluations, we define a successful trajectory as follows: the model takes an initial observation and iteratively predicts a series of actions, executed across four steps. After these steps, we evaluate the object's state.If the Degrees of Freedom (DoF) of any part of the object has changed by more than 30%, we deem the trajectory successful. This approach allows us to assess the efficacy of the robot's manipulation capabilities in dynamically altering the object's state based on the model's iterative predictions.Baselines And Ablation Study -We compare our results with the following baselines and ablation study. Data Collection: Adaptive data collection using a GMM-based heuristic grasping method to sample action sequences. Where2Act [43] : Calculates priors for discretized action primitives, using the complete object point cloud instead of segmented movable points. ActAIM [12] : Combines a Conditional Variational Autoencoder (CVAE) [46] with a transformer-based action predictor [47] to sample and forecast manipulation trajectories, employing an unrealistically simplified floating gripper. Goal-RVT: A supervised version of ActAIM2 that uses directly provided goal images, bypassing GMM prior sampling, to feed into the action predictor, demonstrating comparable performance to other supervised methods. VQVAE-RVT: Inspired by Genie [48] , replacing the interaction mode selector with a VQVAE codebook, using a codebook size equal to the GMM cluster number in ActAIM2, and sampling task embeddings discretely during tests.Evaluation Metrics -We conduct two types of evaluations to assess the ability of models for successful interaction and discrete sampling, testing across three object categories: Seen Objects (from the training set), Unseen Instances (same categories as Seen but different instances), and Unseen Categories (from outside the training set). Interaction Mode Discovery metric assesses the average sample success rate (SSR) defined as the ratio of successful trajectory samples to total samples. For Goal-RVT, expert-labeled goal images are used to ensure viable trajectories, whereas ActAIM2 samples from all 8 clusters to calculate SSR. Moreover, we measure SSR for dominant interaction modes within the selected 2 clusters as Mode Sampling Evaluation. This indicates that the evaluated method is required to have at least 2 interaction modes discovered and distributed in separate sample options. For ActAIM2 and VQVAE-RVT, we sample and execute actions from 8 distinct clusters, documenting the success rate of the primary interaction mode in the first and second clusters.Goal-RVT and Goal-ActAIM follow a similar approach but use ground truth goal images instead of sampled clusters, adjusting for changes in DoF.Interaction Mode Discovery -Table 1 shows that ActAIM2 often outperforms baselines in interaction success rates for both trained and unseen instances. Where2Act and ActAIM use a simplified scenario with a heuristic floating gripper, whereas ActAIM2 utilizes a complete Franka Emika robot setup but still performs a higher success rate. This means ActAIM2 performs well even when it has to discard trajectories due to unreachable states or collisions. This highlights its ability to accurately predict gripper positions and efficiently sample from a learned latent space Z, surpassing the performance of Goal-RVT's broader goal image approach. We did not report the VQVAE-RVT results here since VQVAE-RVT outperforms around 5% in each test compared to ActAIM2 averagely. However, despite the average sample success rate, we show that VQVAE-RVT does not meet our requirement for discrete sampling in the following two experiments. Mode Sampling Evaluation -ActAIM2 and Goal-RVT were evaluated for their ability to distinguish between interaction modes. ActAIM2 proved stable and successfully identified the primary interaction mode across various categories and conditions, matching Goal-RVT's performance even for a secondary mode. Larger objects like doors and refrigerators consistently showed successful interaction trajectories, demonstrating ActAIM2's adaptability to different object sizes and complexities. In contrast, VQVAE-RVT struggled to offer a viable second dominant interaction mode. Our tests showed that VQVAE-RVT tends to produce similar action heatmaps across different code vectors, complicating the identification of specific interaction modes. These issues are further explored in our reinforcement learning experiments on interaction mode grounding.We evaluate ActAIM2 in a real-world setup using visual data. A Kinova Gen2 robot arm interacts with a small shelf featuring three drawers, performing opening and closing actions on half-open drawers (see Figure 4 ). An Azure Kinect RGB-D camera captures the scene, and after calibrating the camera extrinsic, we process point clouds with ActAIM2. The robot's gripper grasps the drawer edge to pull (open) or push (close) the drawers. We set three initial states with the top, middle, or bottom drawer half-open, introducing task variation by moving the shelf's position and randomly selecting grasp points. ActAIM2 is pre-trained on simulated data and fine-tuned with 60 real-world trajectories collected via expert control using four action primitives. After fine-tuning, ActAIM2 achieved a 75% success rate for both pushing and pulling tasks, accurately classifying interaction modes. Sampling from the 8 clusters representing interaction modes, specific clusters had success rates over 87.5%, mirroring the mode sampling evaluation and demonstrating the benefit of incorporating human demonstration data. Some failures occurred due to the gripper missing the drawer edge when the object's position shifted slightly, suggesting the need for more detailed tuning and increased variation in training to improve object positioning accuracy. For more detailed information on the real robot experiments, please refer to Section 7.1 and our website. 7 Supplementary 
real robot experiments: 
 In this section, we evaluate ActAIM2 using real visual sensory data by training and testing the model in a real-world setup. For further insights into the model's performance, we have included attached videos, which are also available on our website.Real World Setup -We employ a Kinova Gen2 robot arm with a spherical wrist, 7 degrees of freedom, and a parallel jaw gripper (Robot Type: j2s7s300), placed on a table alongside the articulated object shown in Figure 5a . The scene is captured using a statically mounted Azure Kinect (RGB-D) camera in a third-person view. We first calibrate the robot-camera extrinsics and transform the perceived color point clouds to the robot's base frame before processing them with ActAIM2. During evaluation, based on the target pose predicted by ActAIM2, we utilize the MoveIt motion planner to direct the robot to the target pose, incorporating trajectory generation and feedback control.Tasks -We focus on manipulating a small shelf with three drawers showed in Figure 5b , defining two interaction modes for each half-open drawer: opening and closing. For the opening interaction, the gripper manually grasps the edge of the drawer and pulls it out. Conversely, for closing, the gripper grasps the edge and pushes the drawer in. We utilize a small shelf with three drawers as the articulated object and set three plausible initial states: the top, middle, and bottom drawers are each set to half-open. Task variation is introduced by altering the position of the small shelf on the table.Data Collection and Training -Our evaluation focuses on a single task: opening and closing a half-open drawer on a small shelf. Initially, ActAIM2 is pre-trained using domain randomization on our simulated dataset. We currently employ ActAIM2 trained on the single-view dataset referenced in Table 4 . For expert demonstration, we collect real-world interaction data via expert control to either open or close the drawer. The robot control strictly adheres to the simulator setup, utilizing four predefined action primitives: initializing, reaching, grasping, and interacting. To introduce variation in the training trajectories, each expert-controlled interaction randomly selects different parts of the drawer for the robot to grasp. Throughout these interactions, the articulated object remains stationary. Simultaneously, RGB-D images are captured as the robot moves to the target pose, creating a dataset of RGB-D frames paired with target pose annotations. For each initial state (top, middle, and bottom drawers half-open), we collect 10 expert data sets for each mode, totaling 60 trajectories (10 pushing and 10 pulling trajectories for each state).Results -We initially trained ActAIM2 on simulated data and later fine-tuned it using real-world trajectories. This process yielded a notable success rate of 75% for both the pushing and pulling tasks involved in drawer interactions. Importantly, ActAIM2 accurately classifies the interaction modes of pushing and pulling, demonstrating a well-aligned discrete representation. To assess the performance, we sampled from each of the 8 clusters designed to represent distinct interaction modes-either opening or closing the drawer. Clusters specific to pushing and pulling were found to have success rates exceeding 87.5%, mirroring the mode sampling evaluation. This improvement is largely due to incorporating human demonstration data, which helps correct any imbalance in interaction mode representation during training.However, in our real robot experiments, we observed some failure cases where the gripper failed to grasp the edge of the drawer. These failures are linked to task variations, specifically the slight movement of the object along the x-axis, which led to the gripper failing due to the agent's incorrect determination of the object's position. To address this issue, we suggest implementing a more detailed coarse-to-fine tuning approach on the vision data and training with increased variation to improve accuracy in object positioning.We further explore ActAIM2's capability as a prior in reinforcement learning to achieve specified interaction modes given a goal, iterating through all plausible interaction modes to effectively ground and distinguish each mode. In our setup, we used an unseen table (ID:20411) as the articulated object with a sparse reward function defined as r = 1 if |d i -d g |< d ϵ else 0, where d i is the selected ith degree of freedom (DoF), d g is the target DoF, and d ϵ is the threshold. Reinforcement learning was employed using ActAIM2 as the prior, framing the task as a Multi-Arm Bandit problem [49] , which narrows the sampling space and enhances learning efficiency. For comparison, we used ActAIM and RVT-VQVAE as the prior in the baseline experiments, applying the same reward function but updating the sample task embedding via the cross-entropy method [50] .Qualitative and quantitative results are visualized in Figure 6a and Figure 6b . Figure 6a shows agents refining their understanding of interaction modes through repeated sampling, leading to selfsupervised re-clustering. Figure 6b illustrates that using ActAIM2, the reward-equivalent to the average sample success rate-smoothly increases, whereas ActAIM struggles to ground the task embedding to a specific interaction mode representation, and VQVAE-RVT updates slowly due to its inability to provide distinct interaction mode representations.During the data collection phase, ActAIM2 operates under the assumption that: 1) the manipulations are straightforward enough to be captured using a limited set of action primitives such as grasping, pushing, or pulling; 2) an interaction mode is identified upon observing significant visual changes;3) the interaction modes can be categorized into a few distinct types. A more detailed discussion of these assumptions is provided below.Simple Action Space -We employ a scripted, self-supervised method to collect actions that encompass diverse interaction modes. The action space is sufficiently simple, focusing primarily on heuristic grasping and random actions. For more complex tasks, such as hammering, washing dishes, or cooking, our current method fails to collect adequate data. Addressing these more intricate tasks would require a more comprehensive and extensive dataset.Significant Visual Change -Our data collection is entirely self-supervised, devoid of any expert data or privileged information. We define an interaction as successful if it results in a significant visual alteration to the targeted objects. This approach is effective for articulated objects in our studies, such as doors, windows, or tables, which typically remain stationary except for their movable components. However, challenges arise with objects like tools (e.g., hammers, cups, knives), where it is difficult to discern visual changes either in the tools themselves or the targeted objects (e.g., nails, cup holders, or deformable objects). Especially in tasks requiring repetitive actions, like continuously striking a nail or repeatedly wiping dishes, a more nuanced and generalized method is necessary to determine if meaningful interactions are occurring.Discrete Interaction Modes -Articulated objects, by design, often have limited manipulation options. However, when dealing with other objects such as tools, the number of potential interaction modes significantly increases. The functionality of these objects can be diverse; for example, a hammer might be used not only for hammering but also for hooking or reaching. Even the act of grasping these objects presents countless variations, complicating the task of clustering them into discrete modes.Based on the assumptions discussed earlier, we have identified two primary avenues for extending our current research: long-horizon planning tasks and enhancing tool manipulation strategies.Long-horizon Planning Tasks -Leveraging the discrete representation of interaction modes provided by ActAIM2, we propose its application to long-horizon planning tasks. Examples of such tasks include sequentially opening a table drawer, locating and opening a box within the drawer, and finally pressing a button inside the box. These tasks illustrate the potential of ActAIM2 to serve as a foundational prior, streamlining the process to discrete searches within complex sequences. To ensure the robustness of our approach, it is crucial that the model accurately predicts all feasible interaction modes based on the given scenario.Extension to Tool Manipulation Tasks -Another direction for expansion involves applying our work to tool manipulation. Here, defining the interaction modes for various tools will be pivotal. A robust dataset specifically tailored for tool manipulation is essential to support this endeavour. Additionally, a more sophisticated scene descriptor is required to effectively determine which objects to manipulate and which to designate as targets. This development would facilitate more nuanced and effective tool interactions in automated systems.When collecting data, we employ a strategy of random sampling, subsequently filtering successful actions as determined by our vision model without resorting to any privileged information. Drawing inspiration from [43, 51] , we delineate the task of manipulating articulated objects into four fundamental poses: initiation, reaching, grasping, and manipulating. Throughout these stages, we capture the robot's key action poses a i = (p, R, q) i and RGBD observations O i from a configuration of five cameras encircling the articulated object. Upon collecting the trajectory T j = {(a i , O i )|i = 0, 1, 2, 3} j , we also archive the initial and final observations, O init j and O f inal j , respectively, captured from the multi-view cameras with the robot occluded, to facilitate manipulation success evaluation.We introduced our method of identifying successful interacted trajectories, which can be purely from vision data, specifically the initial and final observation. For each trajectory, characterized by the initial observation O init j and final observation O f inal j , we utilize a pre-trained image encoder E O to transform the image observations into a latent vector v. The task embedding z j for each trajectory T j is defined as follows:EQUATIONIn our implementation, we employ a pre-trained VGG-19 network [52] , without the final fully connected layers, to serve as our image encoder E O . To determine the success or failure of a manipulation, we introduce a threshold z, defining a trajectory T j as successful if z j > z. It is important to note that this process does not rely on any privileged information. To illustrate the validity of our method, we define the trajectory's success as a 30% change in the ground-truth DoF value. The efficacy of this criterion is validated against the ground-truth DoF values, demonstrating a 97.4% accuracy rate across our training and testing dataset. The collected trajectories must exhibit the diversity of interaction modes of the articulated objects. Thus, we employ three distinct methods of action sampling, as outlined below. The final dataset is a composite of these three methods.1. Random Sampling -We generate play data for manipulation without prior interaction. First, we select an interaction point p 1 ∈ R 3 on the articulated object, ensuring it lies within the robot's workspace. Subsequently, we sample a uniformly random manipulation rotation R 0 ∈ SO(3) and a manipulation position p 2 within the valid area, applying filters to exclude any configurations that would result in a collision. The robot's initial position p 0 is also determined through random sampling, which is a specified distance from the interaction point p 1 , ensuring a feasible starting position for the manipulation task. Based on the previous sampling, we define the randomly sampled action sequence as {(p 0 , R 0 , 0), (p 1 , R 0 , 0), (p 1 , R 0 , 1), (p 2 , R 0 , 1)}.2. Heuristic Grasping Sampling -Heuristic grasping sampling is employed to select interaction points on the articulated object to enhance the precision of grasping actions. Utilizing the RGB-D observations, we crop the articulated object and transform it into an RGB point cloud, which undergoes preprocessing with DBSCAN clustering [53] , aimed at identifying segments with significant geometric features, such as handles or buttons. After clustering, each segment is analyzed by a pre-trained GraspNet model [54] to generate a set of potential grasps. From this set, grasps with the highest scores are selected, with the grasp point designated as the interaction point and the grasp orientation as the gripper rotation for the trajectory. The initial and manipulation poses are determined using the previously described random sampling approach. This heuristic approach to grasping not only bolsters the stability of grasp actions but also enriches the dataset with a higher proportion of complex interaction modes, such as "grasp to open", enhancing the dataset's diversity and utility for training models to manipulate articulated objects in 'hard' interaction scenarios.The dataset we developed for training purposes is available on our official website. Our dataset was constructed through a combination of random sampling, heuristic grasp sampling, and Gaussian Mixture Model (GMM)-based adaptive sampling, featuring the Franka Emika robot engaging with various articulated objects across multiple interaction modes. It encompasses categories such as faucets, tables, storage furniture, doors, refrigerators, and switches, with 8 unique instances per category. For each instance, we collected 150 trajectories, ensuring comprehensive coverage of the objects' interaction modes. Objects were scaled to realistic size and initialized in a 'half-open' state, denoting a median value for each degree of freedom (DoF). The data collection methodology is detailed in Algorithm 1.This section outlines the detailed implementation of the model architecture, encompassing both the mode selector and the action predictor components.Require: Initial observation O i , Number of GMM component K, hyper-paramter M for GMM in each cluster Ensure: All sampled trajectories are filtered successful by evaluating ϵ > ε D ← ∅ ▷ Set the initial dataset to be empty while D not have enough data doD r = {(a, o) i } ∼ RandomSampling ▷ Random Sampling G = {g i } ∼ GraspNet(O i ) ▷ Sample Grasp using GraspNet D g = {(a, o) i } ∼ GenerateTraj(G) ▷ Gnerate trajectory based on grasp D ← D ∪ D r ∪ D g ϵ i ∼ D▷ Compute task embedding in current D Cluster ϵ i with GMM, assign cluster label on each trajectory{D j |j = 1, ..., K} ← D D GM M ← ∅ for j in range K do Extract D j in D based on cluster label p(D j |π, µ, Σ) = N n=1 M m=1 π m N (x n |µ m , Σ m ) ▷ fit GMM Dj ← {(a, o) i } ∼ p(D j |π, µ, Σ) ▷ Sample action from GMM D GM M ← D GM M ∪ Dj D ← D ∪ D GM MThis section revisits the stochastic variables' definitions and distributions, as previously emphasized. The distributions of the model parameters are formalized as follows:EQUATIONEQUATION)EQUATION)EQUATIONHere, µ c k , Σ c k , µ β , Σ β are the model parameters to be optimized. Furthermore, we delineate the generative model and compute the inference at test time by defining the posterior as follows:EQUATIONThis necessitates the computation of three additional network parameters: q ψx , q ψy , q ψc . We then elaborate on deriving the posterior q ψc (c|x, y, O i ) for categorical variables c, employing the Gumbel Softmax for the representation of categorical distributions.Notice that c is a categorical parameter that c ∼ Multi(π). We defined that c ∈ C = {c 1 , c 2 , ..., c k } and the each class probability is described as {π 1 , π 2 , ..., π k }. We use the Gumbel Softax trick which provides a simple and efficient way to draw samples c from a categorical distribution with class probabilities {π 1 , π 2 , ..., π k }. The following form represents the categorical c as,EQUATIONwhere {g 1 , g 2 , ..., g k } are i.i.d samples drawn from Gumbel(0,1). Assuming that categorical samples c are encoded as k-dimensional one-hot vectors ω lying on the corners of the (k -1)-dimensional simplex ∆ k-1 We use the softmax function as a continuous, differentiable approximation to arg max, and generate k-dimensional sample vectors ω ∈ ∆ k-1 . We defined ω as Where τ is the temperature as the hyperparameter. Therefore, we define the density of the Gumbel-Softmax distribution as,EQUATIONEQUATION)Now, given the representation of the categorical distribution of c from Equation 21, we derive how we compute the posterior q ψc for c. We consider the posterior q ψc (c = c j |x, y, O i ) given c = c j ,EQUATIONTherefore, we derive the posterior q ψc directly and leave 2 posterior network q ψx , q ψy to be trained.Based on the following discussion, we draw the generative model and variational model view as graphical models in the Figure 8 .In the implementation detail, we write parameters p β = (µ β , Σ β ) and p ξ = (µ c k , Σ c k ) to generate a Gaussian distribution with each representing the mean and variance. We implement the network µ c k , Σ c k , ψ x , ψ y with a multi-layer ResNet and implement the network µ β , Σ β as a multi-view transformer since both O i and ϵ represent multi-view information with the same number on the channel as the correspondent view number. We show our model µ β , Σ β architecture in Figure 9 .We illustrate the functionality and application of our mode selector through two distinct plots, highlighting both the training process and the inference mechanism for task embedding generation.Figure Figure 11a depicts the model's operation during training, where it processes the conditional variable O i along with the ground truth data ϵ, to accurately reconstruct the task embedding.Conversely, Figure Figure 11b demonstrates the inference stage, where the model, requiring only the initial observation O i and a discretely sampled cluster (employing an 8-cluster configuration for implementation), successfully generates the corresponding task embedding ϵ.We provide the architecture of the action predictor which is a joint transformer that takes in task embedding ϵ and novel view as input. The detailed implementation is shown at Figure 10 . O i = (O i 0 , O i 1 , O i 2 , O i 3 , O i 4 ), and the Mixture of Gaussian (GMM) variable x. It is important to note that x can be represented as a multi-view feature vector, with our encoding approach preserving the separation of multi-view channels. Initially, the multi-view RGBD images are passed through a pre-trained VGG-19 image encoder to extract feature vectors for each view. Subsequently, these feature vectors, along with the GMM variable x, are inputted into a joint transformer. This transformer, featuring four attention layers, is tasked with producing the means and variances associated with the reconstructed task embedding ϵ. We supplement our presentation with additional qualitative results, further elucidating the model's proficiency in learning the disentanglement of interaction modes. Initially, we demonstrate the efficacy of the mode selector through a t-SNE plot. This choice of visualization is motivated by our methodology of training the mode selector and action predictor independently, allowing for a focused examination of the mode selector's performance. mirroring the approach of a conditional generative model. It highlights the contrastive analysis between the initial and final observations-the latter serving as the ground truth for task embedding-to delineate generated data against the backdrop of encoded initial images as the conditional variable. The process involves inputting both the generated task embedding data and the conditional variable into a 4-layer Residual network-based mode encoder, which then predicts the categorical variable c. Following the Gaussian Mixture Variational Autoencoder (GMVAE) methodology, the Gaussian Mixture Model (GMM) variable x is computed and introduced alongside the conditional variable to the task embedding transformer decoder. This model is tasked with predicting the reconstructed task embedding, sampled from the Gaussian distribution as outlined in the architecture of the mode selector decoder, and calculating the reconstruction loss against the input ground truth data.(b) Inference Process: In the inference phase, the agent discretely samples a cluster from the trained Gaussian Mixture Variational Autoencoder (GMVAE) model to calculate the Mixture of Gaussian variable x. This variable x, in conjunction with the conditional variable (initial image observation), is then inputted into the mode selector transformer decoder. The objective is to reconstruct the task embedding for inference, effectively translating the conditional information and sampled cluster into actionable embeddings.Subsequently, we extend our qualitative analysis with figures akin to those presented in the main paper, offering a comprehensive view of the model's capabilities. These additional figures serve to reinforce the insights gained from the initial results, showcasing the model's nuanced understanding of interaction modes through the distinct visual representations of the data. 7.6.1 Mode selector TSNE plot Figure 16 Utilizing our pre-trained Conditional Gaussian Mixture Variational Autoencoder (CGMVAE) mode selector, we conduct disentanglement learning visualization on our comprehensive dataset. Specifically, we focus on the "single drawer" object (object ID: 20411), employing the mode selector to delineate the generated clusters and compare them with the ground truth task embeddings. The data for this visualization is derived from our dataset, and we calculate the task embedding ϵ j for each data point as the difference between the initial and final object states, represented byϵ j = v init j -v f inal j = E O (O init j ) -E O (O f inal j ) .Subsequently, we employ a t-SNE plot to simultaneously visualize the ground truth and generated task embeddings. In this visualization, distinct colors within the ground truth plot indicate data points originating from different interaction modes. Similarly, varied colors in the generated plot correspond to data points arising from disparate clusters within the Mixture of Gaussians model. Through this approach, we demonstrate that:1. The ground truth task embeddings ϵ are distinctly clustered based on the interaction modes. 2. The CGMVAE model effectively generates clusters that categorize data points by their respective categories c. 3. The reconstructed data closely aligns with the ground truth data points, with the majority of the clustered data encompassed within the respective ground truth clusters.This visualization underscores the efficacy of our generative model mode selector in extracting task embeddings for further application in the action predictor, highlighting the model's capability to discern and categorize interaction modes accurately.We present extensive qualitative results in Figure 17a , Figure 17b , Figure 18a , and Figure 18b , demonstrating the model's ability to predict distinct interaction modes through discrete sampling. For each object, we explore three different clusters, each representing a unique interaction mode. The initial state of the robot and the articulated object is depicted from three perspectives: top-down, front, and side views. The heatmaps, derived from the top view during manipulation steps, highlight the variance in action space corresponding to different sampled interaction modes. Subsequent imagery illustrates the robot's movement within the simulator and the outcome following interaction with the articulated objects. It is important to note that comprehensive video demonstrations accompany this document and are accessible on our website, https://actaim2.github.io/.Inspired by the Genie [48] approach, we have compared our ActAIM2 with VQVAE-RVT to assess the efficacy of these models in discerning discrete interaction modes in robotic manipulation tasks. Our primary objective was to evaluate the distinction between interaction modes using a simplified scenario, a single-drawer table, which naturally exhibits two distinct interaction modes: opening and closing.In our experiments, we visualized the latent spaces generated by both ActAIM2 and VQVAe-RVT.Particularly for VQVAE-RVT, the latent space visualization involved examining the distribution of eight code vectors. As depicted in Figure 12 , these vectors clustered into two categories, which ideally should correspond to the two expected interaction modes of the drawer. This clustering pattern was anticipated and desired as it suggests a clear demarcation between the distinct modes of interaction. However, subsequent visualizations raised concerns about the practical efficacy of the VQVAE-RVT model in our application context. When we explored the heatmaps generated by the VQVAE-RVT model, we observed a critical limitation: all 8 code vectors produced essentially the same heatmap, despite their differing positions in the latent space. This heatmap, illustrated in Figure [Y] , consistently depicted all plausible interaction modes for the drawer, regardless of the specific code vector used. This outcome was in stark contrast to the results from ActAIM2, where distinct heatmaps clearly indicated specific interaction actions like pushing or pulling, depending on the sampled cluster within the latent space.These findings led us to conclude that merely replacing the GMVAE component with a VQVAE in the setup did not achieve the desired disentanglement of interaction modes. The VQVAE-RVT model failed to map the code vectors to unique, mode-specific interaction strategies, instead converging on a generalized representation that was not useful for distinguishing between the actionable options of opening and closing the drawer. Consequently, ActAIM2's ability to discriminate between distinct interaction modes via cluster-specific sampling proves superior in contexts demanding discrete and distinguishable action representations.We also explain this from mathematics which provide an intuition of why GMVAE perform much better than VQVAE. Referring to Figure 8 and the ELBO loss computation in Equation 18 from the appendix, compared to the VQVAE, the most unusual term in our ELBO is the c-prior term. The c-posterior calculates the probability of assigning a data point to a cluster by measuring the distance between x and each cluster position from y. The c-prior helps reduce the KL divergence between the c-posterior and a uniform prior by adjusting both the clusters' positions and the encoded point x. This term aims to merge clusters by increasing their overlap and bringing their means closer together. However, like other KL regularization terms, it can conflict with the reconstruction term and may become too dominant with more training data. This over-regularization issue, common in traditional VAEs, also affects GMVAE clusters, leading to large, degenerate clusters.To illustrate the practical applications and effectiveness of ActAIM2, we generated demonstration videos by employing its inference mechanism. The process involves several key steps:1. Generative Mode Selection: Initially, observations are inputted into the generative mode selector of ActAIM2. This component is responsible for reconstructing the task's latent space, which is modeled as a Mixture of Gaussians. This structure enables discrete sampling of clusters, which represent distinct interaction modes that the robotic system can execute. 2. Sampling and Action Prediction: From the reconstructed latent space, we sample the task embeddings by selecting a cluster within the Gaussian Mixture Model (GMM) and its corresponding Gaussian distribution. This sampled task embedding is then forwarded to the action predictor. The action predictor generates the specific actions needed to interact with the environment effectively. 3. Simulation and Recording: As depicted in Figure 14 This comprehensive demonstration not only validates the functionality of ActAIM2 but also provides a visual understanding of its potential in diverse robotic manipulation tasks. The videos highlight the nuanced interactions achievable through targeted sampling within the model's structured latent space. Table 3 : Object Mode Sampling Evaluation with More Interaction Modes: We selected objects with more than two interaction modes and evaluated the mode sample success rates using ActAIM2. In practice, ActAIM2 executes actions sampled uniformly across 8 clusters and then reports the corresponding interaction mode success rates. The reported interaction modes are illustrated in Figure 19 .Object In this section, we have added two additional experimental results: object-specific mode sampling evaluation over ActAIM2, where the objects exhibit more than two interaction modes, and average sample success rate evaluation over ActAIM2 with limited perception information. These results aim to demonstrate the stability and robustness of ActAIM2 when predicting outcomes for more complex objects. Furthermore, we demonstrate that ActAIM2 can still accurately predict general outcomes with minimal visual information. This suggests the potential for simplifying the scene setup to adapt ActAIM2 for real-world experiments.In this section, we assess the mode sampling effectiveness on an object-wise basis as detailed in the experiment section of our paper (see Table 2 ). We focus on the first and second mode clusters identified by ActAIM2, presenting only two modes because we are examining the average mode sample success rate across categories, typically involving around eight different instances. For simpler articulated objects like a table with a single drawer or a faucet with a single switch, there are typically only two valid interaction modes. This is because these objects are initialized at the mean joint value j init = (j max + j min )/2. However, most articulated objects feature more than two joints, leading to multiple interaction modes.To illustrate this, we select more complex objects from various categories, including faucets, tables, doors, and storage units. We first showcase plots depicting the interaction modes for each object from Figure 19 . Following this, we present how ActAIM2 predicts the various interaction modes. These objects and their interaction demonstrations are also featured on our website through illustrative videos.From Table 3 , we find that ActAIM2 is capable of predicting multiple interaction modes for more complex objects and achieves a reasonable success rate among the demonstrated interaction modes.In the main paper, we utilize five cameras positioned around the articulated object to gather visual information. To assess the model's applicability in real-world scenarios, we evaluate its performance when trained with reduced visual information. Specifically, we conduct two additional experiments: one training with only depth information and another using a single view RGB-D camera. We employ the same dataset as in the experimental section but mask out extra information. For qualitative results, please refer to. Our results demonstrate that ActAIM2 can effectively train with less visual data, with only an acceptable level of performance decrease.From Table 4 , we find that training with only depth information does not significantly affect performance, with less than a 5% decrease in success rate when color is omitted. However, using only a single camera view impacts the success rate more substantially, particularly when the model is applied to larger objects. Scalability also significantly decreases with a single view camera. Nonetheless, performance on previously seen objects remains at a similar level. We state that the main differences between our work and [12] lie in 2 major aspects which are data collection (environmental setup) and method (discrete representation).Data Collection For ActAIM, we utilized a floating parallel-jaw gripper for data collection, eliminating concerns related to invalid collisions or reachability issues. The action primitives learned by ActAIM include identifying the point to graspor interact and determining the direction to move post-grasp, akin to the approach used in Where2Act [43] .For ActAIM2, we employed a full Franka Emika robot, necessitating consideration of workspace constraints and potential collisions with the robot body. Directly applying the GMM adaptive data collection method from ActAIM proved challenging; hence, we incorporated a heuristic grasping strategy to aid in identifying graspable parts of the object. We developed an analytical technique to determine which segments of the point cloud are likely to correspond to handles on articulated objects. This was followed by using DBSCAN to identify and filter potential heuristic handles. Subsequently, we employed ContactGraspNet [55] to predict possible grasping points, as illustrated in Figure 5 of the Appendix. We combined these heuristic grasping actions with randomly sampled data to execute GMM adaptive sampling, similar to ActAIM, ultimately integrating these components into our final dataset.Method Both ActAIM and ActAIM2 share the approach of decomposing the policy into a mode selector and an action predictor. The key difference lies in how they represent the interaction mode. In ActAIM, a simple Conditional Variational Autoencoder (CVAE) is used as the mode selector to represent the interaction mode for manipulating articulated objects. However, the latent space in this approach forms a Gaussian distribution, which cannot be easily sampled or clustered.In Section 5.3, we experimented on interaction mode grounding, demonstrating that ActAIM struggles to associate task embedding with a specific interaction mode. The purpose of this experiment was to show that if the structure of the latent space is well-organized (meaning that all interaction modes are distinctly learned), the grounding should converge after a few interactions. The results revealed that ActAIM fails to provide a clear and discrete representation of interaction modes.Furthermore, we visualized the interaction mode representation from ActAIM, as shown in Figure 14 of the Appendix. The visualization indicates that the interaction modes learned by ActAIM are noisy and disorganized, with different modes scattered throughout the space. This explains why the experiment in Section 5.3 did not successfully ground specific interaction modes.Interaction modes are characterized by significant visual changes in the observation space. In contrast to methods such as SayCan [7] , VADER [56] , or RLAfford [57] , our approach focuses more on visual affordance without incorporating action data. The interaction mode seeks to predict and categorize the limited future states of articulated objects, such as opening or closing a partially open drawer. Ideally, the mode selector p(ϵ|o) could function as the encoder in a video generative model. In the context of articulated object manipulation, where visual information is readily available, our mode selector model is trained using a pre-trained visual encoder (VGG-19 in our case). This mode selector could be substituted with any world model or video prediction model that forecasts possible future states. Ultimately, the interaction mode infers a distribution over plausible future states achievable through interaction, based on an input image of a scene. For articulated objects, any visual change in the object, especially if it occludes the robot, is significant. We aim to extract and cluster such information to facilitate learning in the action decoder.In this section, we discuss 3 major limitations of the current ActAIM2 which are imbalanced success rate in different interaction modes, lack of general scalability, and limited to articulated object manipulation. Moreover, we provide potential solution to all these limitations.The experimental results demonstrate that while ActAIM2 can represent different interaction modes using discrete clusters, the success rates among these modes are imbalanced. The Mode Sampling Evaluation table shows that the average success rate in the second mode cluster is much lower than in the first mode cluster for ActAIM2. Additionally, Figure 6b illustrates this imbalance using a half-open drawer demonstration: pushing the drawer in has an over 80% success rate, whereas opening the drawer only achieves about a 20% success rate.This imbalance in success rates could impact the application of ActAIM2, particularly in interaction mode grounding and long-term planning. As shown in section 5.3, the interaction mode grounding experiment using RL required nearly 50 iterations to distinguish between closing and opening modes, with the average success rate for opening still not being high enough.In more complex scenarios with multiple articulated objects, this imbalance might affect the planning process if ActAIM2 is used as a prior. It can be difficult to determine whether the sampled interaction mode will lead to a "nothing happens" outcome or a rare interaction mode. Therefore, multiple sampling attempts are needed, reducing the efficiency of data use.Based on our experience with model training, we believe that an imbalanced dataset is the cause of this issue. Although we use the GMM adaptive data collection method to increase interaction mode diversity, the challenge remains that modes requiring precise grasping need more accurate action labels. To address the problem of imbalanced success rates, we propose collecting more stable data. Specifically, we suggest incorporating more expert tele-operation control data into the dataset to achieve more reliable grasping, rather than relying solely on completely self-generated data.In the experimental table, we observed a significant decrease in success rates from seen objects to unseen objects, and further to unseen categories. The success rate for generalizing across different categories dropped by nearly 50%. While ActAIM2 shows some level of scalability, it performs poorly when scaling across different categories.To address this scalability issue, we suggest training our model on more data. Currently, ActAIM2 is trained on only 6 categories with 8 unique instances, and we have collected 150 trajectories for each articulated object. We believe that incorporating a greater variety of object categories is necessary to better demonstrate scalability across different types of articulated objects.ActAIM2 is designed by decomposing the policy into a mode selector and an action predictor, where the mode selector defines distinct interaction modes. In this work, we define an interaction mode as a significant visual change in the articulated object, assuming that any visual change is meaningful if it pertains to the object. However, this definition doesn't hold true for tool manipulation tasks. For example, in a hammering task, simply moving the hammer around on the table without grasping it is meaningless, yet it would still qualify as an interaction mode under the current definition.This misalignment leads to inefficiencies during training when applying ActAIM2 to tasks like hammering. Additionally, data collection becomes a significant challenge due to the longer task duration and the need for more precise actions.To address these issues, we suggest first improving the data collection process and redefining the concept of interaction modes for tool manipulation tasks. Specifically, we should introduce stable grasping as an additional interaction mode and utilize large video-language models to assess whether the resulting visual changes are meaningful.Figure 19 : We illustrate objects from various categories that exhibit more complex interaction modes. These objects are qualitatively evaluated based on their sample success rates for each presented interaction mode. Video demonstrations of manipulating these objects are also available on our website.ConclusionActAIM2 marks a major advancement in self-supervised learning for robotic control, enabling robust discrete sampling across diverse interaction modes. Our specialized data collection and dataset lay a solid foundation for future enhancements. Extensive comparisons show ActAIM2's superior performance over existing models, enhancing discrete interaction mode learning and strengthening self-supervised discovery techniques for practical applications.Limitations. ActAIM2 offers a promising self-supervised method for learning interaction modes, but balancing success rates across modes remains challenging, especially for less common interactions. Iterative data collection targeting these modes could improve performance. Additionally, ActAIM2 assumes interactions are predefined simple action sequences, which limits its application to complex multi-stage manipulations. For a detailed discussion on assumptions and future work, seeAppendix 7.3. 
 
 Appendix:  


Decision: This work introduces a variant of ActAIM that uses a CGMVAE to represent action modes and that can handle multiple views. It appears to be conceptually simple, quite robust and effective in simulation. However, the reviewers perceive this work as rather incremental. It does not clearly state how exactly ActAIM2 differs from ActAIM, and it is unclear how it would fare in the real world. All reviewers ask questions about evaluation. A systematic ablation study would help. Also, Reviewer BZdG asks an important question about the relation to VQVAE-RVT.

As part of the rebuttal, the authors added an appendix describing real-world experiments, additional results, comparisons and limitations. It also satisfactorily answers many questions of the reviewers. The most important points should be integrated into the final version of the paper, if accepted.

Reviews: review: title: title: Interesting premise, good execution, although could be presented better
paper_summary: paper_summary: Clustering-based approach to manipulating articulated objects with multiple articulation mode
main_review: main_review: ## Summary

In this work, the authors present ActAIM-2, which is a clustering based method to understand different manipulation modes in objects that can be manipulated in multiple different ways, and then using the discovered manipulation modes to complete the (conditioned) manipulation task. The algorithm works in two stages: the first stage is to train the manipulation mode conditioning, which is trained via a clustering of difference between the first and last frame embeddings. Once these modes are discovered and clustered, the next step is to train a manipulation-mode conditional action predictor model. This model predicts actions in the next-best-pose style, thus bypassing the sequential-ness of most manipulation problem formulations.

The authors then evaluate this algorithm on manipulable objects on the SAPIEN dataset. Compared to the baseline, their algorithm shows higher success rate in both pure success rate and also on mode-specific success rate. Moreover, they show success in sampling modes and grounding the modes using reinforcement learning.

## Strengths

1. I thank the authors for listing out the assumptions, which is a good step to motivate future work.
2. Using GMM is simple, and it seems to work; not everything needs the most complicated algorithms.
3. Interaction mode discovery over multiple object types seem to work well, which shows the robustness of this algorithm.

## Weaknesses
While the paper reads interestingly, there are some anomalies in the result reporting that makes it less appealing. One is not reporting the VQ-VAE-RVT results in Table 1 (line 250-251)

> We did not report the VQVAE-RVT results here since VQVAE-RVT outperforms around 5% in each test compared to ActAIM2 averagely. However, despite the average sample success rate, we show that VQVAE-RVT does not meet our requirement for252
discrete sampling in the following two experiments.

The authors do not clearly mention the discrete mode sampling/RL fine-tuning as a goal too much in the beginning, and it feels like a post-facto justification to disqualify VQVAE-RVT and champion their own method. As a scientist I would like to know how much work was put in improving VQ-VAE RVT sampling, because the VQ-VAE codebooks are generally not of the same structure as normal scalar space, and so special care may be needed for proper sampling.

Secondly, relying on only SAPIEN simulator is slightly unsatisfying, since the physics there is not the most grounded in physics. It would be better to have some real world experiments as well.

Thirdly, the algorithm assumes multi-view RGB-D access which seems difficult to set up in the real world.

Finally, naming the method ActAIM2 seems somewhat anonymity breaking, given that ActAIM exists already.
questions: questions: 1. What are the blockers to applying this in a real world settings? Could the sampling based data collection be replaced by human demo collection in the real world?
2. What are the authors' opinion on why VQ-VAE sampling fails to get a second mode?
3. Why stop sampling modes at two? Shouldn't there be objects with more than two modes of interaction?
limitations: limitations: 2
review_summary: review_summary: I do not feel strongly about this work, and I wonder how it can be translated into reality. I am willing to change my mind based on a solid recommendation for real-world deployment.
title: title: Needs improvement
paper_summary: paper_summary: In this paper, the authors proposed a method to discover action modes for articulated objects and manipulate these objects using these modes. The authors trained a transformer-based Conditional Gaussian Mixture Variational Autoencoder (CGMVAE) to learn the action modes after collecting data by defining actions as moving the end effector between 4 poses: initiation, reaching, grasping, and manipulating. During data collection, grasping is handled by a pre-trained model, and a Gaussian Mixture Model(GMM)-based adaptive sampling is used to ensure the diversity of the data to discover modes. The success is defined by enough visual change between initial and end images. Afterward, a transformer-based action selector predicts actions using the interaction modes. The authors conducted simulator experiments to verify the model’s capability in comparison with recent methods [3, 43, 12] in the literature. The video only includes robot executions in simulation.
main_review: main_review: The paper is very well written. The motivation, method, and data collection are well presented, and the experiments include many objects and comparisons to support the paper. The limitations due to the work’s assumptions are clearly explained. Furthermore, it is positive that the data is shared with the community. I suggest the authors also open-source the models for reproducibility. 

Major concerns:

The novelty seems limited for this venue compared to ActAim [12]. After checking [12], I believe the major changes are the model updates. Instead of training a CVAE, a CGMVAE is trained, and the action predictor is updated with a transformer-based model that can handle multiple views. The main paper should clearly state the contribution on top of [12].

The method is not evaluated in the real world, which may raise concerns about its applicability. The reasoning behind the lack of real robot experiments should also  be stated. 

Other improvements:

Please explain the failure cases during the experiments, whether grasp, execution, or mode selection fails.

The limitations of the models should be stated as well. How would different colors, different instances, and different sizes of objects affect the behavior? Currently, it seems that it can somewhat generalize but a more detailed explanation is needed. 

The discussion section only summarizes the results. Explain the relation between design choices of approaches and performance differences. It does not need to be as detailed as the VQVAE-RVT comparison in the Appendix, but at the moment, it is not clear why ActAim2 is better than the baselines.

Please explain unintroduced symbols on page 4 and fix typos ‘/algoname’ in Fig. 11.
questions: questions: What are the differences between this work and [12] in the method and the data collection?

How would color change in the environment affect the model? Can different visuals of objects be in extrapolation regions of the models?

Is the pre-trained encoder used in CGMVAE?

What is a different instance of the object? What changes?

What are the second modes for one joint object, such as a faucet, drawer, and door?

What is the percentage of failures during random sampling in data collection?
limitations: limitations: 2
review_summary: review_summary: My questions are mostly answered. I updated my score.
title: title: Review for Submission461
paper_summary: paper_summary: ActAIM2 is a manipulation method learns a discrete representation of robot manipulation interaction modes in an unsupervised manner, without expert labels or simulator-based privileged information. The method consists of an interaction mode selector and a low-level action predictor. The selector generates discrete representations of potential interaction modes, while the predictor outputs corresponding action trajectories.
main_review: main_review: - **Clarity** - The paper presents an interesting idea but the method is difficult to follow. . The paper would benefit significantly from a clearer, more simplified explanation of the method and the experiment design.
- Originality and significance - Fair
- **Strengths**
    - I like the concept of combining object interactions with robot manipulation which has been used as a prior to enable autonomous data collection! This shows potential to improve the efficiency for robot manipulation.

- **Weaknesses**
    - Real world applicability may be limited - I am skeptical of how this method would fare on a real robot. While the concept of self supervised data generation already exists - my understanding of the data collection method is that it allows the robot to fail and then filters out the failures heuristically. 
    - The priors used are handcrafted, please clarify the usage of self supervised method - is it the autonomous data collection? 
    - While the method is interesting, the paper is hard to follow and could improve with better writing.
questions: questions: - Would the authors have any intuition to why the VQ-VAE learnt representations much worse than the GMM vae repesentations? Could this be attributed to codebook initialization?
- It is unclear if the method is self supervised, when all interaction modes are provided. Making the distinction in the paper would improve clarity.
- How are interaction modes significantly different from using affordances as priors? (Additional citations could help.)
- Figure 2: typos - multiview transformer(figure); mode selector(caption)
limitations: limitations: 1
review_summary: review_summary: I like the method presented but it is explained poorly. The current limitations section could also be improved.