FlowBotHD: History-Aware Diffuser Handling Ambiguities in Articulated Objects Manipulation
Authors: Yishu Li
Wen Hui Leng
Yiming Fang
Ben Eisner
David Held
Venue: CoRL 2024 (2024)
Keywords: Ambiguity
Multi-modality
Occlusion
Articulated Objects
Diffusion
TL;DR: We present FlowBotHD, a history-aware diffuser that handles multi-modality and occlusions in articulated object manipulation

Abstract: We introduce a novel approach to manipulate articulated objects with ambiguities, such as opening a door, in which multi-modality and occlusions create ambiguities about the opening side and direction. Multi-modality occurs when the method to open a fully closed door (push, pull, slide) is uncertain, or the side from which it should be opened is uncertain.  Occlusions further obscure the door’s shape from certain angles, creating further ambiguities during the occlusion. To tackle these challenges, we propose a history-aware diffusion network that models the multi-modal distribution of the articulated object and uses history to disambiguate actions and make stable predictions under occlusions. Experiments and analysis demonstrate the state-of-art performance of our method and specifically improvements in ambiguity-caused failure modes. Our project website is available at https://flowbothd.github.io/.

Flow History-Aware Diffuser Handling Ambiguities in Articulated Objects Manipulation
Yishu Lis, Wen Hui Leng", Yiming Fang*, Ben Fisner, David Held
Robotics Institute, School of Computer Science
Carnegie Mellon University, United States (yishul, wleng, yimingf, baeisner, dheld Gandrewn cmu. edu edu
Abstract: We introduce novel approach for manipulating articulated objects which visually ambiguous, such doors which symmetric or which are heavily occluded. These ambiguities
uncertainty over different pos- sible articulation modes: for instance, when the articulation direction (e.g. push, pull, slide) or location (e.g. left side, right side) of fully closed door are uncer- tain, or when distinguishing features like the plane of the door are occluded due to the viewing angle. To tackle these challenges, we propose a history-aware dif- fusion network that can model multi-modal distributions over articulation modes for articulated objects; our method further uses observation history distinguish between modes and make stable predictions under occlusions. Experiments and analysis demonstrate that our method achieves state-of-art performance ar- ticulated object manipulation and dramatically improves performance for artic- ulated objects containing visual ambiguities. Our project website is available at https://flowoothd.gg wtt.::/..............o....oo
Keywords: Ambiguity, Articulated Objects, Diffusion
DIFFUSION HISTORY
FlowBot HD
History- aware Diffusion
Multi Multi-Modality Lefo Hight? Pushtt pult?
Ceneral
Occlusions
Severe ocdusion os certain angles
2. Able to open Malti modal Door different conf guosions
Figure 1: Our method, FlowBotHD, consists of history-award diffuser that handles multi-man modality and occlusions articulated object manipulation. Simulation and real world experiments show our model's capability of opening general objects (including handling ambiguities and occlusions) as well as an ambiguous door.
Introduction
Ambiguities are common manipulation tasks. Consider opening a door without a visible handle or hinges: when the door is fully closed, the way in which it opens may be totally ambiguous.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

It may swing open from the left side or the right side, and may swing either inwards or outwards. Without further information, even experienced humans can't 't always tell by sight whether opens by push, pull or even slide. Another type of ambiguity is caused by occlusions. When a door opens to certain angles such that the door's principal plane is parallel to the viewing angle, might look like vertical line instead of flat panel (see Figure 3). Occlusions cause loss of information, introducing ambiguities in the understanding of the object shape.
Faced with object ambiguities, a human might interact with the object and observe the outcome to disambiguate the true mode; for example, a person might push the door and see if it opens, and not, they might try pulling the door open. Furthermore, momentary occlusions are not typically a problem for humans: we can remember past observations supplement our current perception. We therefore would expect an intelligent robot have the ability to learn from multi-modal train- ing data, predict multi-modal solutions for ambiguous observations, and with the help of historical observations, make stable predictions even during occlusions.
To generalize across different articulated object kinematics and geometries, we build on prior work that models articulated object manipulation with 3D articulation flow (3DAF) vectors as proposed in Eisner et al. [1]. This approach predicts set of 3D vectors for each point on the articulated object that indicates its opening direction. Results from this approach have shown impressive abilities to generalize to novel objects in the real world, after training only in simulation. However, since this prediction is unimodal and deterministic, this approach often fails to predict the correct flow when facing the ambiguous settings described above.
Therefore, this paper, we propose a history-aware diffusion method to model multi motaliy and handle occlusions for articulated objects. We use diffuuson model multi-modal distributions over multiple possible object opening directions, and we historical observations to adjust or disambiguate the predicted distribution accordingly. Our experiments and analysis demonstrate the benefits of our method in such ambiguous situations. The contributions of this paper include:
(1) A novel approach to model multi-modal ambiguity of articulated objects caused by visual ambiguity or occlusion.
A novel structure for incorporating object history disambiguate articulation modes.
(3) Simulated experiments across a wide variety of PartNet-Mobilit objects.
(4) Real-world experiments that demonstrate the feasibility of our method in real-world.
Related Work
Articulated Object Manipulation: Manipulating articulated objects is an important and chal- lenging task. One line of work investigates building manipulation systems with analytical meth- ods [2, 3, 4, 5, 6]. With the help of large-scale datasets [7, 8), recent lines of work turn to learning- based methods. Several works model manipulation choices with explicit articulation parameters and poses [9, 10, 11, 12]. To improve the model's generalization ability to different kinematies, several recent works learn to predict geometry-
kinematics-free representations such visual affor- dances [7, 13, 14] and articulation flows [1, 15] which achieve significant improvements in object generalization. Different from previous searnin-basse works which tend to use regression models for prediction, we propose to use diffusion models for better handling of multi-modality.
well-explored topic in various robotics fields [16, 17J. Previous works have included uncertainty in tasks including visual percep- tion and tracking [18. 19, 20, 21], motion planning [22, 23, 24, 25], safety control [26] and also articulation parameter estimation [27]. With similar motivation, our work focuses on building model robust to visual uncertainty caused by multi-modality and occlusions.
Uncertainty Modeling in Robotics: Modeling uncertainty
Diffusion Models in Robotics: Diffusion models, a class of generative models, learns to denoise an input approach given distribution. Following its impressive success in image generation, diffusion has also shown great potential in robotics tasks. Recent works have applied diffusion models to various tasks of robotics, including reinforeement learning [28, 29], imitation learning [30,

---

31, 32, 33, 34] and motion planning [35, 36, 37]. In contrast to prior methods, we show how 3D diffusion models can be used to handle ambiguities in articulated object manipulation.
3 Background: 3D Articulation Flow (3DAF)
Our method builds upon the 3D Articulation Flow (3DAF) representation of Eisner et al. [1], which we briefly review here. We assume that the parts of the articulated object are connected in a kine- matic tree with no closed-loop kinematic chains, and also that each joint has only one degree of freedom. Suppose that an articulated object is described by point cloud P {po. ................................ . } for a set of 3D points p: E R on the object visible surface. Following Eisner et al. [1], for each point Ps, we define the 3D Articulation Flow (3DAF) fi E I as the motion that point would undergo if there were an infinitesimal positive displacement 50 of the joint between that point's articulation part and its parent. Eisner et al. [I] showed that, given a suction gripper, we can most efficiently open the articulated object using the predicted 3DAF by attaching the gripper the point Pi... with the largest predicted flow norm émax arg max; ||llll and then moving the gripper along its predicted flow direction. The set of 3D Articulation Flows F {fo.-... ív-1) for all points in the point cloud is hierarchy-free and invariant under object translation and scaling, which allows for effective learning over a large set of objects of different categories and effective transfer from simulation to the real world.
Method
The task we propose to tackle is opening fully-closed articulated objects from a single-view depth- sensor. The inputs to our model are a history of 3D point cloud observations of the articulated object being manipulated. We assume that the articulated objects do not consist of closed-loop kinematic chain and that each joint has only single degree of freedom.
We aim to enable a robot to manipulate ambiguous articulated objects, focusing on two types of ambiguity: ambiguity in opening direction due to lack of visual cues, and ambiguity due to self- occlusion during articulation. Both types of ambiguity are known failure modes of prior determin- istic systems [1]. At high level, our objective is predict all the modes that describe how potentially ambiguous articulated object might open, and then use these predictions determine how to manipulate the object. Below we will describe our proposed history-aware diffusion model which can represent multimodality (Section 4.1) and is robust to occlusion (Section 4.2), as well as a system for manipulating articulated objects based on multimodal predictions (Section 4.3).
4.1 Diffusion for Multi-Modality
Our system builds on the 3D Articulation Flow representation from prior work [1] described in Sec- tion 3. Our first task is to construct system which accurately predict multi-modal affordances for articulated object, in which affordances are captured by 3D Articulation Flow. For instance, given an ambiguous door that can open right or left (as shown in Figure 4), we'd like to be able to generate one set of flow vectors to describe the possibility that the door opens rightward, and a dif- ferent set of flow vectors to capture the possibility that it might open leftward. Diffusion processes have been shown to be a good choice of model architecture for this setting, as they can represent multi-modal distributions over high-dimensional data [38. 30].
We now describe how we can use diffusion to predict multi-modal 3D Articulation flow. To incor- porate observed 3D point cloud inputs Pobs into the standard diffusion model paradigm, we use PointNet++ [39] encoder Is that encodes the current inputs and a TransffomeT (DiT) [40] denoiser Dg. During training, we first add noise to the ground truth flow F.,, to obtain a noisy flow F following the DDPM [38] diffusion formulation. Then we pair each point and its corre- sponding noisy flow as the current state S = {pobs, in J [1... [I...N] NJ. where N is the total number of observed points in Pots We encode the current state Sour Vots go (Scur) to obtain 3D scometry-awant point-wise features. Following the standard diffusion training procedure [38),
3

---

Current Observation Encoder
Current Observation
+
History-aa aware Denoiser
without Positional Embedding
Point Cloud
Scate Embeddiling
tiick
Noisy fovv
7
70
FC
History
History Injection
Predicted
Supervise
Ground Truth Noise
Hist History PCD
History
History Row
PointNet
History Encoder
Figure 2: FlowBotHD structure: history observation and the history flow are input into history encoder to obtain global history latent. This history latent injected into the current observation encoder through fully-connected projection layers and element-wise multiplication. The current observation encoder outputs history-aware point-wise embedding. The history-aware embedding is then input to DiT-based denoiser to predict the 3D articulation flow [1], i.c. the predicted future motion of the object it were to be opened a small amount.
we train the denoiser D3 and encoder 94 with the following learning objective:
[(0, Φ) Bl[le, Do( Do(Vo -1- L, Uo (Scur))
From the predicted noise, we estimate the flow following the standard diffusion equations [38].
4.2 Aggregating History for Disambiguation and Occlusion Handling
While this proposed 3DAF diffusion model is capable of modeling multi-modal distributions, it lacks the ability disambiguate based on past interactions. For example, suppose the door occluded in current observation, shown in Figure 3; if the robot succeeded in the past timesteps partially open the door, it should output distribution that places more probability a flow which agrees with the previously observed motion.
To accomplish this objective, we propose to modify the above architecture to integrate the past history with history encoder. We define a compact history state representation the previ- time step's point cloud observation Phis- along with the previous step's flow Friss as Sais {Pris, fuis, J, [l... MM), where M is the number of points in the previous point cloud. We then pass Sais though global point cloud encoder h. based on PointNet++ [39] to get global history context vector This h., (Snis) E Rdu,, where dnis is the fixed dimension of the history latent. We then inject this additional context into the "decoding" layers of our U-net encoder Jø. similar to the method proposed by Shridhar al. [41] (see Figure 2). For each decoding layer j of gs with point- wise activations a, E Rd we apply a learned linear projection W, E IR Rhh xd, the history latent Phiss and combine with the original activation using the Hadamard product: a/ = aj @ (W; ahis).
During training, we generate a synthetic history by sampling a random small angle to perturb the object to generate the "history" point cloud observation, and we obtain the corresponding ground truth articulation flow as the history flow. At inference time, we use the model's ' previous point cloud observations and previous flow predictions history. To account for situations at inference time when no history yet exists (i.e. the first timestep), and include learnable "start" vector 2n in place of the history vector Ahis in such cases. We train this model end-to-end with the standard diffusion objective outlined in Section 4.1.
also train with samples that include history,

---

4.3 Articulation Manipulation Policy
Along with the history-aware diffusion model, we propose a novel articulation manipulation policy that utilizes the flow predicting model real manipulation tasks, taking into consideration the multi- modality of diffusion and the incorporation of object history.
Switch Grasp Point (SG): Following Eisner et al. [1], to achieve the best manipulation efficiency with a suction gripper, we want to grasp at the point with the greatest leverage; thus the optimal action given the predicted flow is to grasp the point that has the maximum flow norm [1]. Due to the multi-modality of diffusion, the prediction at different steps will vary; each prediction might imply a different point that the robot should grasp for maximum leverage. In order to correct for past potential grasping mistakes as well to maintain a reasonable re-grasping cost, we propose to switch the grasp point whenever the leverage (defined the current largest flow norm) of the newly predicted grasp point is larger than that of the current grasp point by a threshold: || marlll - || 6,, where F is the current step's 's flow prediction for all points in the point cloud, g is the index of the point that is closest to the current grasp point, and 6 is the threshold which we set to 0.2 in practice.
Consistency Check (CC): Learned diffusion models may not always perfectly represent the true distribution over cistory-conditioned 3DAF values; for example, they might predict multiple modes even when there is unambiguously only a single mode. To handle this, we perform a consistency check for every prediction after the first. Assuming that our model will sample the true mode given enough samples for any observation, we design a heuristic consistency check to filter the actions that are not consistent with the previously executed motion. Specifically, we define the consistency direction Ghis as the measured gripper motion caused by executing previous selected flow fais (we only keep the previous execution its motion was greater than 0). For a subsequent step to be considered "consistent", the predicted action's 's direction shouldn't deviate too much from that of the
fri previous step; thus we angle threshold eg for the consistency check: arccos [1 60.
where f.ax is the current maximum sampled flow direction fmax = max, ft. We sample 3DAF flows from the diffusion model and compute maximum--eveergg actions [1] until we find one that satisfies the consistency check, before executing the sampled action.
History Filter (HF): If we rely solely on consisteney check for next actions, the model may end up in a degenerate state where its previous predictions are incorrect, enforcing that only incorrect predictions are permitted by consistency. Therefore, we include a history filter that only updates the history point cloud Pais and history flow Fris if the previous action successfully achieves reasonable gripper motion without losing contact. Specifically, after an execution, we only update the history two criteria are satisfied:
1. The measured gripper motion Jour caused by the currently-executed action is consistent with past executions: arccos Rea--uu Ohis
|1-1 < C. 909 where Jeur is the current gripper
movement and Ghis is the gripper movement from the previous execution.
2. The gripper motion Jeur is larger than threshold ||gearl| > Cm where Jeur is the current gripper movement and cm is the threshold
The pseudocode for our method, FlowBotHD (History and Diffusion) can be found in Algorithm 1.
Experiments and Analysis
We use the PartNet-Mobillty dataset [7] to train and evaluate the models. We split subset of this dataset containing 21 categories of objects (the same categories as in prior work [13, 1]) into a train set and test set in which the test set consists of unseen samples of the training categories (707 training objects and 166 testing objects). To enable the models to open objects from fully closed state, we create mixed dataset which contains both randomly opened samples (as in prior work [1]) and training samples that are fully closed. The fully closed examples the most ambiguous, since it might be unclear which direction these objects open.
5

---

Algorithm 1: The FlowBotHD articulation manipulation policy
Require: 0 parameters of trained FlowBotHD network D while NotFullyOpened() do
P obs - Observation
F (P ubs Stis) // Sample the current flow direction from the diffusion model
f fmax c GetMaxFlow(F) // Find the point with the max predicted flow [1] then
if Inconsistent (faxx Ihis) and not ExceedMaxTrialPerrseae
continue;
// If the sampled action fails the Consistency Check // (Sec. 4.3) then resample
end if:
Parosp +- SelecttrraspPont(()
if NeeedssttchGrasppoiiiii (PgrsspPPPPsphii Pgrasphiis Pgrasp) then
Phrmsphhis Perasp
// If the Switch Grasp Point check is passed // (Sec. 4.3) then change the grasp point
Crasp(Pgrap):
end if
Apply force to the gripper in the direction of f for small duration 8t.
Jeur GetDeltaGripper((
if Consistent Caaasssttet((uurr:e. (9eur: 9his) then
Phis f Jeur
if Seur > Em then
Shis f- (P obs: F)
11 Apply the History Filter (Sec. 4.3)
end if
end if'
end while
5.1 Simulation results
To evaluate each model's ability to open objects, we construct simulation environment in the Py- Bullet [42] simulator consisting of suction gripper and objects from the PartNet-Mobility dataset, where the environment's 's goal is to successfully articulate the object. We initialize every object as fully closed and filter out instances that cannot open from the fully closed state even with the ground truth action sequence (e.g. due self-collision). Following prior work [1], we compute evaluation metrics: Success Rate: the percent of objects the model succeeds open 90% of the joint's full range of motion, and Normalized Distance [13]: the mean distance between the end state of this
|| Juteaall trial and the fully opened state, normalized with its range of motion dgoul | - Jaksel where Jend the joint angle at the end the policy rollout, Jelosed and japened are the fully closed angle and the fully opened angle respectively. We evaluate each test object for trials.
Baselines: To compare model with the original FlowBot3D [1] setting, trained FlowBot3D on dataset with only randomly opened ("RO") samples which aligns with the original training strategy of Eisner et al. [1]. We also retrain Flowbot on the mixed dataset ("MD") that we use to train our model in which we also include examples in which the object is fully closed. Further, for a fair comparison to our method, we implement a baseline which we allow FlowBot to switch its grasp point ("SG") using the same criteria as for our method.
Comparing the Baseline models with Ours in Tables and 3 (see Appendix for normalized distance metrics), we can see that our full method (FlowBotHD) with the complete set of policy improve- ments (switch grasp point + consistency check + history filter) achieves the best overall performance. Specifically, compared to the original FlowBot3D [1] model, it increases the success rate averaged across all samples from 87% to 95%, or 61% reduction error rate (from 13% to 5%). We ob- serve a particular increase in success rate in the door category (from 23% to 77%), which contains the most visual ambiguities of any category when instances are fully closed. These quantitative re- sults demonstrate our model's ability to overall improve the performance and more importantly, to better handle multi-n molti- cases.

---

CC HF AYG. AVG,
Baselines
(RO)
0.71
0.23
FlowBot (RO)
0.793
0.903
11.85
0.75
0.82
0.60
0.87
0.55
0.82
0.96
0.60
0.53
1.00
0.86
0.50
FlowBat (MD)
X
0.800
0.875
00.60
0.80
1.00
0.82
1.00
1,00
0.89
0.76
0.33
0.72
0.91
0.91
0.78
0.92
0.80
0.57
1.00
1.00
1.00
FlowBot
Ablations
Ohrs- No History
0.692
0.806
[0.70 0.00
0.75
0.84
0.84
0.60
0.82
0.77
0.44 0.55
0.80 0.86
0.76
0.72 0.95
0.20 0.47
1.00
0.86 0.95
History
(0.70
Ours- No History
0.743
0.876
(0.70
0.30
0.75
0.95
0,92
1.00
0.73
0
0.11
0.84
0.94
0.91 0.84 0.92 0.57
0.20
0.53
1.00
0.86
0.70
Oun- No Diffusion
0.673
0.720
0.55
0.30
0.45
0.89
1.00
0.40
0.85
0.30
0.56
0.82
0.70 0.58 0.16 0.57
1.00
0.40
1.00
1.00
No Diffusion
0.683
0.735
(0:50
0.89
0.10
0155
0.58
0.08
1,00
0.53
1.00
1.00
Ours
BotH
X
0.735
0.865
00.35
0.10
0.90
1.00 0.96
0.50
0.89
0.30
0.36 0.99 0.55
0.92 0.49 0.92 0.47
1.00
0.43
1.00
0.83 0.95
B
0..884
0.937
(0.65
0.96
0.64
0.70
1.00
0.95
FlowBorHD
0.758
0.904
00.30
0.00
0.85
1.00
0.96 0.90 0.96 0.50 0.45 0.99 0.95 0.95 0.64 0.92 0.53
0.90
0.43
1.00
0.86
0.85
BotHB
(0.75
0.98
Table 1: Success Rate Metric Results (t): Fraction of successful trials (normalized distance 0.11) of different object categories after a full rollout across different methods; the higher the better. AVG and AVG, refers metric averaged over categories samples. For the policy column, SG refers to switch grasp point, CC' refers to consistency check and H I' refers to history filter. We only add history filter in history-award methods and only add consistency check with diffusion-based methods because inconsistent predictions in deterministic methods can't be refined even with re- re-predictions.
OCCLUSION
FlowBo33D
Flow Visuaiization at 90'
Performance
Drop under
Coonn Cootnsrriisit iisitt 0.04
Fully Closed
Open 30
FlowBotHD
Casain siillatyy 0.6
Performance
Under
Fully Open 140"
Open 90* (Severely Occluded)
Figure 3: We demonstrate our model's performance improvement over the baseline in severely oc- cluded case. We open the object different angles, make predictions, and plot the RMSE metric (L) against the open ratio. The baseline (top) fails at the point of severe occlusion, whereas our method (bottom) continues make stable predictions using history. We also include flow visualizations (right) to show the quality of the predictions in both cases.
5.2 Ablations
To understand the impact of our design decisions, we perfomed ablation studies in which we re- move various components of the model: No history, no diffusion, and varying whether we use switch grasping ("SG" column), consistency check ("CC" column), and history filtering ("HF" col- umn). Looking Tables 1 and 3, observe large performance improvements when including history awareness in the diffusion model (comparing Ours to "Ours- No History").. This improve- ment demonstrates that the extra guidance of history is beneficial for diffusion to reduce the multi- modality in cases of past success in which the opening direction is now disambiguated.
These tables show that our policy improvements of switching grasp point, applying consistency check, and applying history filter all contribute increased model performance. The importance of the consistency check potentially implies that one challenge for using diffusion models for visual

---

WWhtenn the tnttal prod correct (aligns with the doar co figurationn::
Custom Made
HowBoOHHL
Possible
When the inttill predid on wrong Iniseligns with the door coniguration:
Ns civieenntion
PUT
Figure 4: Demonstration of FlowBotHD opening custom made door in different configurations. The first two rows show examples in which the model's s initial predictions aligned with the current door configuration. The last row shows example in which the model's initial prediction fails and the model switches its grasp point, ultimately leading to a success.
manipulation policies is randomness and, result, inconsistency. By filtering out inconsistent predictions, policy takes actions that lead to higher success rates than without filtering.
5.3 Qualitative Analysis
To understand the ability of our model to handle occlusions which are not immediately apparent from our quantitative metrics, in Figure 3, we visualize the model predictions for different door opening angles. At certain angles, the point cloud is severely occluded such that only the vertical edge of the door visible from the camera's perspective. We can see from the RMSE plot that FlowBot3D consistently mispredicts the 3DAF examples where the opening angle causes severe occlusions. FlowBotHD performs substantially better in these occluded cases by leveraging history, predicting the correct direction even at the point of occlusion. The flow visualization shows clear improvement between our model and the baseline for this ambiguous case.
5.4 Real-World Experiments
We performed real-world experiments on general articulated objects, as well a custom made multi-modal door demonstrate FlowBotHD's ability to function in real-world ambiguous scenar- ios (Figures 1 and 4). For the multi-modal door, we demonstrate our model's ability to open the door in all 4 possible modes, and also the ability to adjust to better grasp points with the switch grasp point policy. Static visualizations of the real-world roll-outs on our custom-made door and general articulated objects are presented in Figure 4 and Appendix Figure 15, and full videos can be found on our project website. Details of our real-world setup can be found in Appendix F.
Conclusion
In this work, we propose a history- aware diffusion model FlowBotHD to handle ambiguities en- countered while manipulating articulated objects, along with a carefully designed heuristic manipu- lation policy. Both quantitative experiments and qualitative analysis show the overall effectiveness of our model and specific improvements in multi-modal severely occluded cases.
Limitations: While our model greatly improves the ability of the policy to handle ambiguities, we have observed trade-offs in accuracy and consistency when building on top of a diffusion pipeline. Our diffusion model sometimes makes multimodal predictions even when the object is opened enough to disambiguate the true opening direction. While we currently mitigate this issue by adding a consistency check in the policy, an ideal model would learn disambiguate perfectly based solely history. Therefore, we hope to further explore how to exert better control over the diffusion's 's learned distributions for better consistency and accuracy.

---

Acknowledgments
This material is based upon work supported by the National Science Foundation under NSF CA- REER Grant No. IIS-2046491 and Toyota Research Institute.
References
[1] B. Eisner, H. Zhang, and D. Held. Flowbot3d: Learning 3d articulation flow to manipulate articulated objects. arXiv preprint arXiv:2205.044 04382, 2022.
[2] R. Martin-Martin, S. Höfer, and O. Brock. An integrated approach to visual perception of articulated objects. 2016 IEEE international conference on robotics and automation (ICRA), pages 5091-5097. IEEE, 2016.
[3] K. Desingh, S. Lu, A. Opipari, and O. C. Jenkins. Factored pose estimation of articulated objects using efficient nonparametric belief propagation. 20/9 International Conference on Robotics and Automation (ICRA), pages 7221-7222 IEEE, 2019.
[4] B. Abbatematteo, S. Tellex, and G. Konidaris. Learning to generalize kinematic models to novel objects. In Proceedings of the 3rd Conference on Robot Learning, 2019.
[5] R. Staszak, M. Molska, Mlodzikowski, J. Ataman, and D. Belter. Kinematic structures estimation the rgb-d images. In 2020 25th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), volume I, pages 675-681. IEEE, 2020.
[6] A. Jain, R. Lioutikov, C. Chuck, and Niekum. Screwnet: Ennegory-independeed articu- lation model estimation from depth images using screw theory. 202/ IEFE International Conference Robotics and Automation (ICRA), pages 13670-13677. IEEE, 2021.
[7] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11097-11107, 2020.
[8] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su. Partnet: A large-scale benchmark for fine-grained and hicrarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 909-918, 2019.
[9] L. Yi, H. Huang, D. Liu, E. Kalogerakis, H. Su, and L. Guibas. Deep part induction from articulated object pairs. arXiv preprint arXiv:1809.0744 07447, 2018.
[10] X. Li, H. Wang, L. Yi, L. Guibas, A. Abbott, and S. Song. Category-level articulated object pose estimation. In Proceedings of the IEEE/VV conference computer vision and pattern recognition, pages 3706-3715, 2020.
[11] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu. Shape2motion: Joint analysis of motion parts and attributes from 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8876-8884. 2019.
[12] R. Hu, W. Li, O. Van Kaick, A. Shamir, H. Zhang, and H. Huang. Learning to predict part mobility from single static snapshot. ACM Transactions on Graphics (TOG), 36(6):1-13, 2017.
[13] Z. Xu, H. Zhanpeng, and S. Song. Umpnet: Universal manipulation policy network for articu- lated objects. IEEE Robotics and Automation Letters, 2022.
[14] R. Wu, Y. Zhao, K. Mo, Z. Guo, Y. Wang, T. Wu, Q. Fan, X. Chen, L. Guibas, and H. Dong. Vat-mart: Learning visual action trajectory proposals for manipulating 3d articulated objects. arXiv preprint arXiv:2106. 211222106.144400 14440, 2021.

---

[15] H. Zhang, B. Eisner, and D. Held. Flowbot++: Learning generalized articulated objects ma-
nipulation via articulation projection. arXiv preprint arXiv:2306. /2893, 2023.
[16] H. F. Wurrant-Whyte. Uncertain geometry in roboties. IEEE Journal Robotics and Automa- tion, 4(1):23-31, 1988.
[17] Thrun, Probabilistic algorithms roboties. Ai Magazine, 21(4):93-93, 2000.
[18] R. Smith, M. Self, and P. Cheeseman. Estimating uncertain spatial relationships in robotics. In Machine intelligence and pattern recognition, volume 5, pages 435-461. Elsevier, 1988.
[19] M. Mohammadzadeh Babr, M. Paghihabdolahi, D. Ristic-Durvant, and K. Michels. Deep learning-basee based occlusion handling of overlapped plants for robotic grasping. Applied Sciences, 12(7):3655, 2022.
[20] H. Chandel and Vatta. Occlusion detection and handling: review. International Journal of Computer Applications, 120(10), 2015.
[21] P. Wunsch and G. Hirzinger. Real-time visual tracking of 3d objects with dynamic handling of occlusion. In Proceedings of International Conference on Robotics and Automation, volume 4, pages 2868 2873. IEEE, 1997.
[22] A. Lazanas and J.-C. Latombe. Motion planning with uncertainty: a landmark approach. Arti- ficial intelligence, 76(1-2):287-317, 1995.
[23] G. Kahn, P. Sujan, S. Patil, S. Bopardikar, J. Ryde, K. Goldberg, and P. Abbeel. Active explo- ration using trajectory optimization for robotic grasping in the presence of occlusions. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 4783-4790. IEEE, 2015.
[24] I. Zhang and J. C. Trinkle. The application of particle filtering to grasping acquisition with visual occlusion and tactile sensing. In 2012 IEEE International Conference on Robotics and Automation, pages 3805-3812. IEEE, 2012.
[25] S. Dasgupta, Gupta, S. Tuli, and R. Paul. Uncertainty- aware active learning of nerf-based object models for robot manipulators using visual and re-orientation actions. arXiv preprint arXiv:2404.0I1 01812, 2024.
[26] Choi, F. Castaneda, C. J. Tomlin, and K. Sreenath. Reinforcement learning for safety- critical control under model uncertainty, using control Iyapunov functions and control barrier functions. arXiv preprint arXiv:2004.077 075584 2020.
[27] A. Jain, S. Giguere, R. Lioutikov, and S. Niekum. Distributional depth-based estimation of object articulation models. In Conference on Robot pages 1611-1621. PMLR, 2022.
[28] Z. Wang. J. J. Hunt, and M. Zhou. Diffusson policies expressive policy class for offline reinforcement learning, arXiv preprint arXiv:2208.06/99 2022.
[29] A. Ajay, Y. Du, A. Gupta, Tenenbaum, T. Jaakkola, and Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15653 2022.
[30] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 2023.
[31] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301. 10677, 2023.
[32] M. Reuss, M. Li, X. Jia, and R. Lioutikov. Goal-conditioned imitation learning using score- based diffusion policies. arXiv preprint arXiv:2304.02532. 2023.
10

---

[33] Z. Xian, N. Gkanatsios. T. Gervet, T.W. Ke, and K. Fragkiadaki. Chaineddiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In 7th Annual Conference on Robot Learning. 2023.
[34] Y. Ze. G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024.
[35] K. Saha, V. Mandadi, J. Reddy, A. Srikanth, A. Agarwal, B. Sen, A. Singh, and M. Kr- ishna. Edmp: cooemmlle-o--oss--geue ar09v:2309. 1414, 2023.
diffusion for motion planning. arXiv preprint
[36] M. Janner, Y. Du.
J. B. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 229.999000, 2022.
09991,
[37] A. Simeonov, A. Goyal, L.. Manuelli, L. Yen-Chen, A. Sarmiento, A. Rodriguez, Agrawal, and D. Fox. Shelving, stacking, hanging: Relational pose diffusion for multi-modal rearrange- ment. arXiv preprint arXiv:2307.04751, 2023.
[38] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-666 2020.
[39] C. R. Qi, L. Yi, H. Su, and L. J. Guibas. Pointnet++: Deep hierarchical feature learning
point sets in metric space. Advances in neural information processing systems, 30, 2017.
[40] W. Peebles and S. Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[41] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Conference on robot learning, pages 894-906. PMLR, 2022.
[42] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016.

---

Appendix
Table of Contents
A Training Details
12
B Simulation Details
13
Multi-modallty Analysis
C.I Multi-modality of Dataset
14
C.2 Model Performance on Multi-moda Cases
D Occlusion Analysis
17
E Policy Analysis
17
E.1 History Filter Analysis...
17
Switch Grasp Point Analysis
E.3 Step Number Analysis
21
F
Real-world Equipment
21
F.1
Hardware
21
F2
Workspace
21
E3
Foreground Segmentation
21
F4
Robot Control Paradigm
22
F5
Custom Multi-modal Door
22
A Training Details
Model Structure: FlowBotHD consists of current state encoder Js, a history state encoder h and denoiser Dg. Both current and history state encoder are based on PointNet++ [39], where the history encoder only includes the PointNet++ global encoding module that outputs global latent of 128 dimensions. We kept the same model architecture and hyperparameter set as the original PointNet++ paper. We inject the history latent into every layer of the PointNet++ framework in the decoder (see Figure 2). This is performed through Hadamard product of the current point cloud encoding with the latent encodings every decoding layer to produce the final output of the model. We base the denoiser on DiT with 5 layers, 4 heads, and hidden size of 128.
Dataset Details: Similar to the original FlowBot3) paper [1], we create a randomly opened dataset (RO) in which we randomly sample the view perspective, the target joint, and the open ratio. To open objects from the fully closed state, we create another mixed dataset (MD) in which we half of the objects fully closed and the other half randomly opened. To enable history-aware training, generate dataset with 1/3 of the objects fully closed, 1/3 randomly opened but without history and the final third randomly opened and with history at random interval.
We set diffusion timestep to be 100 steps. We train the model for 450 epochs using AdamW optimizer with a weight decay of 1e-5. We use learning rate of le-4, and batch size of 128. We evaluate the model every 20 epochs, and save the checkpoint with the best winner- take-all RMSE metric. The Winner-take-al metric means that repeatedly make predictions for each sample 20 times and record the best RMSE.
Training Process:
12

---

B Simulation Details
We design a simulation environment in the PyBullet simulator that simulates operating a suction gripper to operate MorDiet-Mobllty objects. We first roll out the simulation with ground truth flow and filter out samples that can't open with the ground truth motion, due errors the simulator. We repeat each sample for times during evaluation. The final test object categories evaluated on after filtering and repeating are listed in Table 2.
Icon
Name
Count
Icon
Name
Count
Icon
Name
Count
Icon
Name
Count
Oven
20
Microwave
10
Table
245
Phone
Bucket
10
Window
55
Furniture
765
Door
30
Box
20
Kettle
25
Dishwasher
45
Laptop
40
Toilet
Safe
12
WashMachine
25
TrashCan
35
KitchenPot
25
Refrigerator
75
FoldingChair
15
Stapler
20
Table Simulation Objects and Sample Counts
To manipulate the objects, we first attach the suction gripper to the object. This achieved by first teleporting the gripper to position close the target grasp point and then gradually moving towards the target grasp point until contact is detected. After contact, apply physical force constraint between the gripper and the object. The move action is then implemented applying certain velocity along the predicted direction.
Hyperparameter in Policy: There are 3 hyperparameters to set in Algorithm 1: the threshold for switch grasp point et, the threshold for consisteny check eœ and the threshold for good movement Cm- We set c 0.2, ca 30 and 6. =le- in simulation. Table 3 demonstrates the quantitative simulation results for the normalized distance metric, computed in the same way as in Eisner et al. [1].
SG CC HF
AVG. AVG.
Baselines
FlowBot (RD)
0.220
0.117
0.16
0.53
0.29
0.22
0.00
0.43
0.10
0.15
0.45
0.14
0,05
0.09
0818
0.09
0.27
0.31
0.46
0.07
0.15
0.26
FlawBoi (MD)
0.183
0.143
0.31
0.19
0.06
0.22
0.00
0.06
0.14
0.21
0.47
0.30
0.09
0.12
0.23
0.12
0.09
0.24
0.63
0.07
0.05
0.08
FlowBoi
0..66
Ablations
Ours- No History
x
0.255
0.176
0.29
0.37
0.18
0.19
0.16
0.36
0.14
0.23
0.45
0.28
0.18
0.14
0.20
0.31
0.07
0.53
0.54
0.07
0.09 0.10
History
Ours No History
0.176
0.110
0.28
0.28
0.19
0.10
0.08
0.06
0.10
0.12
0.75
0.11
0.07
0.10
0.10
0.08
0.10
0.11
0.45
0.07
0.11 0.27
Oun- No Diffusion
0.265
0.213
0.35
0.55
0.15
0.00
0.51
0.14
0.19
0.31
0.15
0.31
0.37
0.11
0
0.55
0.07
Oun- No Diffusion
0.194
0.183
0.39
0.40
0.11
0.45
0818
0.17
0.30
0,28
0.35
0.08
0.48
0.07
0.15 0.07
Ours
DDd
0.181 0.139
0.55
0.50
0.13
0.05
0.01
0.39
0.12 0.15
0.33
0.08
0.15
0.10
0.10
0.11
0.09
0.52
0.07
0.10 0.09
Flow HooDBoDDD
0,103
0.086
0.33
0.05
0.00
0,06
0.06
0.02
0.08
0.07
0.25
0.09
0.10
0.27
0.07
0.08
Bo
0.110
0.096
0.53
0.52
0.13
0.05
0.04
0.10
0.06
0.15
0.24
0.07
0.05
0.07
0.21 0.13
0.23
0.11
0.49
0.07
0.12 0.10
Flow
0.25
Table 3: Normalized Distance Metric Results (І): Normalized distances to the goal articulation joint angle after a full rollout of the methods. The lower the better.
We demonstrate the simulation process in Fig 5. We visualize the flow predictions (the first row), the simulation process (the last row, x-axis the step number, the y-axis is the open ratio), and the trajectory plot along with policy signals (the middle plot). In these visuals, we can see our model and policy's 's pattern opening object: make one or more attempts at opening the beginning of the episode, and once the door is opened a bit, make consistent predictions based on past history and consistency check.
13

---

The first example is quite smooth: our model succeeds in moving the door on the first step. It con- tinues to make consistent and history-aware predictions until the door is fully opened. Qualitatively, the consistency check didn't 't filter out many predictions due to high prediction quality, and due to the smooth execution the history is updated at each step, meaning that the policy always uesd the pre- vious step's 's history for the next step's prediction. In the second example, our model makes several attempts open the object when in the fully closed state. The observation history is not updated during these steps. Once the door is opened a bit, the history information and the consistency check enable the model to execute consistent actions. In the bar plot, we se that for step 11, the consistency check filters out 12 inconsistent predictions, demonstrating the effectiveness of applying consis- tency check. Also, we can see that at step 6, the door is opened a bit, but the history is not updated due to the small movement (indicating the prediction not good enough) which demonstrates the importance of tuning the history filter.
C Multi-modality Analysis
C.1 Multi-modality of Dataset
Analyzing how many of the objects in the dataset are provides valuable insights of the dataset. Take door category as an example, doors without handle typically have 4 possible modes when fully-closed (push left, push right, pull left and pull right); handles can indeed disambiguate which side of the door opens but still leaves ambiguity on whether the door opens by push or pull. Among the 28 training door joints and 7 test door joints, most of them (20 out 28 for training, out of 7 for test) are with handles (2 possible modes- push and pull), and the other samples are without handles (4 possible modes push left, pull left, push right and pull right).
or typical multi-modal paradigms
For other object categories, since there is generic way determine whether an object is ambigu- ous, it is hard to manually count ambiguous samples for all the categories (not to mention ambiguity might be subjective), therefore we describe broadly where ambiguities typically happen. There are three typical multi-modal objects (See Figure 6) : 1) Joints without handles, 2) Joints with small handles that might be erased when downsampling the point cloud, and 3) Joints with handles that may afford different opening modes: for example a dishwasher/oven with a handle on the top of its front surface, although the handle eliminates many modes, it is unclear whether it opens by pulling out like drawer, or pulling from the top to the bottom like a lid or door.
Similar to counting modes of general articulated objects, there is no precise way to count multi- modality in predictions without including manual labeling. To approximately evaluate the multi- modal predictions of the model, we make 50 predictions for each object in the test set and calculate their RMSE with respect the ground truth flow. We record how many of the objects include both RMSE < 0.2 (presumably correct prediction) and RMSE 0.6 (presumably different mode than the ground truth mode). We can from the bar plot in Figure 7 that the model predicts multi-modality mostly in conceptually multi-modal categories like Door, WashingMachine, and Re- frigerators. However it also makes multi-modal predictions with conceptually unambiguous cases, which aligns with limitations of this work we addressed in 6 about the difficulty of striking a balance between precision and multi-modality.
C.2 Model Performance Multi-modal Cases
To specifically analyze our model's improvements on handling multi-modality, we evaluated base- line FlowBot3D trained on randomly opened dataset (RO) mixed dataset (MD) and our model FlowBotHD on doors. We open each test door from 0% to 100%. We average the metrics within 10% open to compute the performance for closed doors, and we average the metrics above 10% open to compute the performance for open doors. As we the basclines by large margin across all metrics. We observe that FlowBot3D trained on randomly
from Table 4, model outperforms
14

---

SGP
2.00
0.9
Hittory
175
0.7
1.50
05-
1.00
0.73
022
0.50
0.0
0.00
10
15
20
25
500p
SCP
Hatoory
Open 688
0.0
10
15
20
2kpp
Figure 5: Simulation Visualizations: The plot in the middle is a simulation trajectory plot with x-axis step number, and left y axis the open ratio. We visualize the history update signal with red polygons, step with red polygon means updating this step's prediction as the latest history. Yellow triangle represents the switch grasp point (SGP) signal, meaning that this step requires new grasp point. The bar plot on the background corresponds to number of trials we take to generate prediction that satisfies the consistency check trial. The axis for the bar plot is the right y-axis.
15

---

Door (Withou Hardles]
Door (WIP Handle]
Fridge (Nichout Handles)
Wasthin Machine (Alifout Handlas)
mittiigh riseed
With (lexdi
Conceptually, ambiguities typically happen with 1) Joint without handles, 2) Joints with small handles that might be missed when downsampling, and 3) Joints with handles which are still possible to open in different ways.
Figure 6: Ambiguous Showcases:
Multimodal Prediction Ratio (Test set)
0.6
0.5
0.4
Padi
0.2
0.1
0.0
FohdingChair
Table
kattle
Laptop
WashingMachine
Oven
Stapler
Door
Bucket
TrashCan
Dishwasher
Tbilet
Microwave
Safe
Refrigerstor
KitchenPot.
StorageFumiture
Category
Figure 7: Multi-modal Prediction Ratio for Different Categories: We make 50 predictions for each object in the test set, calculate their RMSE with the ground truth flow, and record how many of the objects include both correct prediction and different modes.
16

---

Mode
step
100 step
dooed
Obseiittoo
Mode
Figure 8: Multi-Model Diffusion Process Visualization: We visualize the denoising process (100
steps). We can see that the flow starts from pure random noises and gradually converges to different modes: Pull and Push.
opened dataset outperforms the one trained on the mixed dataset, showing that multi-mode training data can confuse the regression models' training process.
Covine (T)
RMMSBL
MAG(])
Closed 1.10 open!
0.203
0,3129
0.9045
0.4436
0,4069
I.1833
0.2468
0.215
IL1847
Rundomly Open 10%
0.351
0.2720
0.9889
02218
0.4617
0.1246
0.1762
0.215
0.0919
Table 4: Multi Multi-Modality Analysis: We compare model with baselines doors. The cosine metric means the cosine similarity between the predicted flow and the ground truth flow, the higher the better. RMSE metric means the root mean square between prediction and ground truth, the lower the better. MAG metric means the flow magnitude error, the lower the better.
Fig visualizes the denoising process that produces different modes based on the same fully closed door, demonstrating our model's ability of preserving multi-modality for ambiguous examples.
D Occlusion Analysis
We analyze our model's performance under occlusions on different object categories. door exam- ple is included in the Fig of the main paper, and include occluded fridge and furniture examples in Fig 9. We and robust predictions regardless of the occlusions while FlowBot3D makes less consistent predic- tions when severely occluded.
from the visualizations that with history, FlowBotHD is able to produce stable
E Policy Analysis
We conduct further analysis on components of our roll-out policies, including the occasional neg- ative effect of the history filter, the efficiency switching the grasp point during policy exccution and the effect of limiting the number of steps during execution.
E.1 History Filter Analysis
Since train the model with only ground truth data, propose to use history filter to prevent incorrect predictions in the history from disturbing the history distribution; our history filter removes previous predictions that fail to further open the object. However, occasionally we find that including incorrect predictions in the history can lead to better performance; below we analyze when this might occur.
Consider the below dishwasher (which opens by pulling the upper side) example, exper- imented with 4 types of history input: No history (which can occur either at the beginning due
17

---

Occlusion (Refrigerator)
Flow Visualization at 90"
Visualization Perspective
Visualization Perspective 2
FlwwBorDD
under
Open 90"
(Severely Occluded)
FlowBotHD
Statle Performancs Ooduuion
Flowooa3
FlowBotHD
FlowBotHD
Occlusion (Furniture)
Flow Visualization at 90"
Perspective
How8or3D
Performancs
Drop Occlysion
Open 90* (Severely Ocoluded)
FlowBotHD
Performanes Under
RlowBor3D
Rawioo3D
Figure 9: Occlusion analysis: We open the object to different angles, make predictions and plot the Cosine Similarity metric (1) and RMSE metric (L) against the open ratio. We also include flow visualizations from different viewing perspective to intuitively show the quality comparison of the predictions. We can see from the visualization that FlowBotHD can make good and consistent predictions even under severe occlusions.
18

---

Object
History [Cosine Siilanty 90 degres)
History 2
(Cosina Simianty <30 degros)
Box Plot of RMSE Distribution w/ Different History
0.5
RMSE
0.3
0.2
History
GT History
History
History
History Type
Figure 10: History Filter Analysis: Above shows the visualization of three types of history, the box plot below visualizes the effect of different histories on the next step's prediction quality.
the history filter removing incorrect predictions). history from ground-truth trunh predictions, history from predictions with cosine similarity the ground-truth. > 90 degrees, history from predictions with cosine similarity to the ground-truth 30 degrees..
We can see from the box plot of RMSE distribution (the lower, the better) of 50 samples in Figure 10, that the history from the ground-true truth predictions leads to the best performance on average, and the history from predictions with cosine similarity the ground-truth > 90 degrees leads worse performance than history. However, interestingly, also that including history from predic- tions with cosine similarity to the ground truth 90 degrees sometimes leads to better performance than including history at all.
This experiment shows that including history from incorrect predictions can sometimes lead to better performance and sometimes worse depending on accuracy the previous predictions in the history. This implies that the history filter can sometimes inadvertently filter out helpful predictions in the history that might be close the ground truth. Nonetheless, results show that using the history filter average leads to better performance than not using it.
F.2 Switch Grasp Point Analysis
Switching grasp point costs extra time, therefore we should only allow necessary switches to happen. In our method, the stablility of grasp selection is achieved via the leverage change threshold 8l as proposed in Section 4.3. By adjusting this threshold, we control how frequently the policy switches the grasp point.
19

---

Switch Grasp Rate per Step (FlowBotHD w/ SGP)
1.0
0.0
Switch Grasp hate
0.6
0.4
0.2
0.0
10
15
20
25
Figure 11: Switch Grasp Rate per Step Plot: As seen in the plot, the switch grasp point rate gets lower among the first few steps, and increases slightly at the later stages.
Distribution of Switch Grasp Points (FlowBotHD w/ SGP)
0.5
0.4
Requency
0.3
0.2
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Switch Counts
Switch Grasp Count Histogram: we show the distribution of the number of switch grasps across the dataset. About 60% of objects only require one grasp action (the first step). For the other 40 %, most of them require only < 5 extra switches of the grasp point.
Figure 12:
We performed an additional analysis to calculate the number of grasp actions each object needs. By average, our method needs 1.76 times grasp actions including the initial grasp, so on average each object requires 0.76 switch-grasps. The results show that our method does not require a lot of switching and still leads improved performance over the baseline.
We made 2 supplementary plots to visualize the frequency of switching grasp points in our simula- tion experiments. Figure 11 shows the switch grasp point rate per step. We can see that the switch grasp point rate gets lower among the first few steps the consistency increases with multi-mon modality decreased. But increases slightly at the later stages possibly meaning that occlusions could hap- pen and destabilize the grasp point predictions. Figure shows the distribution of the number of switch grasps across the dataset. We can see that about 60% of objects only require one grasp action (the first step). And for the other 40%, most of them require only <5 extra switches of the grasp point.
20

---

E.3 Step Number Analysis
During roll-out with limited number of steps (in our case, 30 steps), failures happen either because of bad prediction quality that will never succeed, less efficient strategy grasp point that might eventually succeed given enough steps. To further analyze the time dependency of the success rate we measured, we made a plot of the opening success rate at each time step.
Saccuss hate par Sep Sotale 2071
Succuss hats per 2kg (Tlowddt S0Y
Fuccass Patt per Step Illonded Dut 907)
Figure 13: Success Rate over Timestep: We can see that both baseline with and without switch grasp policy (SGP) and FlowBotHD (ours) have nearly reached a plateau after 25 steps. We infer that the performance not much limited by the 30 step timesteps constraint.
We see in Figure 13 that both FlowBot3D (baseline) with and without switch grasp policy (SGP) and FlowBotHD (ours) have nearly reached plateau after 25 steps, indicating that further timesteps after 25 would likely have limited performance increase. In paper, allow each method 30 timesteps to open the object before we compute the success rate for our analysis. Therefore we infer that most of the remaining failures are the result of incorrect predictions and the performance not much limited by the 30 timestep constraint.
Real-world Equipment
F.1
Hardware
In the real-world demos, deploy our model on Franka Emika Panda Robot. We obtain point cloud input data from an Azure Kinect Depth Camera. The robot's end effector is a Schmalz Cobot Pump with a suction cup 3 em wide in diameter.
F.2 Workspace
We constructed our workspace (Fig 14) in a 1.3 by 1.3 by 1.1 m space. The Azure Kinect Camera was placed so that it pointed toward the center of the workspace at an angle that allowed the door to be seen clearly. To avoid collisions with the table when motion planning, we added a box representing the table top using Movelt's collision box construction tool.
F.3 Foreground Segmentation
To isolate the objects in the point cloud, we had to programmatically segment the table, background, and robot points.
To segment out the table and background, threshold the x, y. and values of the points to effectively crop all points outside of the box where the object would be. For example, all points with a value below 0.02 were removed, as that marks the top of the table.
The robot points are filtered out in real-time using a ROS package called Robot Body Filter that takes in the robot's URDF file. The filter assumes perfect calibration of the camera, which leads to some trailing points from the robot occasionally remaining in the scene. We filter out those points using outlier removal since they sparser than the object's 's points.
21

---

Figure 14: Real World Workspace: This picture depicts the ambiguous door, Franka robot, and Azure Kinect camera.
F.4 Robot Control Paradigm
The robot is controlled using position control by inputting the desired end-effector pose to Movelt, which would then motion plan and execute the path.
To make contact, the robot aligns itself with the chosen grasp point and estimated flow direction, then moves to a point 10 em in the flow direction away. Then, function called to start the pump, and the robot moves 2 em increments towards the contact point. Another function detects when vacuum has successfully been established, meaning the robot can progress opening the object.
For each step, the robot moves 5 cm in the direction of the respective predicted flow vector. After the robot attempts to execute the initial step, we use the switch grasp policy to determine it should attempt to remake contact. To remake contact, the robot moves back to its initial starting pose, then repeats the process described above. The robot can remake contact up times.
F5 Custom Multi-modal Door
We custom built multi-modal door out of plywood, with four pairs of hinges. This allows for it to be configured to open in all four of the possibilities of a standard door (forward to the right, forward to the left, backward to the right, and backward to the left). The door's 's current configuration is determined by two thin allen key-shaped metal rods, which connect a pair of hinges. Those hinges are then effectively in use and determine how the door opens. The door's frame 25 cm by 8.5 cm by 43.8 cm, with a 31.75 em by 36 em by 1.25 em stabilizing base. The door itself 16.83 cm by 1.75 em by 43.5 em, with nothing on its front back face differentiate which way it opens. To ensure that it fits within workspace, built it to be much smaller than normal door, then scaled the point cloud up by three before passing it into the model.
Given the custom made door, our model has the ability to make multi-modal predictions and open the door in all 4 different ways: pull left, pull right, push left and push right. also has the ability to actively switch grasp point when the initial prompts fail align with the current configuration. In all the experiments, the action of 'push' executed as pulling from the back.
22

---

General Objects
Closed
Open
Closed
Open
Chest
Jar
Cookie Jar
Laptop
Figure 15: Demonstration of FlowBotHD opening general articulated objects in real-world.
23


Reviews:
[{"review_id": "7Y16ZHikpP", "reviewer": "Reviewer_LDDR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a new method for manipulating articulated objects with inherent ambiguities, such as doors with uncertain opening mechanisms (push, pull, slide) and directions. These ambiguities are compounded by occlusions that obscure the object's shape.", "summary_of_recommendation": "Weak accept.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 1, "quality_of_limitations": 2, "questions_for_rebuttal": "Limitations:\n\n1.  while the proposed method shows strong theoretical and simulated results, there is a lack of hardware contribution, and the real-world experiments are somewhat limited in scope.  This absence means the full practical impact and reliability of the method in diverse, real-world settings remain uncertain.\n\n2. The experiments do not include direct robot testing, which could provide deeper insights into the method’s performance in practical robotic applications. Incorporating such robotic experiments could offer valuable validation and could be more appropriate for discussions in other conferences focused on practical robotics and implementation challenges.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zHAIgGgM9t", "reviewer": "Reviewer_niS8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Interacting with articulated objects in real-world environments presents challenges like ambiguous manipulation directions, multiple optimal actions, and occlusions. To address this, the authors introduce FlowBotHD, a history-aware diffusion network that models the multi-modal distribution of object articulation. By incorporating historical context and consistency checks, FlowBotHD disambiguates actions and makes stable predictions even under occlusions. The authors demonstrate their approach's effectiveness by conducting experiments on multiple simulated objects and a real-world door-opening task.", "summary_of_recommendation": "The authors' proposed method shows potential for effectively handling ambiguities and occlusions when interacting with articulated objects. However, to fully substantiate the claims and demonstrate the broader applicability of the approach, it is essential to provide more comprehensive evaluation with real-world examples.  I am inclined to recommend this paper for acceptance, contingent upon the authors addressing these concerns through further experimentation and analysis.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "- RGB/RGBD Baselines: Given that RGB features can aid disambiguation in varying lighting conditions, did you consider comparing your method to RGB-only or RGBD-only baselines, such as those in [2, 3, 4]? This could further highlight the value of your approach.\n- In-Depth Analysis of Results: In Table 1, the history filter appears to negatively impact performance for some revolute joint objects (e.g., microwave (col. 6), locker (col. 9), washing machine (col. 14)). Could you provide a more detailed analysis in the experimental section, exploring potential reasons for this and discussing any mitigating strategies?\n- History Window Length: Why was a history window length of 1 chosen? Would a longer history window potentially improve disambiguation? Did you conduct any ablation studies to determine the optimal window length?\n- Frequent Grasp Switching: How does your approach mitigate the issue of frequent grasp switching between time steps? Are there mechanisms in place to ensure stable grasp selection?\n- What value of $\\delta m$ was used in your experiments? \n\n\n*References*:\n- [2] Mo, K., Guibas, L. J., Mukadam, M., Gupta, A., & Tulsiani, S. (2021). Where2act: From pixels to actions for articulated 3d objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6813-6823).\n- [3] Jiang, Z., Hsu, C. C., & Zhu, Y. (2022). Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5616-5626).\n- [4] Heppert, N., Irshad, M. Z., Zakharov, S., Liu, K., Ambrus, R. A., Bohg, J., ... & Kollar, T. (2023). Carto: Category and joint agnostic reconstruction of articulated objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21201-21210).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WGIkAsRPvU", "reviewer": "Reviewer_aMTY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an extension to FlowBot3D, a method for predicting the movement flow of articulated objects. The extension is aimed at solving multi-modality and occlusion ambiguities with regards to object movements, and uses diffusion models and history encoding to solve both ambiguities respectively.", "summary_of_recommendation": "This paper proposes a valid solution to challenges in a previous approach. While it has some drawbacks, in particular being slightly incremental, I find the contributions good enough to merit acceptance, especially if the other concerns are addressed.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Line 189 states “it might be unclear which direction these objects open”: is there a quantification on how many of the objects in the dataset are actually ambiguous in that sense? While it is clear that doors, for example, are more ambiguous than laptops, even inside the door class I assume some have visual tells such as handles. This quantification would be interesting both in advance (when surveying the dataset), and when learned by the model, i.e., how often does the model predict objects to have multi-modal flows. \n\n2. Is a rollout limited in the amount of time steps? One might expect that even less efficient grasp points to succeed given enough time, as long as they are using the correct flow direction. Do failures mostly happen because of time constraints or because of choosing the wrong grasp points?\n\n3. In the results table, it seems like there are cases where the original FlowBot3D outperforms FlowBotHD. To the best of my understanding, it seems like FlowBotHD extends the previous approach, using a more expressive distribution. Especially with the consistency check heuristic, it is hard to understand why there would be cases where the proposed approach is outperformed. Could the authors share examples for some failure cases where FlowBot3D outperforms FlowBotHD? Is there any insight on when and why this happens?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Clearly presented with extensive experimental validation.
- History aware diffusion networks that offers potential for handling uncertain situations in object manipulation like occlusions.

Weaknesses:
- Lack of real world results and direct evaluation on robots
- Scope of contribution is limited: provides a solution to limitations of prior methods

Reviewer LDDR focused on the lack of real-world experiments and direct robot testing. In response, the authors provided details about their real-world experiments, including the hardware, workspace setup, foreground segmentation process, and robot control paradigm. They also discussed the custom-built multi-modal door used in their experiments.    

Reviewer niS8 also pointed out the limited real-world evaluation and suggested including more real-world examples. Additionally, they asked about RGB/RGBD baselines, in-depth analysis of results, history window length, frequent grasp switching, and the value of om. The authors addressed these points by providing further analysis of their results, explaining their choices in the experimental setup, and discussing the challenges of incorporating RGB/RGBD baselines. They also included real-world experiments with a custom-built multi-modal door.    

Reviewer aMTY questioned the scope of the contributions, the limited real-world experiments, and specific aspects of the method, such as the quantification of ambiguous objects, rollout time steps, and cases where FlowBot3D outperforms FlowBotHD. The authors responded by providing a more detailed analysis of ambiguities in the dataset, discussing the time dependency of their method, and explaining the reasons behind the occasional outperformance of FlowBot3D. They also extended their real-world experiments to address the reviewer's concerns.
Decision: Accept