Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
Authors: Eugenio Chisari
Nick Heppert
Max Argus
Tim Welschehold
Thomas Brox
Abhinav Valada
Venue: CoRL 2024 (2024)
Keywords: Imitation Learning
Manipulation
Conditional Flow Matching
TL;DR: Imitation learning via conditional flow matching, and an investigation of the rotation vector space.

Abstract: Learning from expert demonstrations is a popular approach to train
robotic manipulation policies from limited data. However, imitation learning
algorithms require a number of design choices ranging from the input modality,
training objective, and 6-DoF end-effector pose representation. Diffusion-based
methods have gained popularity as they allow to predict long horizon trajectories
and handle multimodal action distributions. Recently, Conditional Flow Matching
(CFM) (or Rectified Flow) has been proposed as a more flexible generalization
of diffusion models. In this paper we investigate the application of CFM in the
context of robotic policy learning, and specifically study the interplay with the
other design choices required to build an imitation learning algorithm. We show
that CFM gives the best performance when combined with point cloud input
observations. Additionally, we study the feasibility of a CFM formulation on
the SO(3) manifold and evaluate its suitability with a simplified example. We
perform extensive experiments on RLBench which demonstrate that our proposed
PointFlowMatch approach achieves a state-of-the-art average success rate of 67.8%
over eight tasks, double the performance of the next best method.

Learning Robotic Manipulation Policies from Point Clouds with Conditional Flow Matching
Eugenio Chisari, Nick Heppert, Max Argus, Tim Welschehold, Thomas Brox, Abhinav Valada Department of Computer Science, University of Freiburg, Germany http://pointflommateo cs. uni-freiburg.de
Abstract: Learning from expert demonstrations is promising approach for training robotic manipulation policies from limited data. However, imitation learning algorithms require a number of design choices ranging from the input modality, training objective, and 6-DoF end-effector pose representation. Diffusion- based methods have gained popularity as they enable predicting long-horizon trajectories and handle multimodal action distributions. Recently, Conditional Flow Matching (CFM) (or Rectified Flow) has been proposed as more flexible generalization of diffusion models. In this paper, we investigate the application of CFM in the context of robotic policy learning and specifically study the interplay with the other design choices required to build an imitation learning algorithm. We show that CFM gives the best performance when combined with point cloud input observations. Additionally, we study the feasibility of a CFM formulation on the SO(3) manifold and evaluate its suitability with simplified example. We perform extensive experiments on RLBench which demonstrate that our proposed PointFloWMatch approach achieves state-of-the-ar art average success rate of 67.8% over eight tasks, double the performance of the next best method.
Keywords: Imitation Learning, Manipulation, Conditional Flow Matching
Introduction
Imitation learning (II.) is the widely studied problem of training policies from a given set of expert demonstrations [1, 2. recent years, imitation learning has gained popularity in the robot learning community, as leveraging the prior knowledge of the expert demonstrator allows training complex behaviors with small amounts of data. The primary approach to learning an IL policy Behavior Cloning (BC) [4, 5], where deterministic mapping from state actions is learned in supervised manner from the available data. While BC has achieved significant success for different tasks, robot policy learning remains challenging problem, given the requirement of high precision, the sequential correlation (i.e. not i.i.d.) of data, and the multimodality of the action distribution, which all add complexity compared to other supervised learning problems.
Recently, generative models have been demonstrated be effective tackling some of these challenges. Most prominently, Diffusion Policy [6] adopts score-matching formulation of generative diffusion models. The forward diffusion process starts with expert robot trajectories and gradually adds Gaussian noise until the signal approximates pure noise. The denoising process reverts these steps and it is used as a training signal for the model. This is a stochastic process that results in Gaussian conditional probability paths mapping Gaussian noise to data, with specific choices mean and standard deviation [7, 8]. The authors show that diffusion policies can handle multimodal action distributions and to directly predict long sequences of actions.
Diffusion policies sparked strong interest from the community and led to many exciting results in robotic manipulation applications [9, 10). Nevertheless, relying on diffusion models also has some
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

dastaaant diffusion models need to explicitly define an iterative forward diffusion process, which inherently defines the final noisy distribution and the probability path the model learns to denoise along. In the typical case of Gaussians, a closed-fomm solution is available, enabling us to directly generate fully noised and intermediate, partially noised, samples. In turn, in the cases where no closed-form solution for the forward diffusion process available, training time will increase [11]. To overcome these limitations, Conditional Flow Matching (CFM) has been proposed as efficient generalization of diffusion models [12, 13, 11]. CFM is simulation- approach, i.e. it starts directly from noise without requiring forward diffusion process. It is, therefore, more general, since it allows transporting any starting probability distribution into the data distribution.
Inspired by recent flow-based generative models, we propose PointFlow novel imitation learning algorithm for robotic manipulation. SointFloMMatel uses point cloud observations that prove to be more effective than images [9, 10] and builds upon CFM formulation learn the distribution of the expert robot trajectories. As CFM is able to model arbitrary probability paths, it also allows formulating the regression on the R x SO(3) manifold. We evaluate the performance of our proposed method on the popular RLBench benchmark [14] and compare it against strong recent baselines with both image and point cloud observations: Diffusion Policy [6], 3D Diffusion
Policy [9], CifinedDiffuser [10], and AdaFlow [15]. In summary, this paper makes the following main contributions:
PointFlowMatch, novel method based on the recent conditional flow matching framework to train robotic imitation learning policies from point clouds.
An investigation of two different approaches to handle 3D rotations in the context of CFM for policy learning.
Extensive evaluations against recent state-of-the-art baselines and an ablation study of our main design choices.
We make the code, models, and videos publicly available at http://poinnflommacccooo uni uni-freiburg de.
2 Related Work
Generative models for imitation learning in robotic manipulation have attracted significant interest. In the following, we highlight recent advances in this topic.
Generative Models for Policy Learning: One interesting class of generative models explored for robotic policies is the Action Chunking Transformer (ACT) [16]. It adopts CVAE [17] structure, where the encoder predicts latent variable and the decoder outputs the action sequence. different approach taken by Florence et [18], who propose Implicit Behavior Cloning (IBC). IBC defines the distribution over action inherently handles multimodal distributions but is unstable train due to the need for negative sampling when computing the loss. An early attempt to apply diffusion models to robotic applications is presented in Diffuser [20]. where the authors investigate the use of diffusion models from the perspective of planning. Refining this idea, Chi et al. [6] propose the seminal work Diffusion Policy (DP). DP inherits the advantages of IBC (handling multimodal action distributions) while avoiding its downsides. Instead of learning an energy function, DP learns the gradient of the action distribution, which exhibits better training stability.
Energy-Based Model (EBM) [19]. This approach
Diffusion Policy from Point Clouds: The success of DP inspired numerous follow-up works. A key characteristic common to many of them is the choice of point cloud observation representations, instead of RGB images, as used in the original DP method. ACT3D [21] is transformer model that predicts the next best waypoint, instead a full trajectory. 3D-DP [9] learns to predict full trajectories, and it also adopts a modified MLP as an efficient point cloud backbone. ChainedDiffuser [10] proposes hierarchical model where high-level key points are predicted via ACT3D and low-level trajectories connecting these points are generated via the standard trajectory diffusion. Another hierarchical model is HDP [22], which builds PerAct [23] as a waypoint predictor and combines both joint and end-effector pose predictions. While hicrarchical models achieve strong results,

---

specifically investigate the performance of the underlying trajectory diffusion models and therefore focus our analysis on non-hierarchical single-task policies.
Conditional Flow Matching: Conditional Flow Matching (CFM), also called Rectified Flow (RF), is a recent generative model training technique that connects noise to data via straight paths [12, 13, 11]. The more established diffusion models can be seen special case of the CFM framework, which more general and conceptually simpler. Very recently, CFM was demonstrated be superior to diffusion models in various applications, ranging from image synthesis [24] to protein backbone generation [25]. In the context of robotics, AdaFlow [15] is an image- baaage-based policy trained with CFM, which also includes a variance estimation network to predict the variance of the current state and adjust the number of integration steps dynamically in order to reduce inference time. Concurrently to our work, Braun et al. [26] investigate the application CFM to Riemannian manifolds, but do not provide direct comparison with the standard Euclidean approach, while Rouxel et [27] show the application of CFM for shared autonomy teleoperation of humanoid robot.
3 Technical Approach
We consider an imitation learning problem: given a dataset of n expert demonstrations D = ( "' the goal is to train a policy т S - A that maps the observed states s the optimal actions a. Given the widespread accessibility of depth cameras, we assume that both RGB and depth observations available the policy. In the following sections, we detail our choices of observation and action spaces, training formulation, and model architecture.
3.1 Observation and Action Spaces
It has been demonstrated that, for robot policy learning, point cloud observations yield better results compared to RGB images [9]. These findings are supported by the fact that most recent state-of- the-art approaches [21, 22, 10] building upon DP adopt point clouds as their visual observation representations. Point cloud observations are effective as they directly encode the three-dimensional structure of a scene, and clearly separate geometric and semantic features, which are instead blended together in raw RGB images. This separation is especially helpful the low data regimes common in robot policy learning. Therefore, in our approach PointFlow we also adopt point clouds as the visual observation representation.
The input our policy consists of the Tobs last observations, which include both the robot state and processed point cloud, where Tots is tuning hyperparameter. Point clouds are processed in the following manner. For each available camera, we use the intrinsics and extrinsics to project the observed depth values in a common 3D space, and we merge all observations into a single point cloud. Last, the final point cloud computed by applying voxel-downsampling to attain uniform density, and by cropping all points outside the relevant workspace around the robot.
The action space of our model is 10-dimensional, consisting of end-effector position (3-dimensional), end-effector orientation (6-dimensional), and gripper open/close action (1 (1-dimensional), We choose absolute position control following Chi et al. [6]. which showed it to be more suited to action-sequence prediction compared to velocity control. We represent the gripper orientation as the 6D vector formed by truncating and flattening the relative rotation matrix, as proposed by Zhou et al. [28]. At test time, we employ a closed-loop receding horizon control strategy, i.e. we predict Tpred horizon steps into the future, command only the first predicted step to the robot, and then repeat the prediction.
3.2 Conditional Flow Matching for Policy Learning
In CFM, for arbitrary samples z, the prediction problem is formulated as an ordinary differential equation (ODE) of the form
d
dt zt) = v(z(t),t).
z(t ( Po-
where Po is the arbitrary start distribution, commonly chosen random Gaussian noise with zero mean and unit variance, and d [0, + Rd is the velocity vector field that induces the

---

probability path from the start to the target distribution. The goal is to learn the appropriate vector field that transports the chosen probability distribution to the data distribution in unit time. The vector field by can be learned by supervision via the loss function
[[" [10g((21,20)) - vo (z 1)2 dt
min E Exo~po
(2)
where Ugr is the choice of ground truth vector field we regress against. While this choice is not unique, CFM adopts the time-independent, straight line vector field l'gt = 21 20. which has the desirable property of producing straight probability paths [12, 11]. In practice, the vector field vy is parametrized by a neural network and the loss in Eq. (2) can be minimized via mini-batch gradient descent. In the context of policy learning, the target samples 2 are the robot trajectories from the expert demonstration dataset. Once the model is trained, we can sample 2p from the start distribution Po and numerically integrate the ODE from Eq. (1) up to time = I to generate new robot trajectories that follow our expert data distribution. An overview of the implementation of CFM for policy learning is shown in Algorithm
Algorithm CFM for Policy Learning (Euclidean)
Algorithm CFM Policy Learning (SO(3)
1 def brain_step(batehe
def train_step stt.p(ttepb)))
2 obs, target_tra, = batch
obs, berget_traj batch cond obs (ebc
cond (ob_eecccer(bbs) rand() # uniform
rand # uniform
5 randn () normal
z0 randn_S03() # IGSD(3)
6 z1 target_traj
z1 target_traj
7 target_ves vel z1 20
target_ves (og(In(())) z1)
20 + t * target_vel pred_vel vel_vel= (zt, t, cond)
z0 @ Exp(t * sarget_vel)
pred_vel net (2t, t, cond)
loss mse mee (pred_vel, target_vel) return loss
loss = mse vel, target_vell vel) return loss
def Inference_step
13 def inference_step(obb)
cond obs_ ibs_der (obs)
14
cond obs_ abs_enc (obs)
15 z randn() normal
15
randn_SOSS S03( # (3)D(3)
for & in k_atepa:
for k in k_steps:
t = k_steps
k k_steps
k_ k_steps
18
k_steps
pred_vel (z, t, cond)
pred_vel net t, cond)
20
z = 2 peld_vel dt
20
@ Exp (pred_eee dt)
return
3.3 Conditional Flow Matching for Data in SO(3)
Our robot state representation includes the end-effector 3D orientation. Since 3D orientations live on the SO(3) group's 's manifold, we need to pay particular attention to how we handle the orientation's ss prediction. There are two strategies we can employ to tackle this challenge.
Euclidean Formulation: The first strategy to follow the standard CFM formulation in Euclidean space for both training and inference and project the resulting vector to the SO(3) group's manifold only the end of the inference process, output a valid rotation matrix. This projection can be implemented Grahm-Schmidt process in the case of 61) representation of rotations the SVD-based orthogonal Procrustes in the case of 9D representation of rotations [29].
SO(3) Formulation: The second strategy is to define the starting random distribution and target vector field directly on the 3D rotation manifold so that the entire resulting probability path lives on (3))). In particular, the random initial state is sampled as x(t = 0) ~ IGsota)> where IGso(3) the isotropic Gaussian distribution on SO(3). Since the rotation group smooth manifold, its velocity a specific point is expressed in the tangent space, denoted as so(3). The mapping from the tangent vector space to the manifold is defined by the exponential map Exp(o), while the inverse mapping from the manifold to the tangent space is defined by the logarithm map Log(o). For

---

Robot State
Point Cloud
PointNet Backbone
Lot
T times
Noisy Traj.
CFM 1-D UNet
Clean Traj.
Figure 1: Diffusion and CFM repeatedly applied to noisy trajectory, thereby iteratively yielding clean trajectory that be executed on the robot. The generative models also take input encoded observations.
a detailed overview of the topic, we refer the readers to [30]. For our CFM setting, the natural choice of target vector field is given by the tangent on the manifold that forms the geodesic interpolation between the start and target samples
Ze 20 Exp (4 (21, z0)), where (21, = Log (zo-²z1)
At test time, we can integrate the predicted velocity vectors in tangent space along the manifold geodesic to solve the ODE in Eq. (1) and generate end-effector orientations that directly lie in 50(3). The implementation the (O(3) formulation CFM for policy learning is shown in Algorithm While the latter approach has been studied in recent works such as Riemannian Flow Matching [31] and FoldFlow [25]. comparison of the strategies in the context of policy learning still lacking. In our evaluation (Appendix A.I), we investigate both strategies and provide the counter-intuitive insights that the first, simpler approach might be preferable for policy learning in robotics.
3.4 Model Architecture and Training Setup
To encode the point cloud observation into low-dimensional features, we use modified version of PointNet [32]. Compared to the original architecture, we remove both T-nets used for input and feature transformation, see [33]. These are used to improve rotation and translation invariance, which desirable for classification and segmentation but not for robotics tasks. The model used to de-noise trajectories is conditional ID U-Net, similar to [6]. This model takes input the random noise samples and the condition information and returns the sample velocity in the case of CFM and the noise in the case of diffusion. For an overview, see Fig. 1. The condition information consists of the concatenation of the robot's 's proprioceptive state (i.e. end-effector poses) as well as the encoded visual observation, and it added to the bottleneck state of the U-Net model.
We train our model with the AdamW optimizer and learning rate of -5 and weight decay le -6 We apply cosine annealing of the learning rate and linear warmup of 5000 steps. We use batch size of 128 and apply EMA on the weights. The input point clouds are downsampled 4096 points and the target robot trajectories are subsampled with factor of 3.
4 Experimental Evaluation
We evaluate our proposed PointFlowMMatch RLBench [14], popular robot learning benchmark. Each environment consists of the 7-DoF Franka Emika robot arm to execute actions and set of five cameras around the workspace. We use RLBench's 's automatic expert demonstration execution to collect 100 trajectories for each task, which consists of all robot states and camera observations. For evaluation, we rollout the policy on each task for 100 episodes and report the average task success rate across all episodes. Following RLBench's test protocol, for each episode the task setup randomized. We further perform three evaluation runs with three random seeds and average over all runs. As result, all models evaluated on the same set of 300 random episodes. Following ChainedDif. fuser [10], consider set of eight tasks that require continuous interaction with the environment,
5

---

unplug
close
open box
open fridge
frame
open
books
shoes
charger
door
hanger
oven
shelf
box
Figure 2: Example images of the eight RLBench tasks.
Table Performance comparison of PointFlowMMatch with different baseline methods on RLBench set of tasks. We report the success rate (SR) (1) as well as the delta our method. On average, our method performs considerably better. For OI OI--ChaineDDffuuse we report the results from the original publication, which do not include the standard deviation.
Method
unplug charger
close door
open box
open fridge
open oven
Mean SR
hanger
shelf
box
SR
Dif. Policy [6]
38 38.02336
19.3422
75.744.2
0.04
16.0
0.040.
0.346e
0.040..
18. 18.7433
49.1
AdaFlow [15]
46.3#16
13.3+33.
77.3
2.0
12,7=a.
0.0e0.0
0.3=0.6
0.0
19.0
3D-DP
33.3 33.
26.0
98.g
4.3
12.3225
0.3
3,7406
0.040.
28.5422
39.3
OL-ChDif [10)
65.0
21.0*NA
46.0*NA
37.0
43.0
16,0
40.0
9.0+NA
34.6*NA
33.2
MaichlowMatch
83.6 83.6
68.3
99.4 19.4
31.9229
38.6227
75.9
68.8 68.8
7..0
67.8
where point-to-point motion planning approaches typically struggle: implug-chargess close_door, open box, open fridge, take frame off hanger, open oven, put books on bookshelf, and oo _box (some names abbreviated figures and tables).
4.1 Benchmarking Results
In this section, compare the performance of PointFlow with recent state-of-the-art methods for imitation learning based generative models. For fair comparison, keep the number of denosining and/or integration steps k constant to 50 for all methods.
For image-based methods, we compare against the original Diffiusion Policy [6] as well as AdaFlow [15] which also uses the CFM learning objective. For the scope of this analysis, we do not implement the adaptive step feature AdaFlow, since we mainly focus on the quality of prediction and not on inference speed. For both methods, we follow the original Diffusiion Policy [6] approach and fuse images from all five viewpoints after the backbone feature extraction. As point cloud-based baselines, specifically investigate the performance of hon-hierarc single-task policies. For this reason, consider the Open loop trajectory diffusion baseline presented in ChainedDiffuser [10], which is the underlying diffusion model without the higher level waypoint policy forming the hierarchy.
consider Chained Diffuser [10] well as 3D-DP [9]. In this work,
We report the results in Tab. This evaluation highlights the strong performance PointFlowMatch. which achieves a success rate percentage points higher than the second-best model. This result demonstrates that the combination of our choices of observation type, encoder architecture, and training objective leads to highly effective imitation learning algorithm. In line with the findings of our ablation study in Sec. 4.2, we also highlight how the point cloud based policies outperform the image-based policies.
4.2
Ablation Study
In this section, we aim to compare the key design choices of our method and answer the following research questions:
Are point clouds a more effective observation representation compared to raw RGB images?
Is Conditional Flow Matching a more effective training objective compared the more established diffusion models?
Is the S0(3) formulation of CFM more effective at learning to match the data distribution on the
3D rotation manifold, compared to CFM in Euclidean space followed by projection SO(3)?

---

Table 2: Ablation of observation type (images vs point clouds), vector field formulation (R" (3))), and training objective (DDIM vs CFM) for our method, evaluated on RLBench tasks showing the mean success rate (SR) (1) as well as the delta to the proposed model. All models are evaluated with & 50 inference steps. Given the similar performance of the three point cloud-based baselines, we report the average across training seeds, each tested on the same 3 evaluation seeds, for total of 9 evaluation runs.
Obs. type
Rot. formal.
Training objective
unplug charger
open box
open fridge
open ovan
books sholf
Mean SR
Delta SR
door
hanger
box
Img.
36
CFM
513353
2
10.0
21.3=93
11.0420
19.7547
26.3549
10.1
-27.7
Pod.
3
DDIM
84
74.9
99.31
30.8 30.8
39,1
75, 75.2437
67 67.8458
71.9 71.9i8a
68.0
0.2
Pod.
50(3
CTM
82,3
68.8
99,4586
34.0
38.4
74.9*20
68.026.
73.2#12
67.4*44
0.4
Ped.
24
83.6 63.6+55
68.3*5.
99.4 19.7
31.9
38.6=27
75.9
68,8
76,0
67.8#4.1
80
40
Success Rate (-)
60
30
Inference FPS [Hz]
k-steps
2
4
8
16
50
DDIM
Inference
DDIM
30
50
89
149
190
362
40
CFM
20
Time [ms]
CFM
29
49
85
149
189
354
Succes
DDIM
19.4
13.5
66.3
66.9
66.3
68.0
20-
10
Rate
CFM
36.8
61.9
66.0
68.8
69.3
67.8
2
8
16
& Inference Steps
Figure 3: Comparison of CFM and DDIM for varying values of the number of inference steps k. We compare the inference time (L) measured in [ms] as well as the inference FPS (1) in [Hz] against overall success rate (1) for both formulations.
How does the number of inference steps k affect performance?
In order to answer these questions, we carry out an ablation study where we vary each of the aforementioned design choices and investigate their impact on overall performance. In particular, investigate 1) using images as visual observation, adopting the same ResNet backbone in the original DP [6]. 2) using the Denoising Diffusion Implicit Model (DDIM) [34] instead of the CFM training objective, 3) adopting the SO(3) formulation of CFM for the end-effector orientations to learn probability paths directly on the 3D rotations manifold, 4) testing both the DDIM and CFM-based models with varying inference steps k. ranging from 1 to 16. We report the results in Tab. 2 and Fig. 3. From the numerical evaluation, we observe that the largest impact on performance determined by the choice of observation type. The policy using raw RGB inputs 27.7 percentage points worse than the proposed model, highlighting the strengths of point cloud observations.
First, we consider the two different formulations of the CFM framework that be used to learn about the distribution of 3D rotations: euclidean and SO(3). The outcome for this ablation is very similar, with the score averaged across all tasks showing that the SO(3) formulation achieves a lower success rate, by 0.4 percentage points. This result is surprising and at first counterintuitive, as we expected that learning probability paths directly on the target manifold would achieve higher performance. Given the intriguing result of this ablation, we add further analysis of this comparison in Appendix A.
Regarding the choice of training objective, the results in Tab. 2 show that for i 50 inference steps, the CFM objective does not improve over the popular DDIM framework, which achieves a similar success rate, within the standard deviation. Nevertheless, when considering the two objectives across a range of value of & inference steps in Fig. 3, we find that CFM demonstrates stronger performance, especially in the low l regime. This is in line with the results obtained for the image synthesis domain, where Stable Diffusion 3 [24] shows that CFM achieves similar scores to the diffusion objectives, but performs better with fewer inference steps, required for faster inference. This characteristic, paired with the higher flexibility, generality, and ease of implementation, in our opinion, makes the CFM formulation advantageous compared to established diffusion objectives.
7

---

Figure 4: We demonstrate PointFlow Match on real robotic setup. We evaluate on two tasks: open box and sponge on plate.
4.3 Real Robot Experiments
To demonstrate the application of PointFlowMMatch real robotic platform, we perform set of experiments with a Franka Emika Panda manipulator. We evaluate two table-top tasks: open box and sponge mounted on the robot end-effector and the second mounted externally. Similarly to the simulation setup, we merge the point clouds obtained from both cameras and process it via voxel-subsampling To prevent overfitting and improve the model's robustness to the noisy point clouds, we apply random transformation and random noise-jitter augmentations during training. PointFlowMatch achieved success rate of 72% for open box and 48% for sponge on plate. The main failure mode we observed in this experiment consisted of the robot reaching correctly for the object to grasp. but missing it by small margin. A selection of policy rollouts for both tasks is shown in the accompanying video.
plate, shown in Fig. 4. We use two Realsense RGB-D cameras, the first
5 Conclusion
We present PointFlowMatcch a novel method for imitation learning from a fixed of demonstration examples. Our method combines the recently developed conditional flow matching framework with point cloud observation encoding. In developing most important components. Additionally, we study formulation of CFM on the SO(3) manifold. Our method achieves state-of-the-arr results for single-task non-hierarchical policy learning, with an average success rate 68.6 %, double that of the closest baseline method.
method, performed an ablation study of the
Limitations:
There a few limitations as BC that directly predict an action, diffusion and CFM models both have longer inference times, given their iterative nature. In addition to this, as usual in the fixed-data imitation learning setting, CFM cannot extrapolate out of distribution and thus, only learns motion correction behavior when included in the demonstration set. Our point cloud observation space merges multiple views, this requires depth cameras with known intrinsic and extrinsic calibration. We also trained single-task policies and did not investigate generalization capabilities. Finally, implementation real robot requires manually collecting expert demonstrations.
proposed method. Compared approaches such

---

Acknowledgments
This work was funded by the Carl Zeiss Foundation with the ReScal.e project and the German Research Foundation (DFG): 417962828.
References
[1] E. Chisari, T. Welschehold, Boedecker, W. Burgard, and A. Valada. Correct me if am wrong: Interactive learning for robotic manipulation. IEEE Robotics and Automation Letters, 7(2): 3695-3702, 2022.
[2] C. Celemin, R. Pérez-Dattari, E. Chisari, G. Franzese, L. de Souza Rosa, R. Prakash, Z. Ajanović, M. Ferraz, A. Valada, and J. Kober. Interactive imitation learning in robotics: A survey, Foundations and Trends0 in Robotics, 10(1-2)11-17, 2022.
[3] N. Heppert, M. Argus, T. Welschehold, T. Brox, and A. Valada. Ditto: Demonstration imitation by trajectory transformation. arXiv preprint arXiv:2403. 15200, 2024.
[4] Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[5] J. O. von Hartz, E. Chisari, T. Welschehold, W. Burgard, J. Boedecker, and A. Valada. The treachery of images: Bayesian scene keypoints for deep policy learning in robotic manipulation. IEEE Robotics and Automation Letters, 2023.
[6] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
[7] Ho, Jain, and Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[8] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In Int. Conf. on Learning Representations, 2021.
[9] Y. Ze, G. Zhang, K. Zhang, C. Hu, M. Wang, and H. Xu. 3D diffusion policy: Generalizable visuomotor policy learning via simple 3D representations. In Robotics: Science and Systems, 2024.
[10] Z. Xian, N. Gkanatsios, T. Gervet, T.-.. Ke, and K. Fragkiadaki. ChainedDiffuser: Unifying trajectory diffusion and keypose prediction for robotic manipulation. In Conf. on Robot Learning, 2023.
[11] Y. Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In Int. Conf. on Learning Representations, 2023.
[12] X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. Int. Conf. on Learning Representations, 2023.
[13] M. Albergo and E. Handen-Eijnden Building normalizing flows with stochastic interpolants. In Int. Conf. on Learning Representations, 2023.
[14] James, Z. Ma, D. Rovick Arrojo, and A. Davison. Rlbench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 2020.
[15] X. Hu, B. Liu, X. Liu, and Q. Liu. Adaflow: Imitation learning with variance-adaptive flow- based policies. arXiv preprint arXiv:2402.04229 2024.
[16] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. In Robotics: Science and Systems, 2023.

---

[17] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems, 2015.
[18] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson. Implicit behavioral cloning. In Conf. on Robot Learning, 2022.
[19] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, and F. Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[20] M. Janner, Y. Du, J. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. In Int. Conf. on Machine Learning, 2022.
[21] T. Gervet, Z. Xian, N. Gkanatsios, and K. Fragkiadaki. Act3d: 3D feature field transformers for multi-task robotic manipulation. Conf. on Robot Learning, 2023.
[22] X. Ma, S. Patidar, I. Haughton, and S. James. Hierarchical diffusion policy for kinematics- aware multi-task robotic manipulation. In Proc. of the IEEE Conf. on Computer Vision and Puttern Recognition, 2024.
[23] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conf. on Robot Learning. 2022.
[24] Esser, S. Kulal, Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Int. Conf. on Machine Learning, 2024.
[25] A. J. Bose, T. Akhound-Sadegh, G. Huguet, K. Fatras, J. Rector-Brooks, C.-H. Liu, A. C. Nica, M. Korablyov, M. Bronstein, and A. Tong. Sc(3)-stochastic flow matching for protein backbone generation. In Int. Conf. on Learning Representations, 2024.
[26] M. Braun, N. Jaquier, L. Rozo, and T. Asfour. Riemannian flow matching policy for robot motion learning. In IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 0-0, 2024.
[27] Q. Rouxel, A. Ferrari, S. Ivaldi, and J.-B. Mouret, Flow matching imitation learning for multi-support manipulation. arXiv preprint 2rXiv:2407.12388 2024.
[28] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2019.
[29] R. Brégier. Deep regression on manifolds: a 3D rotation case study. In International Conference on 3D Vision, 2021.
[30] J. Sola, J. Deray, and D. Atchuthan. A micro lie theory for state estimation in robotics. arXiv preprint arXiv: 1812.01537, 2018
[31] R. T. Q. Chen and Y. Lipman. Flow matching on general geometries. In Int. Conf. on Learning Representations, 2024.
[32] R. Q. Charles, H. Su, M. Kaichun, and L. J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2017.
[33] C. Qi, H. Su, K. Mo, J. Guibas. Pointnet: Deep learning on point sets for 3D classification and segmentation. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2016.
[34] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. In Inf. Conf. Learning Representations, 2020.
[35] A. R. Geist, J. Frey, M. Zobro, A. Levina, and G. Martius. Learning with 3D rotations, hitchhiker's guide to SO(3). Int. Conf. on Machine Learning, 2024.
10

---

A Additional Experiments
A.1 Simplified 50(2) Expe)-Exper:m No Free Lunch
0.5
0.5
-0.5
-05
02"
-100-
-1.0-
0.
-10
-10
lichts of Euclidean Euclidean Formulation using Ini- Samples Deaws from the Normall Disributionn
Eudlidean Formolation using Ini-
5012) Formulation Initial Samples Drawn Uniformly from SO(2).
(porple) and SO(2) (blue) formula- nons.
Samples Drawn Uniformll from 80(2).
Figure 5: Simplified Example. The left figure shows the edge case when random samples close to the opposite pole of the target sample. Here the SO(3) formulation presents a discontinuity which makes learning more difficult. In the three right figures, we visualize the mean error during inference across different sampling locations for our different formulations. We mark the target with a red cross. One observes that for the Euclidean formulation the error is lower for initial sample points along the axis orthogonal to the target. This is expected as values sampled along the line are naturally mapped the target when normalized. On the other side in the last figure, one observes higher errors close the pole. Additionally, training data bias is visible the error higher on one side of the discontinuity.
To further investigate the counterintuitive result discussed in Sec. 4.2, we set up a toy task on the unit circle. Here, the goal infer rotation of 0 degrees, in other words the coordinate at (1, 0) on the unit circle. For this experiment, the prediction is not conditioned, and all samples against the unique target point (1, 0). As done in Sec. 4.2, we evaluate two formulations, one where the target vector field is defined in the Euclidean space I2² and one where it follows the geodesics on the Riemannian manifold SO(2) [30]. In the following, we refer to them as Euclidean formulation formulean and (2) formulation. We provide an overview of the formulations (robotic and toy experiments) in Tab. 3.
regressed
As in the case of SO(3) (see Sec. 3.1), the input to the vector field prediction network is the truncated version of the full rotation state matrix, i.e. 2D vector. We train both formulations batches of size 1000 for 2000 epochs using the Adam optimizer with cosine annealed learning rate of 1e-3, -3, For evaluations, we evaluate 10000 noise samples and perform 50 integration steps. Given the final state, we calculate the absolute angle error to the target 0).
The Euclidean formulation achieves an average angle error of 0.093 and the SO(2) formulation 0.216 We visualize the mean eиror across initial sample points in Fig. We conclude that both formulations successfully solve the task, but we also observe higher final angle error for the SO(2) formulation, similar to the ablation study from Sec. 4.2.
As shown in Fig. 5a, when random samples are close to the opposite pole of the target sample, the SO(2) case presents strong discontinuities, with vector fields of nearby points pointing in opposite directions. For successful gradient-based learning, however, we need the input-output mapping to exhibit some notion of continuity. The exponential coordinates used to represent vectors on the tangent space do not fulfill the pre-images connectivity constraint, which means that a smooth function interpolate between input samples is not guaranteed exist [29, 35). A discontinuity in the target mapping results in difficulties in fitting the model via gradient-based learning. Additionally, if the training data for the network is not equally distributed on both sides of the pole, we introduce data bias towards either side.
For the Euclidean formulation on the other hand, the need to project the final output to the manifold of valid rotations is also source of error, since the projection is only applied at inference time and not part of the training process. This can be seen in Fig. 5b, where random samples further away from the x-axis present higher absolute angle errors. As a result, we conclude that both formulations exhibit different characteristics, each with their drawbacks, and the choice of one option over the other needs to be evaluated carefully.
11

---

Table 3: Comparison of different state dimensions and their respective velocities, target calculation and pro- gression formulations. For definitions of the Log(o)- )- and Exp(e) maps we refer to Sola et al. [30]. TAs for calculating the final angle only the first column is needed, omit calculating the second orthonormal vector. Thus, instead of performing the full state variable that maps the vector the unit circle.
process, we only perform a normalization of the inferred
Formulation
State
Velocity v
Target Velocity Vgt
State Progression
State Conversion
ED-Euclidean
RR6
R
21 20
+ At
Schhm-Schmidt
SO(3)
R3x3
R
Logsor3) (2nj 21)
Expso(3) (Ar
N/A
Euclidean S2(2)
Ra
R²
21 2g Logso12)
+ At
Srahm-Schmidt'
R2x2
RR
(2o 21)
Expso (At v)
N/A
12


Reviews:
[{"review_id": "ReO08zSMBq", "reviewer": "Reviewer_E1uS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an investigation of conditional flow matching (CFM) versus diffusion policy. The authors also ablate image versus point cloud features and different ways to learn rotations. The experiments consider several SOTA imitation learning architectures on RLBench, showing better performance for CFM and 3D point cloud features as compared to diffusion policy on image-based features.", "summary_of_recommendation": "Overall, I think this is a solid paper accomplishing what it sets out to do in terms of architecture investigations. The use of CFM as compared to diffusion policy is promising as it is more general and does not require both a backward and forward pass. The only concern with this paper that I have is the lack of hardware experiments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "In addition to the weaknesses listed above, I have the following questions for the authors.\n1. In Sec. 3.2, why use a Gaussian distribution if one of the advantages is the generality of the possible distributions? It would be interesting to explore other possibilities.\n2. For my understanding, why is the linear step in Algorithm 1 taken before the network during training and after during inference?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9kDC3kAWiO", "reviewer": "Reviewer_s1fJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces PointFlowMatch, a method combining Conditional Flow Matching with point cloud observations for robotic manipulation learning. Evaluated on RLBench tasks, it outperforms existing methods. The authors analyze various design choices, concluding that point cloud inputs significantly contribute to the method's effectiveness while other strategies such as manifold learning and normalizing flows do not.", "summary_of_recommendation": "The paper presents a novel combination of existing methods in generative modeling and pointcloud neural networks for behavior cloning. While the architecture is novel, the experiments are lacking real-world evaluations. Additionally, no supplementary materials were provided.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "See above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7YCVOPNlEt", "reviewer": "Reviewer_mSNh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper applies conditional flow matching (CFM) to imitation learning. CFM is a more general form of generative models. The method is evaluated against baselines using diffusion models. The authors also investigates the use of 6D representation and SO3 representation in the rotation matrix. The third emphasis of the work is using point cloud as input.", "summary_of_recommendation": "Trying different generative models is nice. But there is only limited discussion on why would using CFM have advantages in the context of robotics.  Results are not strong enough. After the rebuttal: Though the results are not very strong, there is some potential benefits of the framework. The conclusion on rotation formulation could be helpful to other authors. I'm inclined to accept the paper.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- Some result numbers are really close. I’m curious about the significance of the result. It’ll be great if there are some measure of variance there.\n- Are all baseline models trained using the same data generated by the authors?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces PointFlowMatch, a method that combines Conditional Flow Matching (CFM) with point cloud observations for imitation learning in robotic manipulation. The authors demonstrated improved performance compared to baselines on RLBench tasks.

Before the rebuttal, the reviewers acknowledge the importance and relevance of the proposed work but raised several concerns. The primary issue is that the results are not strong enough with the performance only demonstrated through simulation on RLBench. There are no hardware experiments to validate the method in more complicated real-world scenarios. Additional concerns include a lack of rigorous analysis on the advantages of CFM in the context of robotics, missing supplemental materials that were promised, and the absence of discussions on more related works, among others.

During the rebuttal, the authors did an excellent job addressing the reviewers’ concerns, particularly by including hardware results. As a result, all reviewers now rate the paper as a Weak Accept. However, the reviewers mentioned during the discussion that concrete comparisons between diffusion and flow methods will be necessary to further enhance the impact and relevance of the work to the robot learning community.
Decision: Accept