An Open-Source Soft Robotic Platform for Autonomous Aerial Manipulation in the Wild

Authors: erik  bauer
marc  blöchlinger
pascal  strauch
arman  raayatsanati
curdin  cavelti
robert k katzschmann
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:52:23

Abstract: Aerial manipulation combines the versatility and speed of flying platforms with the functional capabilities of mobile manipulation, which presents significant challenges due to the need for precise localization and control. Traditionally, researchers have relied on off-board perception systems, which are limited to expensive and impractical specially equipped indoor environments. In this work, we introduce a novel platform for autonomous aerial manipulation that exclusively utilizes onboard perception systems. Our platform can perform aerial manipulation in various indoor and outdoor environments without depending on external perception systems. Our experimental results demonstrate the platform's ability to autonomously grasp various objects in diverse settings. This advancement significantly improves the scalability and practicality of aerial manipulation applications by eliminating the need for costly tracking solutions. To accelerate future research, we open source our modern ROS 2 software stack and custom hardware design, making our contributions accessible to the broader research community.

Main paper: 
 introduction: 
 Aerial manipulation, which combines the agility of aerial vehicles with the functional capabilities of manipulators, poses a significant challenge in robotics. Traditional mobile manipulation platforms, typically composed of a mobile base and a manipulator, are designed to interact with their environment and perform various tasks. Aerial manipulators, using multicopters as their mobile base, offer significant advantages over ground-based platforms such as legged robots [1] and wheeled robots [2] , particularly in scenarios requiring high traversal speeds or navigation through impassable terrain, such as during floods or in debris-strewn areas.In recent years, aerial manipulation has seen significant advances using soft robotic manipulators [3, 4, 5, 6] . The compliant properties of soft manipulators relax the otherwise high constraints on precisely positioning the manipulator when grasping objects. Furthermore, soft manipulators also dampen the impact on the dynamics of the flying platform, which eases the burden on flight controllers during each grasp.However, a significant limitation of existing aerial manipulation platforms is their dependence on external perception systems, such as motion capture systems [4, 5, 7] , fiducial markers [3, 8] , and VIVE trackers [9] , to perform self-localization and target object localization. These systems simplify the research problem by providing nearly perfect pose information, but also present two core issues.First, external perception systems, particularly for self-localization, assume access to almost flawless pose data. Such strong assumptions often limit the applicability of research findings, increasing the barrier to adaptation in real-world scenarios. Second, motion capture systems can be prohibitively expensive, creating a high financial barrier that restricts initial research efforts and limits the reproducibility of existing studies.To overcome these limitations, we propose a new platform that relies exclusively on onboard perception sensors for aerial manipulation. For self-localization, we use an existing simultaneous localization and mapping (SLAM) system [10] , allowing the platform to estimate its state using visual observations alone. We implement a learning-based pipeline to localize target objects through RGB-D segmentation and point cloud analysis to determine target grasp locations.Our approach enables autonomous aerial outdoor manipulation without requiring significant prior knowledge about the environment or target objects. This work introduces a novel method that relies solely on onboard perception, distinguishing itself from previous approaches by operating without strong object priors such as computer-aided design (CAD) models or semantic object categories. To our knowledge, this represents the first demonstration of such capability in autonomous aerial outdoor manipulation, allowing the platform to adapt to diverse and unpredictable scenarios. With our platform, we aim to facilitate future research in key areas of aerial manipulation, including hardware optimization, perception, planning, and control, by providing an open-source, modular framework that researchers can easily build upon and adapt.Our contributions can be summarized as follows:1. We propose an autonomous soft robotic aerial manipulation platform utilizing soft robotics, SLAM for self-localization, and learning-based target localization. This platform demonstrates zero-shot manipulation capabilities for diverse objects across indoor and outdoor environments.2. We open source our ROS 2 software stack, our custom onboard power electronics stack, and our custom hardware design to facilitate further research on autonomous aerial manipulation. 
related work: 
 The integration of aerial systems with grasping mechanisms has gained considerable attention in recent years [11] . Research has demonstrated the potential of aerial grasping using soft grippers, which adapt to the shapes of objects due to their passive compliance [3] . Further advances include rapid grasping platforms [4] and efforts to achieve more autonomous grasping without relying on fiducial markers [12] .However, many existing systems still depend on motion capture systems to provide the precise location of the aerial platform or the object. For example, several works have focused on improving flying manipulators with stiffness control to improve object interaction dynamics [13] and developing lightweight, low activation force grippers designed specifically for aerial grasping applications [14] . In addition, Wu et. al. [15] has investigated optimizing quadcopter designs to enhance aerial grasping performance.Despite these advancements, the reliance on motion capture systems for self-localization and object priors for target localization remains a common theme. More recent work has demonstrated successful aggressive aerial grasping using onboard perception by employing visual-inertial odometry for self-localization and CAD models for object recognition [5] . Our approach differs from this by leveraging a learning-based target localization system for zero-shot video segmentation integrated with a SLAM pipeline. This method eliminates the need for prior assumptions regarding object geometry or the environment, enhancing the flexibility and applicability of aerial grasping systems in unstructured settings. 
system architecture: 
 In this section, we motivate the key design choices that form the architecture of our proposed aerial system. Due to the highly specialized nature of aerial grasping, we have designed a custom modular quadrotor frame and power electronics stack to facilitate platform development. Additionally, we have developed a custom software stack for seamless and modular development on ROS 2 in conjunction with the PX4 flight-control stack. We open source our custom hard-and software as an accessible starting point for other researchers in the field.The platform is designed with a focus on modularity and maintainability. Compared to consumergrade "ready-to-fly" platforms, which are commonly modified to serve as research platforms, our design allows for simpler customization to accommodate various downstream applications, including aerial manipulation.The platform has an x-shaped configuration optimized for minimized and symmetric inertia in roll and pitch. Our multi-level design requires a vertical assembly: gripper components are located at the bottom, electronic circuit boards and flight controllers occupy the middle level, and the onboard computer and gripper controller are mounted above. This layout simplifies access to components that require frequent adjustments. Acrylic plates provide visual access while protecting the internal components. Electronic speed controllers (ESCs) are integrated within carbon profile arms, and frame connections are made of 3D-printed polylactic acid. The takeoff weight of the platform, including the battery, is 2479 g, resulting in a payload capacity of approximately 500 g.A custom-developed power management board offers overcurrent protection, weighs less, and is more compact than off-the-shelf components. The gripper features two palms, both with two fingers inspired by the Fin Ray ® effect [16] . The fingers leverage their compliance to bend around target objects upon contact, creating reliable grasps regardless of object geometry. The fingers are 3D-printed with 85A thermoplastic polyurethane to offer the necessary flexibility and strength for effective manipulation. Our software design follows the typical autonomy pipeline, including perception, planning, and control routines. The logic flow is illustrated in Figure 2 , and the architecture of the software components is shown in Figure 3 . For low-latency communication, we use ROS 2 [17] instead of ROS 1 [18] , which is still the predominant middleware in most current works [3, 7, 13, 19, 20] . Unlike ROS 1, ROS 2 supports real-time communication, which is highly advantageous for applications requiring aggressive flight maneuvers [3, 4, 5] .Our visual perception system uses an OAK-D S2 camera, which provides aligned grayscale and depth images at 30 Hz to both the SLAM pipeline and the target localization module. The SLAM pipeline also incorporates IMU data streamed at 200 Hz. Using a single camera for both SLAM and downstream perception, we reduce overall weight and complexity compared to setups that use separate cameras for visual-inertial odometry (VIO) or SLAM and downstream perception [5] .For planning, a mission control module manages the overall mission logic, observes the current state, and detects and handles exception states or failure modes. Working closely with this module, a trajectory planner computes the subsequent flight maneuvers based on the current mission state and specifies reference trajectories to the Pixhawk flight controller. The Pixhawk 6C runs PX4 Autopilot [21] , an open-source flight control system that communicates over MAVLink [22] . To control the gripper in parallel, a separate ROS 2 interface receives commands over the network and forwards them to an Arduino Nano via UART serial communication.During testing, we frequently employ a VICON motion capture system to obtain ground-truth pose information on the quadcopter and the object. These ground-truth estimates are also used to test various modules in isolation. To access the VICON system measurements and publish them to the ROS 2 network, we employ an open-source repository from OPT4SMART [23] . The VICON system is configured to provide measurements at 200 Hz.Achieving robust motion estimation through visual observations remains a challenging problem. While different open-source methods exist that perform well in synthetic benchmarks [24, 25, 26, 27, 28] , we find that most of these published methods suffer from unsatisfactory robustness across different real-world scenarios. Multiple problems arise: high sensitivity to the camera and IMU calibration values makes it challenging to get repeatable localization results due to minor differences in calibration. Other parameters are often fine-tuned for specific synthetic benchmarks, necessitating a lengthy search to make pipelines work across different real-world environments.The modular design of our pipeline allows for seamless integration of various SLAM algorithms. We conducted a comprehensive comparison of several open-source options, including SVO [28] , Kimera [27] , Rovio [24] , and RTAB-Map [29] . Our evaluation revealed that Spectacular AI [10] consistently provided the most robust and precise state estimates. Notably, the modular nature of our system enables researchers to easily substitute any alternative SLAM pipelines as baselines when developing new state estimation algorithms. Further comparison details can be found in the Appendix (Section 6.3).We propose a novel target localization pipeline, which can be used for zero-shot grasping of different objects. In contrast to existing pipelines for aerial manipulation [5, 12] , this approach requires neither CAD models nor a semantic class label of the target objects, which enables strong generalization for zero-shot grasping.The pipeline is visualized in Figure 4 . The process is initialized with input from the operator, who selects a point of the target object in a livestream of the onboard camera. To produce a segmentation mask across subsequent image frames, we use SAM2 [30] . Given the segmentation mask and its corresponding depth image, we obtain a partial point cloud of the object.We select the grasping points [31] using the following strategy. First, we estimate the centroid of the point cloud and determine its base. We then duplicate the point cloud and rotate it by 180 degrees (assuming symmetry of the object) to estimate its front and back. Using this assumed symmetry, we determine a new centroid for the object. Finally, we determine a set of candidate points for grasping. We do so by taking the intersection of the point cloud and a cutting plane that goes through the estimated centroid and is normal to the longest axis. 
experiments: 
 To evaluate the capabilities of our proposed system, we conduct two experiments. First, we compare the accuracy of the SLAM system to that of a motion capture ground truth to quantitatively determine the quality of the state estimates using only onboard perception. Second, we show a comprehensive overview of grasp success rates for 8 commonly used household objects, using only onboard perception to perform all grasps without any explicit prior knowledge of these objects.First, we aim to investigate the pose errors induced by the Spectacular AI SLAM pipeline. To compute the relative pose error (RPE), we assume that the quadcopter pose in the motion capture system is the ground truth. In Figure 5 , we visualize an exemplary grasping trajectory expressed in the north-east-down convention, along with ground-truth data from a motion capture system. The error appears to correlate with the platform's velocity, observing a peak RPE of 0.042 during a fast descent toward the object.Overall, real-world tracking performance meets the high-precision requirements of aerial manipulation tasks as long as it initializes well. To ensure sufficient image features are present in the camera We evaluate the accuracy of the Spectacular AI SLAM system against motion capture ground truth during a sample grasping mission. All coordinates are expressed in NED (northeast-down) convention. The two trajectories have been aligned using Umeyama alignment with EVO [32] . We observe a peak RPE of 0.042 during a fast descent towards the object, whereas the mean RPE is 0.028.frames at SLAM initialization, we find that increasing the platform's initial position height to at least 80 cm above ground significantly boosts the quality of SLAM initialization.To demonstrate the autonomous capabilities of our platform, we performed nine indoor experiments in a controlled setting to ensure repeatability. The general setup consists of a predefined search area with a target object arbitrarily placed within. The platform must first search for, grasp the object within the search area, and end the mission by dropping the object at a predefined location. We employ a simple search algorithm, visualized in Figure 6 . (b) Same mission as in a), but doing it outdoors with a plush toy instead of a bottle.Figure 6 : The platform executes the same mission indoors and outdoors. First, the platform iteratively advances through a predefined search field. Upon receiving input from our segmentation system (section 3.5, the platform approaches the target object to refine the pose estimate. Based on the refined estimate, it performs a grasping maneuver.Figure 7 : The eight objects we use for grasping experiments shown in Table 1 . From left to right: plush toy, styrofoam object, cardboard box, ramen cup, bottle, kitchen roll, pouch, ball. The objects differ distinctly in size, weight, shape and surface friction. The bar denotes a length of 30 cm.In Table 1 , we present the success rates for nine experiments obtained in the indoor setting using onboard perception only. Across a total of 144 grasp attempts, our system exhibits an average grasping success rate of 85%, which falls in line with previous works that rely on external perception systems [3, 4, 12] . In contrast to these works, we show zero-shot generalization to different objects by eliminating the need for explicit object priors in our grasp planning algorithm.We can observe different failure modes. The most common failure modes are horizontal position errors incurred by the increasingly difficult control as the platform approaches the ground. These position errors can then cause the gripper to miss the target object. Secondly, we can observe that for objects with low height, the platform rarely closes the gripper above the object, which could be attributed to increased thrust closer to the ground.Furthermore, we find that the ability to grasp objects autonomously transfers to other environments, such as outdoor settings, as shown in Figure 6 , given that there are no significant disturbances that impact the controllability of the platform (e.g. wind gusts).Overall, there appear to be few to no downsides concerning the efficacy of autonomous grasping using onboard perception. In contrast, the upsides of exclusively relying on onboard perception constitute a significant step closer to scalable real-world applications of aerial manipulators across different scenarios, allowing for reliable flight without installing extensive infrastructure. We occasionally encountered issues with initializing the platform's starting pose, leading to unexpected rotations at takeoff. This resulted in a misalignment between the actual search pattern and the expected search sector, potentially causing the platform to miss the target object. To mitigate this problem, we placed the platform on an elevated surface to enhance SLAM initialization.Due to the gripper's placement below the quadcopter, all grasped objects are subjected to significant aerodynamic downwash. Consequently, the object's position may shift during the grasping attempt when it is out of the camera's field of view. To minimize this issue, most grasping objects were placed on surfaces with artificially increased friction or mounted on stable supports (e.g. duct tape).Our grasp planning algorithm operates on partial object point clouds and assumes approximate symmetry of target objects. This assumption limits the grasp planning capabilities of the platform, which could be addressed using point cloud reconstruction techniques [33] Furthermore, long-term video segmentation and tracking is not possible with SAM2, as the amount of memory and compute time increases with each frame. While sufficient for short grasping missions, using only short-term video segmentation (less than 400 frames) could be a limitation for other usages. 
conclusion: 
 In this work, we have introduced a new autonomous robotic aerial manipulation platform that can localize and grasp targets indoors and outdoors. Using an off-the-shelf SLAM system for selflocalization and a learning-based pipeline for target object localization, our platform can operate independently of external perception systems. This autonomy removes the constraints imposed by external motion capture systems, enabling the platform to perform aerial manipulation in various environments, including challenging outdoor settings. With 144 flights for nine zero-shot grasping experiments, we demonstrate that our platform shows competitive grasping efficacy (85%) compared to previous state-of-the-art works using external perception systems.Our approach addresses significant limitations in the current state of aerial manipulation research, particularly the reliance on pose information provided by external systems and the high costs and limited mobility associated with such equipment. By eliminating these dependencies, our platform broadens the applicability of aerial manipulation. In addition, the open-source release of our ROS 2 software stack, onboard power electronics, and custom hardware design aims to lower the initial investments to facilitate research and development in autonomous aerial manipulation with soft robots.In summary, our contributions demonstrate the feasibility of autonomous aerial manipulation using onboard perception and pave the way for future research and real-world applications in this domain. Future work could address non-symmetric objects in grasp planning and downwash effects.6 Appendix 
parts list: 
 Additional resources for replicating our hardware platform are available in our public repository at https://github.com/srl-ethz/osprey. All files for sketches, 3D-printed linkages, lasercut plates and an assembly guide can be found there. Given an RC Control unit, a workshop, and miscellaneous small parts such as screws, the total cost should be around $ 1274. See Table 2 Our custom PMB, shown in Figure 8 , supplies power to the onboard computer (19 V at up to 2 A), the gripper's servo motors (6 V at up to 3 A), and various onboard electronics (5 V at up to 2 A). The Arduino Nano is directly connected to the gripper's servos to relay setpoints from the onboard computer. 
 
 Appendix: We validate the platform's effectiveness and versatility through flight and grasping experiments, demonstrating its ability to autonomously grasp a number of objects with a high success rate. 


Decision: The authors propose an open platform for robotic manipulation. The system is unique, and will hopefully enable more future work. The authors showed that the system is versatile and produce a system that should be reproducible, and can grasp a wide variety of different objects. They describe many different potential research directions enabled by their device, and go to lengths to make sure it is reproducible. The paper is clear and reasonably well-written.

Strengths:
- Grabbing objects with a gripper on a flying drone is interesting and a challenging problem
- System seems very novel and proposes a unique problem
- The system is mostly open-source and seems like it will be reproducible by other labs

Weaknesses:
- Authors should better describe what research questions they think the platform can be used for
- Experiments don't show it being used on very many objects
- Authors should clarify their novelty and contributions

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper presents an open source system for aerial manipulation research that relies solely on onboard sensors for perception. This includes the choice of quadruped, OAKD camera, laser range finder, choice of SLAM algorithm, object detection stack, and hardcoded grasping routine. The authors demonstrate grasping results on two objects - a plushy and a water bottle.
main_review: main_review: The paper presents an interesting alternative to mobile manipulation for moving objects around which I have not seen before - mounting a gripper on a drone and using it to grab objects. However, I think the platform is still not at the stage where it can be published and used by the robotics community. 
- The paper claims that the platform is completely open source and unlike prior work only relies on onboard perception. However, the key to enabling this is a proprietary SLAM algorithm from Spectacular AI. Using proprietary SLAM makes it impossible for researchers to modify and improve the onboard perception of the robot. 
-  I am unsure of the utility of this platform because of the limited evaluation. Results are only shown on two objects - a water bottle and a plushy. The former is only shown in a single upright pose, while the latter is highly deformable and very forgiving to incorrect grasps. It is unclear whether their platform would work for more complex objects. 
- The authors are proposing a platform for research, but it is not discussed what research problems it can be used to study. When proposing a platform, ideally authors should showcase multiple potential areas for research and how their platform can help. For example, they could propose a benchmark for control / perception or agile grasping with existing baselines, train / test splits. For eg, they can provide an open source simulator and sim2real pipeline for researchers to study sim2real techniques. In its current form, it is unlikely anyone would adopt this platform.
questions: questions: - When conducting grasping experiments, to what degree is the initial pose of the object varied? Eg. what happens if the bottle is kept on its side instead of upright?
- What are the main failure modes observed in the case of the plushy and the bottle?
limitations: limitations: 3
review_summary: review_summary: I vote to weak reject this paper because in my opinion the proposed platform is not yet ready for use by the wider robotics community due to its reliance on proprietary SLAM + limited demo results.
title: title: The paper proposes an aerial manipulation framework for drones equipped with a soft robotic gripper to fetch objects and transfer them from one point to another.
paper_summary: paper_summary: This paper proposes an interesting aerial manipulation problem setup, tested in the wild using an onboard perception sensor. While traditional methods rely on off-board vision such as using motion capture, this work uses an onboard sensor, allowing it to efficiently perform the task in both indoor and outdoor settings in a cheap and effective way without relying on expensive tracking solutions.
main_review: main_review: The paper is well written, clearly organized as well as well motivated. The hardware design is carefully curated in order to cater to the problem. Since the project is also open-sourced, it will be an impactful contribution to build upon. While the task of aerial manipulation is particularly challenging yet impressive, the results both qualitative as well as quantitative show promise. It would have been interesting to see the experiments on a more diverse variety of objects where the adaptability of a soft-gripper can be clearly seen such as oddly shaped objects, or objects of different shapes and sizes. Certain key factors such as reliance on symmetry of the object (as per the pre-processing of the partial point clouds in a certain way) and laser sensor for when the object is not in the field of view might be a limiting factor for its generalizability. Localization and stability challenges due to sensor noise as well as mobile manipulation pose additional challenges for the successful deployment of such a system.
questions: questions: I request the authors to address the following concerns:

1. It is unclear how the agent deals with the uncertainty estimates from the time of detection when the object is in the field-of-view of the onboard camera to actually grasping it when the camera cannot see it anymore. How does it affect based on the size of the object and hence the precision required to successfully grasp it? 

2. Does the agent behavior change based on the changes in the physical properties of the object such as its mass, odd/asymmetric shape as well as the coefficient of friction?

3. How does poor depth under inappropriate lighting conditions or holes in the depth image under near proximity affect the visual sensing of the agent?
limitations: limitations: 3
review_summary: review_summary: This paper presents some novel and interesting results in the field of aerial manipulation along with hardware experiments. The technical contributions are significant, however the experimental results could have been more diverse and exhaustive, displaying the compelling efficacy of the proposed approach. I propose to give it a weak acceptance.
title: title: An interesting work and kudos to the opensource effort. But seemingly lack algorithmic contributions... Decision updated on Aug 15
paper_summary: paper_summary: This paper introduces an opensource quadrotor platform that carries a soft gripper for aerial manipulation. The platform carries stereo cameras for localization and target detection. The entire autonomy pipeline is introduces as well as the hardware components onboard. I have a hard time summarizing the contributions in this works because I think the major efforts were on the integration existing solutions to build this platform. No specific learning-relevant algorithms or designs have been discussed.
main_review: main_review: The paper clearly presented the open-sourced platform. Based on the demo video in the attachment, I think the platform is suitable for benefiting other research activities that can help improve the status quo of aerial manipulation.

My concern is that the contributions seem inadequate in this work because it is hard to tell what unique contributions are made by the authors in this platform that are not taken from an existing hardware/software/algorithm. Maybe the power module board is new, but the novelty is comparatively minor. Besides, when the authors claim that this is the first work of autonomous aerial outdoor manipulation using onboard perception, they should be aware of the following literature that has achieved it already:

Ubellacker, Samuel, Aaron Ray, James Bern, Jared Strader, and Luca Carlone. "Aggressive Aerial Grasping using a Soft Drone with Onboard Perception." arXiv preprint arXiv:2308.06351 (2023).

One suggestion to the authors is to really develop a new learning-based algorithm in the task of object manipulation given the convenience of experimentation enabled by this platform. I appreciate the efforts of the authors to opensource this architecture. But the contributions do not match the bar of this conference.

update on Aug 15: the decision has been changed.
questions: questions: None.
limitations: limitations: 1
review_summary: review_summary: I recommend rejecting this paper because of lack of sufficient contributions.