DeliGrasp: Inferring Object Properties with LLMs for Adaptive Grasp Policies

Authors: william  xie
maria  valentini
jensen  lavering
nikolaus  correll
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 08:49:52

Abstract: Large language models (LLMs) can provide rich physical descriptions of most worldly objects, allowing robots to achieve more informed and capable grasping. We leverage LLMs' common sense physical reasoning and code-writing abilities to infer an object's physical characteristics-mass $m$, friction coefficient $\mu$, and spring constant $k$-from a semantic description, and then translate those characteristics into an executable adaptive grasp policy. Using a two-finger gripper with a built-in depth camera that can control its torque by limiting motor current, we demonstrate that LLM-parameterized but first-principles grasp policies outperform both traditional adaptive grasp policies and direct LLM-as-code policies on a custom benchmark of 12 delicate and deformable items including food, produce, toys, and other everyday items, spanning two orders of magnitude in mass and required pick-up force. We then improve property estimation and grasp performance on variable size objects with model finetuning on property-based comparisons and eliciting such comparisons via chain-of-thought prompting. We also demonstrate how compliance feedback from DeliGrasp policies can aid in downstream tasks such as measuring produce ripeness. Our code and videos are available at: https://deligrasp.github.io

Main paper: 
 estimate the ripeness of a ripe and unripe avocado :: 
 -A ripe avocado typically has a low spring constant , indicating softness , perhaps similar to Avocado 1 , which has the lowest spring constant -An unripe avocado has a high spring constant , as it 's stiffer , like Avocado 3 , with the highest spring constant 1. Rank the given avocados by ripeness :-Avocado 1 , with the lowest spring constant range 65 , would generally be the ripest .-Avocado 2 , with a spring constant range 143 , would be less ripe than avo1 .-Avocado 3 , having the highest spring constant range 335 , would be the least ripe .So the ripeness rank from ripest to least ripe would be : Avocado 1 > Avocado 2 > Avocado 32. Estimate when each avocado will go bad :-Avocado 1 , being the ripest , based on its lower spring constant , could spoil in about 2 -3 days if stored at room temperature .-Avocado 2 is less ripe than avo1 and may take around 4 -5 days to spoil at room temperature .-Avocado 3 is the least ripe , with quite a high spring constant and could take upwards of a week to spoil , potentially around 7 -10 days at room temperature .Please note that these estimates are quite rough and actual spoilage times can vary due to numerous factors such as specific room temperature , humidity , and the avocado 's initial condition . The visual clues are limited , but from the image , the physical characteristics that usually indicate ripeness ( dark skin , slightly yielding to gentle pressure ) don 't provide significant additional information beyond the spring constant data . 
introduction: 
 Large language models (LLMs) are able to supervise robot control and learning in manipulation from high-level step-by-step task planning [1, 2, 3] to low-level motion planning [4, 5] . LLMs additionally aid in robot manipulation via understanding a given object's semantic properties and delineating appropriate grasp positions conditioned on those semantic affordances [6, 7, 8] .These works usually assume that the acts of "picking" and "placing" are straightforward tasks. This is not the case for contact-rich manipulation, in which LLM-supervised methods [9, 10, 11] still do not account for force-adaptive tasks like grasping a paper airplane or ripe fruits and vegetables and are prone to damaging such objects. Algorithmic methods for grasping delicate objects [12, 13, 14] require custom hardware and are tested on a limited set of items. LLMs provide an opportunity to leverage their common-sense physical reasoning [15] to produce grasp skills which are both forceadaptive and for the open-world. This is particularly important for semi-structured environments like supermarkets that are subject to a constantly rotating stock or dealing with loose food items such as fruits, vegetables, and pastries that come in a large variety of changing shapes. Yet, LLMs trained on public data are unlikely to contain information on every possible object, in particular when considering specialty domains such as warehouse picking [16] or industrial assembly [17] that often include bespoke parts.We propose DeliGrasp, an extension of LLM-supervised robot learning to contact-rich manipulation. We posit that LLMs can infer the physical characteristics of gripper-object interactions, including mass, spring constant, and friction. We then formulate an adaptive grasp controller with slip detection derived from the inferred characteristics, endowing LLMs embodied with any forcecontrollable gripper with adaptive grasp skills for objects spanning a range of weight, size, fragility, and compliance. We also compare DeliGrasp against traditional adaptive grasp algorithms, as well Paired with open-world localization and pose estimation (left), our method (middle), queries LLMs for the salient physical characteristics of mass, friction, and compliance as the basis for an adaptive grasp controller. DeliGrasp policies successfully grasp delicate and deformable objects (right). These policies also produce compliance feedback as measured spring constants, which we leverage for downstream tasks like picking ripe produce (middle).as demonstrate how object knowledge can be augmented by finetuning the LLM using propertybased comparisons.We pair DeliGrasp with an open-world localization pipeline which, given an "object description," identifies the object and an initial grasp position. The same "object description" and associated "grasp verb" are the inputs to DeliGrasp, which produces executable Python code controlling a gripper's compliance, force, and aperture as a complete grasp policy for the given description.By conducting robotic grasping experiments on 12 different objects, we show that DeliGrasp performs successful, minimally-damaging grasps on a custom benchmark of delicate objects which traditional adaptive grasping methods are not capable of. We then improve property estimation and grasp performance using the PhysObjects dataset [11] as well as by eliciting explicit comparisons with chain-of-thought prompting (CoT) [18] . 
related work: 
 LLMs, equipped with an internet-scale amount of common sense information and logical and physical reasoning [18] , are able to comprehend high-level task knowledge, physical context, and robot affordances. For task and motion planning, LLMs can generate navigation and pick-and-place instructions to complete complex and novel tasks [1] . LLM code-writing further augments robot capabilities with closed-loop control [2] , new skill generation [3] , and translating language to robot parameters for low-level dynamic control [4] . For semantically-afforded grasping, LLMs, in conjunction with other learning methods [8, 19] , can identify appropriate grasp locations [6, 7, 20] .For contact-rich manipulation, LLMs have been finetuned on [9, 21] to learn relative object properties such as mass or fragility, but are extensible to low-level control [11] . They have also been prompted with environmental and object properties [10] to parameterize force constraints for robot motion, but not gripper-object interaction. No methods, however, address both the semantics and dynamics requisite for delicate grasping of a broad variety of objects. In comparison, our method utilizes LLMs to estimate object physical properties, which such models are more abundantly pretrained on than code, reward functions, or other niche robot control domains. These estimates are then paired with an underlying algorithmic grasp controller. LLM property estimation, as an approximation of privileged expert knowledge, provides two advantages: simplification of our underlying adaptive grasp controller relative to classical methods, which rely on in-motion slip detection for adaptive grasping and clarity of the low-level controller such that it is interpretable and predictable.Grasping delicate objects can be achieved with hardware, such as relying on the compliance of soft grippers, but these end-effectors still require semantic information for control if these grippers must grasp both lightweight and heavy objects [22] . In this paper, we adopt a control algorithm that relies on interaction force measurements from a rigid gripper similar to [12, 13, 23, 24] for minimally deforming grasps and for measuring spring constants [25] . Where these adaptive grasping methods are hardware-specific, ours can be adapted to any force-controllable end-effector, and LLM-estimated object properties reduce controller complexity by approximating privileged information. 
methods: 
 We source delicate objects for our dataset primarily from supermarkets, kitchens, and food pantries, shown in Figure 2c and describe object mass and object-specific thresholds for unsuccessful, "deforming" grasps in Table 1 , shown in Figure 2a . For objects with empty entries in the "Input Phrase" column, we do not modify or add descriptions beyond the name of the object. The evaluated objects span from 0.8 to 900g and soft produce to rigid plastic, and they are commonly grasped in real-world environments. We compute approximate minimum grasping force, Fmin from measured properties. "Object Description" inputs are paired with a grasp verb, "pick," to DeliGrasp prompts to generate policies.We also qualify what kind of damage or "invalidating deformation" renders a grasp a failure.Our pipeline takes as input a an "object description", as described in Table 1 , and a "grasp verb," which for the delicate object dataset evaluation is uniformly "pick." Our perception method, adapted from [6] , takes the "object description" and semantically segments the object from an RGB-D image with OWL-ViT [26] and "Segment Anything" [27] . We crop the corresponding depth image with the generated mask, produce a point cloud object representing the segmented object, and perform Principal Component Analysis to compute a grasp pose that is aligned with the first three principal axes as well as a minimum object grasp width [28] , an example is shown in Figure 1 , left.We then prompt the LLM to define and generate grasps with a dual-prompt structure similar to that of Language to Rewards [4] , with an initial grasp "descriptor" or "thinker" prompt which produces a structured description, which the subsequent "coder" prompt translates into an executable Python grasp policy that modulates gripper compliance, force, and aperture according to Algorithm 1 (full prompts in Appendix A.6, A.7). To execute grasp policies, we use a UR5 robot arm with the opensource MAGPIE gripper [29] and implement current-draw derived force control, from a range of 0.15 N to 16 N (default output force is 4 N). Our experimental setup is shown in Fig. 2bA . We manually score grasp failures according to whether the object slipped or if the damage experienced matches the "invalidating deformation" criterion put forward in 1. Fig. 2b shows the interaction between our gripper and an arbitrary object with mass m and spring constant (compliance) k. The gripper exerts a composite applied gripper force F a = F lef t + F right that leads to a frictional force F f = µF a , where µ is the Coulomb friction coefficient between the gripper and object [28] that counteracts the force of gravity mg (g is the gravitational constant).For compliant objects, approximated as ideal springs, we can additionally describe the left and right gripper forces F l,r = F spring l,r = kx l,r , where x is the compression of the grasped object.Typically, object slip within a gripper is detected when after the gripper grasps an object at rest, the gripper begins some upward acceleration a lif t , and an object begins slipping with some downward acceleration a slip . Increasing F a to account for ma lif t yields an adaptive minimum applied grasp force F min which prevents slip and is minimally deforming [23] :F min = m(g+a lif t ) µConversely, when an object is slipping with a slip , the applied force F a slip = m(g-a slip ) µ [12] .When a lif t,slip are 0 and the gripper and object are at static equilibrium, F min = mg µ . F min can then be arranged in relation to these quantities, where m * , µ * are LLM-estimated terms approximating ground truth measurements.:EQUATIONFor the delicate objects dataset, we estimate a minimum applied grasping force F min = mg µ with a conservative, uniform Coulomb friction coefficient of µ = 0.33, used to compute F min in Table 1 .Mass and friction, m and µ, which determine successful grasp forces, as well as spring constant k form the basis of our force-adaptive algorithm for non-slipping, minimally deforming grasping of delicate objects. We define a closed-loop force controller: starting from an estimated target aperture that corresponds to the estimated object's width w from RGB-D data, we increase the gripper output force F out and decrease gripper aperture x until sensing a contact force F c greater than the target F min [24] . To determine the gain terms, ∆F out and ∆x, i.e. how fast we close the gripper and ratchet up force, the controller uses the agent-determined k and ∆x; we change F out by c • k∆x, where c = 0.1 is a dampening constant. We describe the controller in Algorithm 1.F c = SetGripper(x = w, 0) while F c ≤ F min do F out += c • k∆x x -= ∆x F c = SetGripper(x, F out )We query an LLM (GPT-4) for these quantities once to formulate grasp policies. Accurate predictions remove the need for parameter tuning [24] and additional gripper sensors [12, 14] . By default, we instruct the LLM to compute F min = mg µ and we do not account for a lif t , choosing to err closer to object slip than deformation. However, we also provide the LLM with agency to deviate from the default F min depending on the "grasp verb" provided.On average (n=30), generating one DeliGrasp policy with GPT-4 takes 9.98s (3.03s for 1x 'Thinker' prompt query, and 6.65s for 1x 'Coder' prompt query). DeliGrasp policies take on average 4.13s to execute (Appendix A.1). Total grasp time sums to, on average, 14.11s, compared to approximately 0.5s for a force-limited grasp, or 1.7s for a classical adaptive grasp [13] .To demonstrate how LLM-knowledge can be improved and potentially include bespoke objects, we finetune GPT-3.5-Turbo on 6000 captions of PhysObjects images [11] , each describing two objects, their materials, and their relative fragility, deformability, and mass. The dataset captures 276 unique common household objects, with which the delicate objects dataset shares only the mandarin and plastic bottle. We also augment the grasp "descriptor" or "thinker" prompt with CoT prompting [18] , paired with finetuned and non-finetuned models, to elicit a series of quantitative comparisons of mass: objects that are 1) lighter and 2) heavier, the mass of a 3) typical object of the category being grasped, and the mass of the 4) user-described object relative to the typical object, thereby producing explicit bounds on the object mass (see Appendix A.8 for greater details on PhysObjects finetuning data format, the full CoT prompt, and CoT query latency).We evaluate DeliGrasp against four classical adaptive grasping baselines, two for each strategy: "In Place" and "In Motion." "In Place" grasping closes the gripper around an object at rest until a measured contact force of 2N or 10N, adapted from [24] . The "In Motion" strategy, adapted from [12, 13] , closes around an object at rest until a contact force of 1.5N or 0.5N and then moves upward at approximately 10 mm/s for 5 seconds. During this upward motion, the "In Motion" strategy performs closed-loop adaptive grasping at approximately 80Hz, increasing applied gripper force and velocity proportional to measured slip force. We describe the baselines implementation and differences from DeliGrasp in Appendix A.3. 
experiments: 
 We benchmark DeliGrasp (DG) against five grasp policies: in-place adaptive grasping with 2N and 10N grasp force thresholds, in-motion adaptive grasping with 0.5N and 1.5N initial force, closing the gripper fully or until it is output force limited (F.L.), closing the gripper to the visual width of the object determined by our perception method, and an ablated DeliGrasp-Direct policy (DG-D) which directly generates F min , ∆F out , and ∆x without first reasoning about an object's physical properties. We also compare DeliGrasp with PhysObjects finetuning (DG-FT) and with finetuning and chain-of-thought prompting (DG-FT-COT), selecting the model configurations with the highest reduction in mass overestimation (Fig. 5 ).We employ each grasp policy 10 times with objects placed randomly within a 30 x 45 cm bounding area on a table, and do not record attempts which receive faulty poses from perception. Quantitative results are shown in Figure 2 . Figure 5 shows actual grasps of an empty paper cup and the same paper cup filled with water as well as the LLM-generated code. CoT to physical reasoning prompting. DG-D uses an ablated approach that skips the first-principle model and estimates algorithmic parameters directly. The four variants are comparatively similar in performance, though DG-D deforms objects slightly more than the others due to higher estimated grasping force. We compare to two classical adaptive grasping strategies: In Place adaptive grasping until a threshold contact force of 2N or 10N and In Motion adaptive grasping with an initial contact force of 1.5 or 10N and then increasing applied force proportional to measured slip. These classical methods are not able to generalize to the full delicate objects dataset and require expert hand-tuning of controller parameters. We include two naive baselines: Visual, which closes the gripper to the estimated object width and F.L., which closes the gripper to its maximal force, 16N.The base DeliGrasp model (80%), finetuned model (76.7%), finetuned model with CoT prompting (75.0%), and ablated DeliGrasp Direct prompt (70.0%) all perform successful, minimally-deforming grasps that the other baselines are not capable of on objects like the paper airplane and raspberry. DeliGrasp dominates hardware-limited (28.3%) and vision-only grasps (29.2%) in 8 out of 12 objects, and is better or on par in 10 out of 12 objects. The "In Place" strategy completes 32.5% (2N) and 31.7% (10N) of grasps successfully. The "In Motion" strategy completes 55.0% (0.5N) and 50.8% (1.5N) of grasps successfully. Per-item performance is shown in Table 2 and full DeliGrasp policy outputs are provided in Appendix A.1.We look closely at failure modes to better understand the performance of each policy. Given two bounds of failure-slip grasps and deforming grasps, DeliGrasp and its FT and CoT variants perform the same or better than DeliGrasp Direct in 10 out of 11 cases. These variants overestimate F min terms (individual estimates in Table 7 ) but primarily perform slip failures. However, for all objects DG-D estimates even higher F min values and deforms objects at a higher rate. DeliGrasp overestimates F min by a factor of 1.95, whereas DG-D, which performs additional invalidating deformations, overestimates F min by a factor of 2.55. Since mass, a common quantity, is more accurately estimated by an LLM, resulting adaptive grasp policies parameterized from mass are more robust than policies directly parameterized by an LLM with more erroneous predictions. All policies slip at high rates on the stuffed animal (grasped by the tail), spray bottle, bag of noodles, and bag of rice. These objects are non-linearly and/or very compliant and high volume, exerting lever forces on the gripper that likely exceed F min .DeliGrasp on average overestimates object mass by a factor of 2.5, and underestimates mass only for one object (Appendix A.1). Figure 5 shows that mass overestimation can be reduced by employing chain-of-thought reasoning (reduction to factor of 1.82), where the LLM is prompted to explicitly relate its mass estimate to that of other objects, or by fine-tuning GPT-3.5 on the PhysObjects dataset For non-DeliGrasp methods, we observe that qualitatively, the different "In Place" strategies are moderately successful on opposite ends of the evaluation set. The 2N strategy still performs invalidating deformation on the paper plane, empty cup, yuba, raspberry and is not forceful enough for the heavier objects. The 10N strategy deforms lighter objects and delicate heavier objects like the cup of water and the avocado.The "In Motion" strategies are more performant, but we observe the same performance trends between the 0.5N (more successful on lighter objects) and 1.5N strategies (more successful on heavier objects). We observe that both 0.5N and 1.5N strategies perform slip failures on small and dense objects, such as the avocado or squeeze bottle, due to low surface area of purchase and large measured slip force preventing the controller from adapting in time. Inversely, the controllers are relatively aggressive on lighter or otherwise delicate objects, leading to deformation on the paper plane, water cup, and avocado. Compared to DeliGrasp, LLM common sense reasoning enables a dynamic grasping range in object mass and stiffness.We note that both classical adaptive grasping methods crush the raspberry. Berry-like objects are low-density, soft, and rough, requiring precise and "delicate" grasping force. The 0.5N "In Motion" strategy applies a low initial force with slight deformation but crushes the raspberry during closedloop control due to its aggressive gain terms. Categorically, force-limited grasps deform and visiononly grasps slip. Force-limited grasps succeed only with robust, compliant objects. Vision-only grasps succeed when the objects are relatively stiff and are sensitive to grasp pose.Since the baseline DeliGrasp prompt policies are performant with overestimated mass, we record grasp performance by querying complex and atypical objects, such as a "larger avocado" (307 g), a wet sponge (92.0 g), a crochet yarn lily flower (26.5 g), and a 10in length of old-growth 2x4 (925 g), and record 10 grasps per configuration (Table 3 ). We compare across different DeliGrasp configurations-the original DeliGrasp prompt with GPT-4-Turbo, the original prompt with the finetuned GPT-3.5-Turbo model, the CoT prompt with GPT-4-Turbo, and the CoT prompt with the fine- We compare grasp performance from four DeliGrasp configurations, the original DeliGrasp prompt with GPT-4-Turbo (DG), the original prompt with the finetuned GPT-3.5-Turbo model (DG FT), the CoT prompt with GPT-4-Turbo (DG CoT), and the CoT prompt with the finetuned model (DG FT COT), on atypical objects such as a large avocao, a wet sponge, a crochet yarn flower, and old-growth 2x4 lumber. Finetuning, paired with CoT prompting, enables semantically appropriate mass estimates for complex and/or atypical objects.tuned model. We observe that pairing the fine-tuned model with CoT prompting fully leveraged PhysObjects-like physical reasoning and produced successful grasp policies from an initial descriptive query, due to more accurate initial mass estimates.The finetuned model with CoT prompting are able to grasp each object, due to more accurate mass estimates resulting from more correct CoT-prompted bounds on object mass. The other methods cause majority slip failures for the larger avocado, the crochet flower, and lumber due to an unchanged mass estimate. For the wet sponge, the other models both overestimate its mass and squeeze the sponge too tightly, or underestimate its mass. The non-finetuned models do not acknowledge the semantic relationship between the provided modifiers ("large," "wet," "crochet yarn," or "old growth") on the mass of a typical object until the user re-queries in the same context with the explicit relationship semantics, such as "the avocado is larger than average, try again." In Appendix A.4, we show how DeliGrasp can also generate atypical grasps conditioned on the verbiage provided (e.g. generating a less forceful grasp for "checking" versus "picking"). 
conclusion: 
 We introduce DeliGrasp, which uses LLMs to 1) infer semantic, common sense physical information which 2) parameterizes adaptive grasp policies effective on a variety of delicate and deformable items that traditional adaptive grasping methods cannot grasp. These inferred m, µ, k characteristics enable more consistent control than direct estimation of grasping forces (LLM-as-code) and classical adaptive grasping methods. We also improve property estimation accuracy and grasp performance on atypical and complex objects with model finetuning on object property comparisons and chainof-thought prompting. We then show how DeliGrasp's compliance feedback can be used to measure produce ripeness.Limitations and Future Work. DeliGrasp policies are performant, but dependent on stable and accurate property estimates. For each LLM-estimated quantity in this work, we select the mode of 10 estimates, which can be done offline and reused for simple objects, but not for semantic modifiers or complex and abstract objects. Furthermore, we do not systematically explore the variance of each quantity, or policy robustness to properties besides mass. One interesting line of future work is leveraging iterative observations by mobile robot systems such as humanoids, rather than additional learning methods, to improve system estimates of such properties. We demonstrate this already, as DeliGrasp policies empirically measure k and object width, superseding less-accurate LLM k estimates and perception-based width measurements. We anticipate that perception and physical interactions can produce higher fidelity information to inform manipulation of unknown objects, akin to how humans explore dense, cluttered, and unstructured settings like supermarkets, home kitchens, and other such representative domains. Though finetuning on PhysObjects did not improve grasp performance on our delicate objects dataset, finetuning on this kind of higher fidelity information about complex or out-of-distribution entities may improve grasp performance. 
 
 Appendix: The MAGPIE gripper has a palm-integrated Intel RealSense D-405 camera. The gripper is powered by two separate Dynamixel AX-12 motors, which allow maximum current control. Translating torque via a four-bar linkage, the gripper force can be controlled and sensed from 0.15 N to 16 N, and has an aperture range from 106 mm fully open to 0 mm fully closed (default closure speed of 100 mm s ).Regarding delicate object evaluation, the YCB [30] and RoboCup@Home [31] datasets include some fragile and deformable objects, but there does not exist a dataset focusing on delicate objects with defined failure modes. By combining (1) LLM estimates of object properties to parameterize for robust and damage-free grasping of delicate objects and (2) an algorithmic adaptive grasp controller, our method is able to generalize across many objects and produce uniform grasping feedback which can further inform the controller, LLM, or user.Computed ∆F out which are less than 0.01 N are set to 0.01 N. Computed F min values below the force sensing threshold of 0.15 N are set to that threshold when checking for slip or setting output force. We also show the corresponding values for the ablated DeliGrasp-Direct policy. We did not record time, experimental k values, or x f inal in these experiments. We also provide the full per-item breakdown mass estimates from the full set of evaluated models.(N) F err (N) ∆F out ∆x x f inal x goal F out Time k max k (g) (N/m) = mg µ (N) (mm) (mm) (mm) (N) (s) (N/m) (N/A. DG DG D DG F T DG F T CoT ID Object Name F m k F F m m µ k F F F F m m F F m m (N) (g) (N/m) (N) Err. (g) Err.(N/m) (N) Err. (N) Err. (g) Err. (N) Err. (g) Err.1 Paper Airplane 0.02 0.8 184.9 0.10 4.00 5 6.25 0.5 20 0.15 6.50 0.10 We record LLM estimates for F , the minimum grasping force, the relative error F Err., mass m, and ratio of mass overestimation m Err. for 5 different DeliGrasp strategies: DG with the default model without finetuning, the ablated DGD which directly estimates F , DG F T with the model finetuned on the PhysObjects dataset, and DG F T CoT for the finetuned model with additional chain-of-thought prompting to elicit PhysObjects-like comparisons. We use the same µ and k values across models to isolate the effect of mass estimates on grasping. Lower mass, and thus F estimates for DG F T DG FT COT for the bag of noodles and hard-shelled taco explain why resulting policies in 7 have more slip failures.We first validate these baseline methods on the evaluated objects from their respective studies, spanning a potato chip, tomato, and paper cup (2g to 150g) for "In Place" grasping and an empty styrofoam cup, filled styrofoam cup, empty plastic water bottle, filled plastic water bottle, and filled cereal box (68g to 384g) for "In Motion" grasping. For the "In Motion" strategies, we tune a separate set of applied force gain and velocity gain for the 0.5N and 1.5N contact force strategies. The 0.5N strategy has more aggressive gain terms than the 1.5N strategy, and our current-draw based force sensing used in "In Motion" adaptive grasping is much lower resolution and frequency than dedicated force sensing resistors and slip sensors used in the original baselines. We note that the "In Motion" strategy, in various implementations, requires, in addition to a custom sensor configuration across force, contact, slip, and/or pressure, expert tuning of controller parameters. DeliGrasp requires no sensing beyond the gripper motor(s) current draw and no parameter tuning. Perhaps the even larger obstacle is that such strategies require motion to execute, limiting efficacy on certain geometries and inducing slip uncertainty in a majority of grasps, whereas ours executes entirely in-place and is equally effective across object types.DeliGrasp explicitly measures a spring constant k while performing Algorithm 1: the adaptive grasp controller checks for contact force F c with each small motion ∆x, and k = Fc ∆x . These measured k (Appendix A.1) show that DeliGrasp estimates of spring constants are incorrect by a factor of 6.5 (2.5 for only the compliant objects) and, more relevantly, can be utilized as a compliance sensor in tasks such as appraising produce ripeness, a correlate of compliance (Figure 6b and 6c ). This sensor data can be used for learning tasks, human collaboration tasks, or for higher-level reasoning tasks such as ripeness checking or appropriateness for specific dishes. We perform this measurement on two assortments of produce: avocados and tomatoes. When receiving the "grasp verb" of "check" or "inspect for ripeness", DeliGrasp bypasses the default minimum grasp force value of mg µ and manually sets a lower F c of 0.5 N for avocados and 0.2 N for tomatoes.Qualitatively, the relative ordering of the measured k for each item type corresponds with judgments from our own human hand grasps. We query an LLM (GPT-3.5) with the item name and compliance data, and show in Table 6c that it is able to reason over and answer abstract, relative questions about when to eat them. Further qualitative questions such as "how" to eat a specific item "right now" yield dishes appropriate for the ripeness of the produce (Appendix A.5).1. Asking "how should I eat avocado k 3 " yields: sliced avocado, grilled avocado, salad toppings, stuffed avocado 2. Asking "how should I eat avocado k 1 " yields: guacamole, avocado toast, sushi rolls, smoothies 3. Asking "how should I eat tomato k 4 " yields: sliced for burgers, grilled or roasted, salsa, stuffed 4. Asking "how should I eat tomato k 1 " yields: caprese salad, gazpacho, pasta sauceWe additionally observe that multimodal models like GPT-4V or OWL-ViT alone cannot reason ripeness over visual images. We query GPT-4V with an image of the same items 5 times and request a ripeness ranking. For the 4 tomatoes, GPT-4V produces the same ranking each query, from most ripe to least ripe, of: [k4, k2, k3, k1], reasoning entirely on the "redness" of the tomatoes. For the 3 avocados, it refuses to generate a ranking, citing a fundamental lack of sensory capabilities.OWL-ViT when directly queried for "the ripest" or "least ripe" produce arbitrarily detects items (sometimes the same item) depending on the angle (and thus lighting, resolution, relative object size) of the captured image.When we pair the spring constant data with the image in a query to GPT-4V, it ignores the visual characteristics and reasons solely on the spring data.: Query:I have labeled 3 avocados for you between 0 , 1 , 2 and provide their average spring constants from an initial ripeness check . Spring constants are inversely proportional to ripeness --a low spring constant means a soft object , and high means stiff . Combining this information with the image , please do the following tasks : 1. Estimate the ripeness of a ripe and unripe avocado . Response:To perform the tasks , we need to use the given spring constants and the visual clues from the image to estimate the ripeness and predict when each avocado will spoil . Typically , a ripe avocado has a spring constant at the lower end because it is softer , while a less ripe or unripe avocado has a higher spring constant because it is firmer .The descriptor takes as input a "grasp verb", "object description", and "object width." It outputs a structured summary of the object, how it should be grasped, and physical characteristics of the object.Control a robot gripper with force control and contact information .The grippe r 's parameters can be adjusted corresponding to the type of object that it is trying to grasp as well as the kind of grasp it is attempting to perform . The gripper has a measurable max force of 16 N and min force of 0.15 N , a maximum aperture of 105 mm and a minimum aperture of 1 mm . Some grasps may be incomplete , intended for observing force information about a given object . Describe the grasp strategy using the following form :[ start of description ] 5. If you deviate from the default force value , explain your reasoning using the optional bullet points . It is not common to deviate from the default value . 6. Using knowledge of the object and how compliant it is , estimate the spring constant of the object . This can range broadly from 20 N / m for a very soft object to 2000 N / m for a very stiff object . 7. Using knowledge of the object and the grasp description , if the grasp slips , first estimate an appropriate increase to the aperture closure , and then the gripper output force . 8. The increase in gripper output force the maximum value of (0.05 N , or the product of the estimated aperture closure , the spring constant of the object , and a damping constant 0. The coder takes as input the structured summary of the grasp and inferred characteristics and generates an grasp policy which implements the adaptive grasping algorithm 1 and compliance sensing.We have a description of a grippe r 's motion and force sensing and we want you to turn that into the corresponding program with following class functions of the gripper : The gripper has a measurable max force of 16 N and min force of 0.15 N , a maximum aperture of 105 mm and a minimum aperture of 1 mm . ''' def get_aperture ( finger = ' both ') ''' finger : which finger to get the aperture in mm , of , either ' left ' , ' right ' , or ' both '. If ' left ' or ' right ' , returns aperture , or distance , from finger to center . If ' both ' , returns aperture between fingers .''' def set_goal_aperture ( aperture , finger = ' both ' , record_load = False ) ''' aperture : the aperture to set the finger ( s ) to ( in mm ) finger : which finger to set the aperture in mm , of , either ' left ' , ' right ' , or ' both '. record_load : whether to record the load at the goal aperture . If true , will return array of ( pos , load ) tuples This function will move the finger ( s ) to the specified goal aperture , and is used to close and open the gripper . Returns a position -load data array of shape (2 , n ) --> [[ positions ] ,[ loads ]] , average force , and max force after the motion .''' def set_compliance ( margin , flexibility , finger = ' both ') ''' margin : the allowable error between the goal and present position ( in mm ) flexibility : the compliance slope of motor torque ( value 0 -7 , higher is more flexible ) until it reaches the compliance margin finger : which finger to set compliance for , either ' left ' , ' right ' , or ' both ' ''' def set_force ( force , finger = ' both ') ''' force : the maximum force the finger is allowed to apply at contact with an object ( in N ) , ranging from (0. A.8 PhysObjects Finetuning Data Format, CoT Prompt, and CoT Latency Evaluation Each text caption is adapted to a pair-wise relative Q&A conversations across these physical concepts concepts, following the structure: Q: Which {has more mass, is more fragile, is more deformable}, material 1 object 1 or material 2 object 2 ? and A: A material object {has more mass, is more fragile, is more deformable}.When evaluating mass estimates from finetuned and/or CoT-prompted models for atypical objects, we query the model 10 times for the mode of estimates (minimum 5) as mass estimates are more varied. We execute these queries in parallel, which is a little bit slower than querying an LLM just once, as the final requests are rate-limited and slower to complete, taking on average 3.73s (n=30).We add three new mandatory bullet points to the description, starting after the description of whether the grasp contains multiple grasps. These three bullets elicit CoT thinking and leverage PhysObjects finetuning by requesting explicit mass comparisons between heavier, lighter objects and a typical vs. the user-described object to be grasped. We modify rule 3, add a rule for an example object (rule 4) to further enforce CoT and leverage PhysObjects finetuning.Control a robot gripper with force control and contact information . The grippe r 's parameters can be adjusted corresponding to the type of object that it is trying to grasp as well as the kind of grasp it is attempting to perform . The gripper has a measurable max force of 16 N and min force of 0.15 N , a maximum aperture of 105 mm and a minimum aperture of 1 mm . Some grasps may be incomplete , intended for observing force information about a given object . Describe the grasp strategy using the following form : entire phrase with an appropriate example object that is similar to the object to be grasped . 5. Using information from the user instruction about the object and the grasp description , set the initial grasp force either to this default value or an appropriate value . 6. If you deviate from the default force value , explain your reasoning using the optional bullet points . It is not common to deviate from the default value . 7. Using knowledge of the object and how compliant it is , estimate the spring constant of the object . This can range broadly from 20 N / m for a very soft object to 2000 N / m for a very stiff object . 8. Using knowledge of the object and the grasp description , if the grasp slips , first estimate an appropriate increase to the aperture closure , and then the gripper output force . 9. The increase in gripper output force the maximum value of (0.05 N , or the product of the estimated aperture closure , the spring constant of the object , and a damping constant 0.1: ( k * additional_closure *0.0001) ) . 10. Provide the full description of the grasp plan , even if you may only need to change a few lines . Always start the description with [ start of description ] and end it with [ end of description ]. 11. Do not add additional descriptions not shown above . Only use the bullet points given in the template . 12. Make sure to give the full description . Do not skip points if they are not optional . """ 


Decision: Post-Rebuttal Meta Review:
-------------------------------------------
The paper presents an interesting grasping framework. Given that the rebuttal and responses addressed the major concerns of the reviewers, I am happy to recommend the paper for acceptance. We look forward to the discussion and changes being reflected in the camera-ready version. 

Original Meta Review:
-------------------------------------------
This paper proposes DeliGrasp, a grasp controller whose parameter selection is based on semantic information gathered from an LLM.

Strengths: The paper proposes an interesting strategy of integrating common-sense knowledge into an adaptive grasp controller and demonstrates this on a variety of real-world objects. The paper is well-written and presents an entire system in an easy-to-understand way.

Weakness: Given the rise of systems that leverage LLMs to provide common-sense reasoning, the paper could more clearly justify how it differs from previous methods (Reviewer M14h) as well as compare to a wide range of baselines (Reviewer M14h, Reviewer uYAW). It would also strengthen the paper to clarify a few details with respect to the experimental setup (Reviewer JraE), the specification of a few figures and tables (Reviewer JraE, Reviewer uYAW) and the computational expense of the framework (Reviewer M14h).

Reviews: review: title: title: Manuscript review
paper_summary: paper_summary: The authors introduce DeliGrasp, which utilizes LLM's physical reasoning and code generation capabilities to infer object properties from descriptions and create adaptive grasp policies. DeliGrasp tackles the challenge of grasping diverse and fragile objects like paper airplanes and ripe avocados. A complete pipeline is presented, encompassing object localization, description, and initial grasp point estimation. A dual-prompt LLM interaction generates executable grasping code. Furthermore, estimating object compliance while grasping enables answering context-specific queries about the objects for downstream tasks, such as assessing avocado ripeness for food preparation. The authors demonstrate the effectiveness of DaliGrasp by conducting multiple successful grasping experiments on a dataset of 12 diverse delicate objects.
main_review: main_review: The authors present a promising method for utilizing LLMs' physical reasoning and code generation capabilities to infer object properties from descriptions and create grasp policies for grasping delicate objects. While the approach demonstrates clear merit, the evaluation strategy could be strengthened to more rigorously demonstrate the effectiveness of the approach and maximize the impact of this work. 

**Strengths**:
- DeliGrasp demonstrates the potential of Large Language Models (LLMs) for generating grasp policies for delicate objects, an area that has been under-explored in LLM-based robotic manipulation.
- The authors present a detailed pipeline that integrates LLM-driven grasp planning with object perception, offering a practical framework for real-world application and object-related query answering.
- By integrating compliance estimation, DeliGrasp showcases the ability to extract relevant object-specific information during grasping, suggesting potential applications in various domains.
- The paper is generally well-written and easy to follow, providing a comprehensive overview of the approach and its potential contributions.


**Weakness**:
- Table 2 requires further clarification. For instance, why does the estimated mass vary for different approaches on the same object (e.g., Object IDs 8 -11) while the reported error remains consistent? A deeper explanation of these discrepancies would strengthen the analysis.
- To better assess the accuracy and limitations of the approach, including LLM-generated values for μ and k, along with ground truth mass values for objects in Table 2, would be beneficial.
- Providing examples of actual prompts and queries used with the LLM would enhance reproducibility and offer a clearer understanding of the method's implementation details.
- Figures 2 and 4 should be improved in terms of resolution and clarity to better support the paper's overall presentation.


**Other feedback**:
- Sort citations in the Related Work section 
- Table 1 does not have a “grasp verb” column, though it is discussed in the text.
- Add descriptive captions to Tables 1, 2, and 3, where all symbols, columns, and rows are explained clearly. Briefly describe the salient points and main takeaways of the results/
- Consider bringing the text in lines 135-139 earlier in section 3 around lines 94-99, as it will provide a user a clearer picture of the complete pipeline.
- Please rephrase the sentences in lines 110-114 for better readability.
questions: questions: - Please explain why the estimated mass varies for different approaches on the same object (e.g., Object ID 11) in Table 2 while the reported error remains consistent. 
- How is the Columbian friction model accounted for while generating LLM responses?
- What are the actual prompts used to generate the description and code?
limitations: limitations: 3
review_summary: review_summary: The authors present a compelling approach for leveraging LLMs in the grasping of delicate objects, showcasing the potential of this methodology. However, the current evaluation strategy raises questions about the generalizability and robustness of their findings. I recommend accepting this paper only if the authors provide additional experimental analysis to address these concerns. Otherwise, I would suggest rejection due to insufficient evidence of the approach's efficacy.
title: title: Review
paper_summary: paper_summary: The paper explores the application of large language models (LLMs) to enhance robotic grasping capabilities by leveraging their ability to reason about physical properties and generate code. It describes a novel approach where LLMs are used to infer the mass, friction coefficient, and spring constant of objects from their semantic descriptions. These physical characteristics are then translated into executable, adaptive grasp policies. The study tests these LLM-parameterized grasp policies against traditional adaptive grasp policies and direct LLM-as-code policies using a custom benchmark comprising 12 delicate and deformable items, such as food, produce, and toys, with a wide range of mass and required pick-up force. Results show that LLM-parameterized policies outperform the alternatives. Further improvements in property estimation and grasp performance are achieved through model finetuning on property-based comparisons and the use of chain-of-thought prompting to elicit these comparisons.
main_review: main_review: ### Strengths:
* The proposed method is thoroughly evaluated on a wide range of objects, showcasing its versatility across different properties.
* The methodology is straightforward and easy to implement.

### Weaknesses:
* Novelty: The core features of the study, specifically the use of LLMs for common sense reasoning and code writing, do not introduce new knowledge to the field of robotics. Previous works have already demonstrated these aspects, suggesting that the current study’s contributions may be incremental. While the integration of these features into robotic grasping is noted, the application itself does not substantially advance our understanding or implementation of LLMs in robotics.
* Evaluation: The absence of a comparison with established classical adaptive grasping methods is a significant oversight. For a comprehensive evaluation of the proposed DeliGrasp method, it is essential to benchmark its performance against existing techniques. This comparison would provide critical insights into the advantages offered by LLMs in this context.

Misc:
* The poor image quality in Figure 4 hampers clarity and understanding of the content.
* Incorrect formatting of double quotes in many places (e.g., Table 1).
questions: questions: * Considering that DeliGrasp requires querying LLMs ten times to derive simple quantities, what is the total computation time for executing this process? Detailed information on computational efficiency is essential for assessing the practicality of integrating LLMs into real-time robotic operations.
* What specific improvements are observed from finetuning the LLMs on the PhysObjects dataset? Additionally, why is the GPT-3.5-Turbo model without finetuning not included in the study?
* Does the PhysObjects dataset used for finetuning the LLM include any of the objects later tested in the evaluation phase?
limitations: limitations: 2
review_summary: review_summary: While the approach is promising and the evaluation demonstrates potential advantages, the incremental nature of the contributions and the lack of crucial comparative analysis weaken the paper’s impact. If these issues are addressed, particularly by enhancing the novelty and depth of evaluation, the paper could be a strong candidate for acceptance in future submissions. A weak reject is recommended at the current state.
title: title: Review of paper 523
paper_summary: paper_summary: The authors propose to use an LLM to predict the mass and friction of an object from a high-level object description. These values parameterize an adaptive grasp controller to pick up the object. The authors further vet the potential of LLM-generated parameters by fine-tuning on a dataset of captioned object images.
main_review: main_review: Strengths:
- Unlike prior work, DeliGrasp leverages the generalization abilities of LLMs without compromising the robustness of a classical adaptive grasp controller. 
- The authors demonstrate that this approach can further improve with fine-tuning
- An open question in robotic manipulation is how best to integrate large language models and this paper presents a timely proof-of-concept

Weaknesses:
- Is there a stronger baseline that force limited? E.g., is there a way to develop a compliant grasp as a baseline?
- The clarity of the paper could be improved:
    - Methods section is too detailed. E.g., the paragraph starting on line 94 could be moved to the appendix. 
    - It's difficult to understand the purpose of table 2. I would recommend moving it to the appendix, summarizing over object, or adding more analysis of the table to the text. Listing the object id instead of the object name only makes it more difficult to understand the table.
    - Section 4.2 comes as a bit of a surprise. It doesn't advance the main point (that surprisingly LLMs are quite capable of estimating manipulation relevant object properties), so I suggest moving to appendix.
    - It's unclear where the fine-tuned vs not fine-tuned models are used in the experimental section. It would be useful to see a direct comparison between these two variants.
questions: questions: - How is success scored in Figure 3? Are the grasps manually labeled as causing damage vs not causing damage?
- Are the deli grasp results in section 4 all using the fine-tuned model?
limitations: limitations: 3
review_summary: review_summary: Overall I recommend acceptance because I think the analysis is interesting, but I think it's important that the authors improve the clarity in the rebuttal period.