DeliGrasp: Inferring Object Properties with LLMs for Adaptive Grasp Policies
Authors: William Xie
Maria Valentini
Jensen Lavering
Nikolaus Correll
Venue: CoRL 2024 (2024)
Keywords: contact-rich manipulation
adaptive grasping
force control
produce manipulation
TL;DR: We prompt GPT-4 to infer object mass, friction, and compliance terms and to translate them into grasp policies that outperform traditional grasps on delicate objects; we also improve property estimation and grasping for atypical objects

Abstract: Large language models (LLMs) can provide rich physical descriptions of most worldly objects, allowing robots to achieve more informed and capable grasping. We leverage LLMs' common sense physical reasoning and code-writing abilities to infer an object's physical characteristics-mass $m$, friction coefficient $\mu$, and spring constant $k$-from a semantic description, and then translate those characteristics into an executable adaptive grasp policy. Using a two-finger gripper with a built-in depth camera that can control its torque by limiting motor current, we demonstrate that LLM-parameterized but first-principles grasp policies outperform both traditional adaptive grasp policies and direct LLM-as-code policies on a custom benchmark of 12 delicate and deformable items including food, produce, toys, and other everyday items, spanning two orders of magnitude in mass and required pick-up force. We then improve property estimation and grasp performance on variable size objects with model finetuning on property-based comparisons and eliciting such comparisons via chain-of-thought prompting. We also demonstrate how compliance feedback from DeliGrasp policies can aid in downstream tasks such as measuring produce ripeness. Our code and videos are available at: https://deligrasp.github.io

DeliGrasp: Inferring Object Properties with LLMs for Adaptive Grasp Policies
William Xie, Maria Valentini, Jensen Lavering, Nikolaus Correll"
Abstract: Large language models (LLMs) can provide rich physical descriptions of most worldly objects, allowing robots to achieve more informed and capable grasping. We leverage LLMs' common sense physical reasoning and code- code-writing abilities to infer an object's physical characteristie- mass friction coefficient and spring constant k-from a semantic description, and then translate those characteristics into an executable adaptive grasp policy. Using a two-finger grip- per with a built-in depth camera that can control torque by limiting motor cur- rent, we demonstrate that LaM-paaraeterrized but first-pricciples grasp policies outperform both traditional adaptive grasp policies and direct LLM-aa-code poli- cies custom benchmark of 12 delicate and deformable items including food, produce, toys, and other everyday items, spanning two orders of magnitude in mass and required pick-up force. We then improve property estimation and grasp performance on variable size objects with model finetuning on property-based comparisons and cliciting such comparisons via chain-of-thought prompting. We also demonstrate how compliance feedback from DeliGrasp policies can aid downstream tasks such as measuring produce ripeness. Our code and videos are available at: https://dele: https://deeigrrasp. .ei.ggithhuut github.io/
Introduction
Large language models (LLMs) are able to supervise robot control and learning in manipulation
from high-level step-by-step task planning [1, 2, 3] to low-level motion planning [4, 5J. LLMs additionally aid in robot manipulation via understanding a given object's semantic properties and delineating appropriate grasp positions conditioned on those semantic affordances [6. 7, 8).
These works usually assume that the acts of "picking" and "placing" are straightforward tasks. This is not the case for contact-rich manipulation, in which LLM-supervised methods [9, 10, 11] still do not account for force-adaptive tasks like grasping a paper airplane or ripe fruits and vegetables and are prone to damaging such objects. Algorithmic methods for grasping delicate objects [12, 13, 14] require custom hardware and are tested on a limited set of items. LLMs provide an opportunity to leverage their common-sense physical reasoning [15] to produce grasp skills which are both force- adaptive and for the open-world. This is particularly important for semi-structured environments like supermarkets that subject to constantly rotating stock dealing with loose food items such fruits, vegetables, and pastries that come in large variety of changing shapes. Yet, LLMs trained on public data are unlikely to contain information every possible object, in particular when considering specialty domains such as warchouse picking [16] or industrial assembly [17] that often include bespoke parts.
We propose DeliGrasp, an extension of LLM-supervised robot learning contact-rich manipula- tion. We posit that LLMs can infer the physical characteristics of gripper-object interactions, in- cluding mass, spring constant, and friction. We then formulate adaptive grasp controller with slip detection derived from the inferred characteristics, endowing LLMs embodied with any force- controllable gripper with adaptive grasp skills for objects spanning a range of weight, size, fragility, and compliance. We also compare DeliGrasp against traditional adaptive grasp algorithms, as well
All authors are with the University of Colorado Boulder, Boulder, CO. Corresponding email: vixi64544colorrdd.cco
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

DeliGrasp
Non-deforming v
Traditional Grasp x
"check the avocados for ripeness and grab an avocado for guacamole today"
LLM Grasp Descriptor
This
m: 280g
sright redium J 0.5
low
compltance avorade R: 500
................................
check
LLM Grasp Coder "poir"
Object Grasp
Fetn:0.55
Controller
33.9
"pick upok a paper alplanne
AF :0.1N
grasping AF 10.1N
Width
Áx :1m
corpliance feedback Ax :2mm
detection and
Sespenttaiion
Compliance Feedback
R5:65.IM4 1h/8 N/11/1..i/m A,
LLM: "A evocaco like would
goed guacamole today"
"pink up rapporery
Pose Estimetion
ppcck up Paaedt tacoo
Figure 1: Large language models (LLMs) have rich physical knowledge about worldly objects, but cannot directly reason robot grasps for them. Paired with open- open-world localization and pose estimation (left), our method (middle), queries LLMs for the salient physical characteristics of mass, friction, and compliance as the basis for adaptive grasp controller. DeliGrasp policies successfully grasp delicate and deformable objects (right). These policies also produce compliance feedback measured spring constants, which leverage for downstream tasks like picking ripe produce (middle).
demonstrate how object knowledge can be augmented by finetuning the LLM using property- based comparisons.
We pair DeliGrasp with an open-world localization pipeline which, given an "object description," identifies the object and an initial grasp position. The same "object description" and associated "grasp verb" are the inputs to DeliGrasp, which produces executable Python code controlling a gripper's compliance, force, and aperture complete grasp policy for the given description.
By conducting robotic grasping experiments on 12 different objects, we show that DeliGrasp per- forms successful, thnimally-damagiine grasps on a custom benchmark of delicate objects which traditional adaptive grasping methods are not capable of. We then improve property estimation and grasp performance using the PhysObjects dataset [11] as well as by eliciting explicit comparisons with chain-of-thought prompting (CoT) [18].
Related Work
LLMs, equipped with internet-scale amount of common sense information and logical and phys- ical reasoning [18], able to comprehend high-level task knowledge, physical context, and robot affordances. For task and motion planning. LLMs generate navigation and pick-and-place in- structions to complete complex and novel tasks [1]. LLM code-writing further augments robot capabilities with closed-loop control [2], new skill generation [3], and translating language to robot parameters for low-level dynamic control [4]. For canaaticalll-affoodee grasping, LLMs, in con- junction with other learning methods [8, 19], can identify appropriate grasp locations [6, 7, 20].
For contact-rich manipulation, LLMs have been finetuned [9, 21] to learn relative object prop- erties such as mass or fragility, but are extensible to low-level control [11]. They have also been prompted with environmental and object properties [10] to parameterize force constraints for robot motion, but not gripper-object interaction. No methods, however, address both the semantics and dynamics requisite for delicate grasping of a broad variety of objects. In comparison, our method utilizes L.MSs to estimate object physical properties, which such models are more abundantly pre- trained on than code, reward functions, or other niche robot control domains. These estimates then paired with underlying algorithmic grasp controller. LLM property estimation, approx- imation of privileged expert knowledge, provides two advantages: simplification of our underlying

---

adaptive grasp controller relative to classical methods, which rely on in in-motion slip detection for adaptive grasping and clarity of the low-level controller such that it is interpretable and predictable.
Grasping delicate objects can be achieved with hardware, such as relying on the compliance of soft grippers, but these end-effectors still require semantic information for control if these grippers must grasp both lightweight and heavy objects [22]. In this paper, we adopt control algorithm that relies on interaction force measurements from rigid gripper similar to [12, 13, 23, 24] for minimally de- forming grasps and for measuring spring constants [25]. Where these adaptive grasping methods are hardware-speciffe ours can be adapted to any force-controllable end-effector, and I.LM-estimated object properties reduce controller complexity by approximating privileged information.
3 Methods
We source delicate objects for our dataset primarily from supermarkets, kitchens, and food pantries, shown in Figure 2c and describe object mass and object-specific thresholds for unsuccessful, "de- forming" grasps in Table I, shown in Figure 2a. For objects with empty entries in the "Input Phrase" column, we do not modify or add descriptions beyond the name of the object.
Table I: Delicate and Deformable Object Properties
ID
Object
Width Mass Fmin (N)
"Object Description" Input to LLM & VLM
Invalidating Deformation
(mm)
(g)
Paper Airplane
20
0.8
0.02
crumples crumples, creased cracks, shatters juices, torn cracks, broken inelastic deform inclastic deform spillage cracks
2
Cup (empty)
75
3.6
0.11
"empty paper cup"
Dried Yuba
30
5.5
0.16
"yuba (dried tofu skin)"
4
Raspberry
20
6
0.18
Hard Taco
65
15
0.44
"hard-shell tortilla"
6
Mandarin
50
56
1.65
7
Stuffed Toy
28
74
2.18
"tail of a stuffed animal"
8
Cup (water)
75
106
3.12
"paper cup filled with water"
9
Bag (noodles)
90
191
5.62
"bag containing noodles"
10
Avocado
60
204
6.00
inelastic deform spillage
11
Spray Bottle
50
250
7.36
"bottle filled with water"
12
Bag (rice)
80
900
26.49
"bag of rice""
N/A
The evaluated objects span from 0.8 to 900g and soft produce to rigid plastic, and they are commonly grasped in real-world environments. We compute approximate minimum grasping force, Fmin from measured properties. "Object Description" inputs are paired with grasp verb, "pick," to DeliGrasp prompts to generate policies. We also qualify what kind of damage "invalidating deformation" renders grasp failure.
Our pipeline takes as input an "object description", described in Table 1, and "grasp verb," " which for the delicate object dataset evaluation is uniformly "pick." Our perception method, adapted from [6]. takes the "object description" and semantically segments the object from RGB-D D image with OWL-vVT [26] and "Segment Anything" [27]. We crop the corresponding depth image with the generated mask, produce a point cloud object representing the segmented object, and perform Principal Component Analysis to compute a grasp pose that is aligned with the first three principal axes as well as a minimum object grasp width[28], an example is shown Figure 1, left.
We then prompt the LLM to define and generate grasps with dual-prompp structure similar to that of Language to Rewards [4], with initial grasp "descriptor" or "thinker" prompt which produces a structured description, which the subsequent "coder" prompt translates into an executable Python grasp policy that modulates gripper compliance, force, and aperture according to Algorithm I (full prompts in Appendix A.6, A.7). To execute grasp policies, we use a URS robot arm with the open- source MAGPIE gripper [29] and implement current-draw derived force control, from a range of 0.15 N to 16 N (default output force is 4 N). Our experimental setup shown in Fig. 2bA. We manually score grasp failures according to whether the object slipped or if the damage experienced matches the "invalidating deformation" criterion put forward in 1.

---

F Frictions
frictioni
Left Grapper
Right Gripper
Object
Fleft
F Espringi
F Fapring
F Fright
astip
(a) Experimental setup
(b) Free body diagram of gripper-object interaction
peper airplane
paper cup
yuba
raspberry
taco
mandarin
stuffed animal
water
noodle bag
avccado
squeeze bottle
rice bag
(c) Delicate objects dataset
Figure A. Our experimental setup with tabletop URS robot equipped with the MAGPIE Gripper [29] Free body diagram describing gripper interactions an object at rest, adapted from [12] C. The delicate objects dataset ranging from 2-900g and various material properties.
3.1 Grasp Force Modeling
Fig. 2b shows the interaction between our gripper and an arbitrary object with mass m and spring constant (compliance) k. The gripper exerts a composite applied gripper force Fa Fiftt + Fright that leads to a frictional force Fy = μFa. where μ is the Coulomb friction coefficient between the gripper and object [28] that counteracts the force of gravity mg (g is the gravitational constant). For compliant objects, approximated as ideal springs, we can additionally describe the left and right gripper forces Fi, Espring..- kst,r, where is the compression of the grasped object.
Typically, object slip within a gripper detected when after the gripper grasps an object at rest, the gripper begins some upward acceleration Atsft. and an object begins slipping with some downward acceleration aslip- Increasing F, account for mouft yields adaptive minimum applied grasp force Fmin which prevents slip and is minimally deforming [23]: F Fmin m(g+aur)) Conversely,
when object is slipping with Ostsps the applied force Faatsp p [12].
When au st,lipp 0 and the gripper and object are static equilibrium, Fmin mg Fmin then be arranged in relation to these quantities, where m μx are LLM-estimated terms approximating ground truth measurements.:
mg
m g
F Fasis
Fmin < LLMM
(1)
μ*
For the delicate objects dataset, we estimate a minimum applied grasping force Fmin mg with conservative, uniform Coulomb friction coefficient of = 0.33, used to compute Fmin in Table 1.
3.2 Delicate Grasping
Mass and friction, and μ., which determine successful grasp forces, well spring constant i form the basis of our force-adaptive algorithm for non-slipping, minimally deforming grasping of

---

delicate objects. We define closed-loop force controller: starting from an estimated target aperture that corresponds to the estimated object's width w from RGB-D data, we increase the gripper output force Fout and decrease gripper aperture æ until sensing a contact force Fe greater than the target Fmin [24]. To determine the gain terms, A and Ax. i.e. how fast we close the gripper and ratchet up force, the controller uses the agent-determined k and Az; we change Fout by c kAr, where 0.1 is dampening constant. We describe the controller in Algorithm 1.
Algorithm Adaptive Grasping for Minimal Deformation
Fe SetGripper(r = w. 0)
while Fe F min do Fout += c. kAr A
Fe = SetGripper(as Fout)
We query an LLM (GPT-4) for these quantities once to formulate grasp policies. Accurate predic- tions remove the need for parameter tuning [24] and additional gripper sensors [12, 14]. By default, we instruct the LLM to compute P'mn- и to object slip than deformation. However, we also provide the LLM with agency to deviate from the default F Fiin depending on the "grasp verb" provided.
and we do not account for Wift, choosing to err closer
On average (n=30), generating one DeliGrasp policy with GPT-4 takes 9.98s (3.03s for 1x 'Thinker' prompt query, and 6.65s for 1x 'Coder' prompt query). DeliGrasp policies take on average 4.1 to execute (Appendix A.1). Total grasp time sums to, average, 14.11s, compared to approximately 0.5s for force-limited grasp, or 1.7s for classical adaptive grasp [13].
3.3 Improving Property Estimation
To demonstrate how LLM-knowledge can be improved and potentially include bespoke objects, we finetune G on 6000 captions of PhysObjects images [11], each describing two objects, their materials, and their relative fragility, deformability, and mass. The dataset captures 276 unique common household objects, with which the delicate objects dataset shares only the mandarin and plastic bottle. We also augment the grasp "descriptor" or "thinker" prompt with CoT prompting [18], paired with finetuned and non-finetuned models, elicit series of quantitative comparisons of mass: objects that are 1) lighter and 2) heavier, the mass of 3) typical object of the category being grasped, and the mass of the 4) user-described object relative to the typical object, thereby producing explicit bounds on the object mass (see Appendix A.8 for greater details on PhysObjects finetuning data format, the full CoT prompt, and CoT query latency).
3.4 Classical Adaptive Grasping Baselines
We evaluate DeliGrasp against four classical adaptive grasping baselines, two for each strategy: "In Place" and "In Motion." "In Place" grasping closes the gripper around an object rest until measured contact force of 2N 1ON, adapted from [24]. The "In Motion" strategy, adapted from [12, 13], closes around an object at rest until contact force of 1.5N or 0.5N and then moves upward at approximately 10 mm/s for 5 seconds. During this upward motion, the "In Motion" strategy performs closed-loop adaptive grasping at approximately 80Hz, increasing applied gripper force and velocity proportional to measured slip force. We describe the basclines implementation and differences from DeliGrasp in Appendix A.3.
Experiments
We benchmark DeliGrasp (DG) against five grasp policies: in-place adaptive grasping with 2N and ION grasp force thresholds, in-motion adaptive grasping with 0.5N and 1.5N initial force, closing the gripper fully or until it is output force limited (F.L.), closing the gripper to the visual width of the object determined by our perception method, and an ablated DeliGrasp- Direct policy (DG-D) which directly generates Fmin. A Foutt and Ar without first reasoning about an object's physical properties. We also compare DeliGrasp with PhysObjects finetuning (DG-FT) and with finetuning and chain-of-tee -f-thuhtt prompting (DG-FT-COT), selecting the model configurations with the highest reduction in mass overestimation (Fig. 5).
5

---

We employ each grasp policy 10 times with objects placed randomly within 30 x 45 cm bounding area on a table, and do not record attempts which receive faulty poses from perception. Quantitative results are shown Figure 2. Figure shows actual grasps of an empty paper cup and the same paper cup filled with water as well as the LLM-generated code.
Table 2: Successful Minimally Deforming Grasps on Delicate Deformable Objects (10 trials per object)
ID Object
DG DG-FT DG-FT DG-D In Place In Place In Motion In Motion Visual F.L.
CoT
ION
2N
1.5
0.5N
Paper Airplane
10
10
10
0
2 Cup (empty)
10
10
10
10
0
5
3
10
10
0
Dried Yuba
9
10
8
7
0
6
10
0
4 Raspberry
10
10
0
0
O
0
Hard Taco
9
6
7
7
0
7
6
10
0
Mandarin
10
10
10
10
10
10
10
10
10
10
7 Stuffed Toy
7
6
7
8
10
10
10
10
0
10
Cup (water)
10
10
10
4
9 Bag (noodles)
7
4
5
8
9
0
5
10 Avocado
10
8
0
11 Spray Bottle
6
6
5
5
2
3
0
3
Bag (rice)
0
0
2
Success (%)
80.0
76.7
75.0
70.0
31.7
32.5
50.8
55.0
29.2
28.3
Number of successful, non-damaging grasps across grasping strategies (columns) on the delicate object evalu- ation set (rows). First, we test four DeliGrasp variants, where FT corresponds to PhysObjects finetuning, and CoT to physical reasoning prompting. DG-D uses an ablated approach that skips the first-principle model and estimates algorithmic parameters directly. The four variants comparatively similar in performance, though DG-D deforms objects slightly more than others due higher estimated grasping force. We compare two classical adaptive grasping strategies: In Place adaptive grasping until a threshold contact force of 2N or 10N and In Motion adaptive grasping with an initial contact force of 1.5 or 1ON and then increasing applied force proportional to measured slip. These classical methods not able to generalize to the full delicate objects dataset and require expert hand-tuning of controller parameters. We include naive baselines: Visual, which closes the gripper to the estimated object width and F.L., which closes gripper to maximal force, 16N.
The base DeliGrasp model (80%), finetuned model (76.7%), finetumed model with CoT prompting (75.0%), and ablated DeliGrasp Direct prompt (70.0%) all perform successful, grasps that the other baselines are not capable of on objects like the paper airplane and raspberry. DeliGrasp dominates hardware-limited (28.3%) and vision-only grasps (29.2%) in 8 out of 12 ob- jects, and is better or par 10 out of 12 objects. The "In Place" strategy completes 32.5% (2N) and 31.7% (1ON) of grasps successfully. The "In Motion" strategy completes 55.0% (0.5N) and 50.8% (1.5N) of grasps successfully, Per-item performance is shown in Table and full DeliGrasp policy outputs are provided in Appendix A.1.
We look closely failure modes to better understand the performance of each policy. Given two bounds of failure- slip grasps and deforming grasps, DeliGrasp and its FT and CoT variants per- form the same better than DeliGrasp Direct in 10 out of 11 cases. These variants overestimate Fmin terms (individual estimates in Table 7) but primarily perform slip failures. However, for all objects DG-D estimates even higher Fmin values and deforms objects at a higher rate. DeliGrasp overestimates Fmin by a factor of 1.95, whereas DG-D, which performs additional invalidating deformations, overestimates F Frin by a factor of 2.55. Since mass, common quantity, is more ac- curately estimated by an LLM, resulting adaptive grasp policies parameterized from mass are more robust than policies directly parameterized by an LLM with more erroneous predictions. All policies slip at high rates the stuffed animal (grasped by the tail), spray bottle, bag of noodles, and bag of rice. These objects are non- con-linearly and/or very compliant and high volume, exerting lever forces on the gripper that likely exceed Fman-
DeliGrasp on average overestimates object mass by factor of 2.5, and underestimates mass only for one object (Appendix A. Figure 5 shows that mass overestimation chain-of of-thuught reasoning (reduction to factor of 1.82), where the LLM is prompted to explicitly relate its mass estimate to that of other objects, or by fine-tuning GPT-3.5 on the PhysObjects dataset
be reduced by employing

---

(reduction to factor of 2.30), combination of both methods (reduction to factor of 1.35), with per- item parameter estimate analysis in Appendix A.2. Though the finetuned model and finetuned model with CoT prompting, DG FT and DG FT COT, performed similarly successful grasps to the base DeliGrasp model and prompt, they produced more accurate, but lower mass, and thus estimates for the bag of noodles and hard-shelled taco, explaining why resulting policies in have more slip failures. In addition to leading more faithful force estimates, this also creates opportunity to extend DeliGrasp to changing inventories or bespoke items in an industrial application.
Model Type
DG 4 DG 3.5|DG CoT 3.5 DG 3. FT CoT 3.5 CoT 4
Mass Overestimation Factor 2.52 1.82
1.51
2.30
1.35
1.82
(a) Ratio of mass overestimation ratio for base Model, PhysOhject-fnetume finetunnd models, and models with addi- tional thain-of-though prompting
Query: pick up the empty paper cup Estimatee properties:
Query: pick up peper cup filled water Estimated properties: m: 250g
10g
μ:: 0.4
μu: 0.6
k: 50.0 N/m
k: 200.0 N/m Controller perameters:
Controller parameters:
Frre 0.025
Firee Au:: 2.0
Ax:: 5.0
Af: 0.025
C
Af: 0.4
Figure (A) We compare mass estimates (row) across different LLMs and prompting strategies (columns): denoting that 4/3.5 eprresponds to GPT-4 and 3.5-Turbo, FT to PhysObjects Finetuning, and CoT to physical reasoning prompting. We observe that both finetuning and CoT prompting improve mass estimates, and that the methods together yield the most improved estimates. We show how semantic modifiers like "empty paper cup" (B) vs. "paper cup filled with water" (C) result drastic differences on weight and other parameter estimates.
For non-DeliGrasp methods, we observe that qualitatively, the different "In Place" strategies moderately successful opposite ends of the evaluation The 2N strategy still performs inval- idating deforrattin the paper plane, empty cup, yuba, raspberry and is not forceful enough for the heavier objects. The 1ON strategy deforms lighter objects and delicate heavier objects like the cup of water and the avocado.
The "In Motion" strategies are more performant, but we observe the same performance trends be- tween the 0.5N (more successful on lighter objects) and I.5N strategies (more successful heavier objects). We observe that both 0.5N and 1.5N strategies perform slip failures small and dense ob- jects, such as the avocado squeeze bottle, due to low surface area of purchase and large measured slip force preventing the controller from adapting in time. Inversely, the controllers are relatively aggressive on lighter or otherwise delicate objects, leading to deformation on the paper plane, wa- ter cup, and avocado. Compared to DeliGrasp. LLM common sense reasoning enables dynamic grasping range object mass and stiffness.
We note that both classical adaptive grasping methods crush the raspberry. Berry-like objects low-density, soft, and rough, requiring precise and "delicate" grasping force. The 0.5N "In Motion" strategy applies a low initial force with slight deformation but crushes the raspberry during closed- loop control due to its aggressive gain terms. Categorically, force-limited grasps deform and vision- only grasps slip. Force-limited grasps succeed only with robust, compliant objects. Vision-only grasps succeed when the objects are relatively stiff and are sensitive to grasp pose.
4.1 Grasping Atypical Objects
Since the baseline DeliGrasp prompt policies are performant with overestimated mass, we record grasp performance by querying complex and atypical objects, such as a "larger avocado" (307 g). a wet sponge (92.0 g). crochet yarn lily flower (26.5 g), and a 10in length of old-growth 2x4 (925 g), and record 10 grasps per configuration (Table 3). We compare across different DeliGrasp configurations the original DeliGrasp prompt with GPT-4-Turbo, the original prompt with the fine- tuned GPT-3.5-Tuss model, the CoT prompt with GPT-4-Turbo, and the CoT prompt with the fine-
7

---

Table Model comparison of mass estimates and grasping success a typical (143g) and large (306g) avocado
Model
DG
DG
DG
DG
FT CoT
FT CoT
Larger Avocado
(g)
200
200
250
300
Sucoess
20% 10%
80%
100%
Wer Sponge
50
200
15
Success
0%
30%
0%
90%
Crochet Yarn Flower
29
Success
0%
0%
0%
100%
Old Growth 2x4
450
450
780
1000
Success
0%
0%
100%
100%
We compare grasp performance from four DeliGrasp configurations, the original DeliGrasp prompt with GPT- 4-Turbo (DG), the original prompt with the finetuned GPT-3.5-Turbo model (DG FT), the CoT prompt with GPT-4-Turbo (DG CoT), and the CoT prompt with finetuned model (DG COT), on atypical objects such as large avocao, a wet sponge, a crochet yarn flower, and old growth 2x4 lumber. Finetuning, paired with CoT prompting, enables semantically appropriate mass estimates for complex and/or atypical objects.
tuned model. We observe that pairing the fine-tuned model with CoT prompting fully leveraged PhysObjects-like physical reasoning and produced successful grasp policies from an initial descrip- tive query, due to more accurate initial mass estimates.
The finetuned model with CoT prompting able to grasp each object, due more accurate mass estimates resulting from more correct CoT-prompted bounds on object mass. The other methods cause majority slip failures for the larger avocado, the crochet flower, and lumber due to an un- changed mass estimate. For the wet sponge, the other models both overestimate its mass and squeeze the sponge too tightly, or underestimate its mass. The non-finetuned models do not acknowledge the semantic relationship between the provided modifiers ("large," "wet," "crochet yarn," or "old growth") on the mass of a typical object until the user re-queries the same context with the ex- plicit relationship semanties, such as "the avocado is larger than average, try again." In Appendix A.4, we show how DeliGrasp can also generate atypical grasps conditioned the verbiage provided (e.g. generating less forceful grasp for "checking" versus "picking").
5 Conclusion
We introduce DeliGrasp, which uses LLMs to 1) infer semantic, common sense physical information which 2) parameterizes adaptive grasp policies effective on a variety of delicate and deformable items that traditional adaptive grasping methods cannot grasp. These inferred m, μ. characteristics enable more consistent control than direct estimation of grasping forces (LL-Maas-code) and classical adaptive grasping methods. We also improve property estimation accuracy and grasp performance on atypical and complex objects with model finetuning object property comparisons and chain- of-thought prompting. We then show how DeliGrasp's compliance feedback can be used to measure produce ripeness.
Limitations and Future Work. DeliGrasp policies are performant, but dependent on stable and accurate property estimates. For each LLM-estimated quantity in this work, we select the mode of 10 estimates, which be done offline and reused for simple objects, but not for semantic modi- fiers complex and abstract objects. Furthermore, do not systematically explore the variance of each quantity, policy robustness to properties besides mass. One interesting line of future work is leveraging iterative observations by mobile robot systems such as humanoids, rather than additional learning methods, to improve system estimates of such properties. We demonstrate this already, as DeliGrasp policies empirically measure and object width, superseding less-accurate LLM estimates and perception-basee width measurements. We anticipate that perception and physical in- teractions can produce higher fidelity information inform manipulation of unknown objects, akin to how humans explore dense, cluttered, and unstructured settings like supermarkets, home kitchens, and other such representative domains. Though finetuning on PhysOhjects did not improve grasp performance on our delicate objects dataset, finetuning on this kind of higher fidelity information about complex or out-of out-of-distribution entities may improve grasp performance.
8

---

References
[1] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho. J. Hsu, J. Ibarz, B. Ichter. A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, Levine, Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, Rettinghouse, D. Reyes, Sermanet, N. Sievers,
Tan, Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, M. Yan. and Zeng. Do as can, Not As I Say: Grounding language robotic affordances. Sttv:2204.0169/ 2022.
[2] J. Liang W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 9493-9500, 2023. doi: AL119/1CR448891.223... 10160091.
[3] M. G. Arenas, T. Xiao, S. Singh, V. Jain, A. 7. Ren, Q. Vuong, J. Varley, A. Herzog, I. Leal, S. Kirmani, D. Sadigh, V. Sindhwani, K. Rao, J. Liang, and A. Zeng. How to prompt your robot: A promptbook for manipulation skills with code as policies. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023. URL https: //openreview wtapt//orrmnnddTA11220
[4] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, brian ichter, T. Xiao, P. Xu, A. Zeng, T. Zhang, N. Heess, D. Sadigh, J. Tan, Y. Tassa, and F. Xia. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023. URL https: http:///ppeeeeeee/////ee// net/forum?id= SgTPAyehXMA.
[5] J. Liang, F. Xia, W. Yu, A. Zeng, M. G. Arenas, M. Attarian, M. Bauza, M. Bennice, A. Bewley, A. Dostmohamed, C. K. Fu, N. Gileadi, M. Giustina, K. Gopalakrishnan, L. Hasenelever, J. Humplik, J. Hsu, N. Joshi, B. Jyenis, C. Kew, S. Kirmani, T.WW. F. Lee, K.-H. Lee, A. H. Michaely, J. Moore, K. Oslund, D. Rao, A. Ren, B. Tabanpour, Q. Vuong, A. Wahid, T. Xiao, Y. Xu, Zhuang, P. Xu, E. Frey, K. Caluwaerts, Zhang, B. Ichter, J. Tompson, L. Takayama, V. Vanhoucke, I. Shafran, M. Matarie, D. Sadigh, N. Heess, K. Rao, N. Stewart, J. Tan, and
C. Parada. Learning to learn faster from human feedback with language model predictive arXiv:2402. 14500 2024.
control.
[6] R. Mirjalili, M. Krawez, S. Silenzi, Y. Blei, and W. Burgard. Lan-grasp: Using large language models for semantic object grasping. urXiv:2310.05239, 2023.
[7] S. Jin, J. Xu, Y. Lei, and L. Zhang. Reasoning grasping via multimodal large language model. arXiv:2402.06998 2024.
[8] A. Rashid, S. Sharma, C. M. Kim, J. Kerr, L. Chen, A. Kanazawa, and K. Goldberg. Lan- guage embedded radiance fields for zero-shot oriented grasping. In 7th Annual Conference on Robot Learning. 2023. URL
[9] Y.-L. Wei, J.-J. Jiang, C. Xing. X. Tan, X.-M. Wu, H. Li, M. Cutkosky, and W.-S. Zheng. Grasp as you say: Language-guider dexterous grasp generation, 2024.
[10] K. Burns, A. Jain, K. Go, F. Xia, M. Stark, S. Schaal, and Hausman. Genehip: Generating robot policy code for high-precision and contact-rich manipulation tasks, 2024.
[11] J. Gao, B. Sarkar, F. Xia, T. Xiao, J. Wu, B. Ichter, A. Majumdar, and D. Sadigh. Physically grounded languagee models for robotic manipulation, 2024.
[12] Z. Ding, N. Paperno, K. Prakash, and A. Behal. An adaptive control-based approach for 1-click gripping of novel objects using robotic manipulator. IEEE Transactions on Control Systems Technology,
2019. 010:10.1110/TSS22288.222266

---

[13] M. Al-Mohammed, R. Adem, and A. Behal. A switched adaptive controller for robotic grip- ping of novel objects with minimal force. IEEE Transactions on Control Systems Technology, 31(1):17-26, 2023. 10.:10..109/1111111...1 10T.110000//TTT12222776666
[14] Y. Gong, Y. Xing. J. Wu, and Z. Xiong. pictile-Based Slip Detection Towards Robot Grasping. In H. Yang, H. Liu, J. Zou, Z. Yin, L. Liu, G. Yang, X. Ouyang, and Z. Wang, editors, Intel- ligent Robotics and Applications, pages 93-107, Singapore, 2023. Springer Nature Singapore. ISBN 978-981-9999 018-9899964-9-444
[15] Y. Wang. J. Duan, D. Fox, and S. Srinivasa. NEWTON: Are large language models capable of physical reasoning? In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 7748-9758, Singapore, Dec. 2023. 18111.11665///22222222
Association for Computational I. Linguistics. Betps://aclanthin
http////....iiingeem.......... URL. org/2023. wtt22222.....nng.. findings-emnlp e.....62................... 652.
[16) N. Correll, K. F. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman. Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1):172-188, 2016.
[17] F. Von Drigalski, C. Schlette, M. Rudorfer, N. Correll, C. Triyonoputro, W. Wan, T. Tsuji, and T. Watanabe. Robots assembling machines: learning from the world robot summit 2018 assembly challenge. Advanced Robotics, 34(7-8):408 4):4044221 2020.
[18] Wei, X. Wang, D. Schuurmans, M. Bosma, Ichter, Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thougle prompting elicits reasoning in large language models, 2023.
[19] H.-S. Fang, C. Wang, H. Fang, M. Gou, J. Liu, H. Yan, W. Liu, Xie, and C. Lu. Anygrasp: Robust and efficient grasp perception spatial and temporal domains. IEEE Transactions on Robotics (T-RO), 2023.
[20] C. Tang, D. Huang, W. Ge, W. Liu, and H. Zhang. GraspGPT: Leveraging semantic knowledge from a large language model for task-oriented grasping. arXiv:2307. 13204, 2023.
[21] R. Xu, F. Luo, Z. Zhang, C. Tan, B. Chang, S. Huang, and F. Huang. Raise child in large language model: Towards effective and generalizable fine-tuning. CoRR, abs/2109.05687, 2021. URL /etps://arriit wtggabs//10...0550//.. 05687.
[22] N. Farrow, Y. Li, and N. Correll. Morphological and embedded computation in self-contained soft robotic hand. arXiv preprint arXiv: 1605.00000 2016.
[23] H. Hasegawa, Y. Mizoguchi, K. Tadakuma, A. Ming, M. Ishikawa, and M. Shimojo. De- velopment of intelligent robot hand using proximity, contact and slip sensing. In 2010 IEEE International Conference on Robotics and Automation, pages 777-784, 2010. doi: 10.110/R0800:22005500222
[24] K. Sullivan, H. Chizeck, and A. Marburg. Using a rigid gripper on objects differ- ent compliance underwater. OCEANS 2022, Hampton Roads, pages 1-4, 2022. doi: 10.1 1LL11000EAAN479912222.9999
[25] T. M. Caldwell, D. Coleman, and N. Correll. Optimal parameter identification for discrete mechanical systems with application to flexible object manipulation. In 2014 IEFERSS Inter- national Conference on Intelligent Robots and Systems, pages 898-905. IEEE, 2014.
[26] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby. Simple open-vocabulary object detection with vision transformers. arXiv:2205.066 06230, 2022.
[27] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-.. Lo, P. Dollár, and R. Girshick. Segment anything, 2023.
10

---

[28] N. Correll, B. Hayes, C. Heckman, and A. Roncone. Introduction to autonomous robots: mechanisms, sensors, actuators, and algorithms. MIT Press, 2022.
[29] N. Correll, D. Kriegman, S. Otto, and J. Watson. A versatile robotic hand with 3d perception, force sensing for autonomous manipulation. arXiv:2402.06011 2024.
[30] B. Calli, A. Walsman, Singh, S. Srinivasa, Abbeel, and A. Dollar. Benchmarking in manipulation research: Using the yale-cmu-berkeley object and model IEEE Robotics
& Automation Magazine, 22(3):36-52, Sept. 2015. ISSN 1070-9932. do1:10.1109//ra.2000 2448951. URL http://dz doi org/10. 10. 1109//RA. 2015 2448951.
[31] Hart, A. Moriarty, Pasternak, J. Kummert, A. Hawkin, Hassouna, D. Pena Nar- vaez, Ruegemer, von Seelstrang. P. Van Dooren, J. J. Garcia, Mitzutani, Y. Jiang, T. Matsushima, and R. Polvara. Robocup@home 2024: Rules and regulations. https:
h http:h/ww.....oo..oooo.oooooo/a. 2024.

---

A Appendix
The MAGPIE gripper has a palm-integrated Intel RealSense D-405 camera. The gripper powered by two separate Dynamixel AX-12 motors, which allow maximum current control. Translating torque via four-bar linkage, the gripper force can be controlled and sensed from 0.15 N to 16 N, and has aperture range from 106 nm fully open to 0 nm fully closed (default closure speed of 100
mam.
A.1 Full Details of DeliGrasp Performance Delicate Objects Dataset
Regarding delicate object evaluation, the YCB [30] and RoboCup@r Home [31] datasets include some fragile and deformable objects, but there does not exist a dataset focusing on delicate objects with defined failure modes. By combining (1) LLM estimates object properties to parameterize for robust and damage-free grasping of delicate objects and (2) an algorithmic adaptive grasp con- troller, our method is able to generalize across many objects and produce uniform grasping feedback which can further inform the controller, LLM, or user.
Computed A Fout which are less than 0.01 N are set to 0.01 N. Computed Fmin values below the force sensing threshold of 0.15 N are set to that threshold when checking for slip or setting output force.
Table 4: DeliGrasp Performance on Delicate Objects Dataset (10 trials)
LLM Inferred m, μ, k and Downstream Terms Mean Values of Experimentally Produced Terms
ID Success
m
k
Fmnn (N) Ferr (N) AFout Az 2 final gooo Fout (N) (mm) (mm) (mm) (N)
Time
kmaz
k
(g)
(N/m)
mg
(s) (N/m)
(N/m)
10
5
0.5
20
0.098
0.08
0.01
11.9
7.8
0.54
4.23 332.46
184.91
10
10
0.4
50
0.25
0.14
0.025
67.83
59.5
0.5
2.94 707.93
389.76
9
20
0.5
200
0.39
0.23
0.04
24.74
19.82
0.6
4.08 240.27
157.69
9
5
0.8
50
0.063
0.12
0.01
15.4
5.98
0.28
7.26 97.46
60.96
9
50
0.5
1000
0.98
0.54
0.2
53.17
46.64
1.59
3.19 342.05
188.39
10
150
0.8
500
1.88
0.23
0.1
45.13
39.83
2.12
3.08 236.41
151.83
9
50
0.8 100
0.61
1.57
0.02
9.79
4.19
0.97
5.29 308.73
192.77
9
250
0.6 200
4.08
0.96
0.04
58.23
53.3
4.36
4.27 958.61
373.24
7
500
0.4 300
12.3
6.68
0.15
71.72
61.11
12.77
2.89 7484.7
4324.33
10
9
200
0.5 500
3.92
2.08
0.1
2
50.88
46.4
4.39
3.04 729.11
298.66
11
6
500
0.4 150
12.2
4.84
0.03
41.94
35.08
12.6
3.64 3456.4
1383.11
12
O
1000
0.5 200
19.6
5.9
0.1
5
35
27.5
20.8
5.7 13671.
6192.2
We also show the corresponding values for the ablated DeliGrasp-Direct policy, We did not record time, experimental i values, or final in these experiments.
Table Ablated DeliGrasp Performance on Delicate Objects Dataset (10 trials)
ID Success Fmin AF out
Ar Tgoat Fout (mm) (mm) (N)
(N)
(N)
1
10
0.15
0.05
4.48
0.98
2
0.5
0.2
57.4
0.922
3
7
0.5
0.3
8.6
1.14
4
8
0.2
0.1
6.4
0.83
5
7
1.5
0.2
49.2
2.3
6
1.25
0.5
32.4
2.01
7
8
1.5
0.5
3.0
3.2
8
8
1
49.6
3.18
9
4
1
4
52.8
6
10
7
1
47.2
2.2
11
5
2
25.0
7.1
12
0
2
105.
10
12

---

ID
Object
Mass (g)
a crr
b b crr
ccarr
d
er
ffem
cr
h
er
iiem
j em
poper airplane
0.8
10 11.5
5 6,25
6.25
10
11.5
1 1.25
5 6.25
10 11.5
5 5.25
10 11.5
10
11.50
emppl cup
3.6
1.78
10 2.78
10
2.78
10
1.78
5 1.39
5 1.39
4.56
1.78
4.56
10
1.78
yuha
5.5
2,64
20
3,64
20
3.64
10
0.82
1.82
10 1.82
8.09
20
2.64
8.09
20
2.64
raspberry
6
5 0.17
20 3.33
1.17
0.33
5 1.17
4 1.33
5
2.33
20
2.33
20
2.33
10
0.67
hard shell tortilla
0.33
1,33
50
3.33
15
0.33
0.00
0.33
mandarin
56
150 1.68
150
2,68
150
2668
100
0.79
50
1.11
75
1.34
150
1.68
150
1.68
150
1.68
150
1.68
stuffod animal
0.32
50
1.32
50
1.32
30
0.59
20
1.73
1.59
50
0.93
50
0.32
5
0.93
10
0.86
cup
106
200 0.89
200
1.89
250
2.36
100
0.06
150
1.42
100
1.06
200
1.36
200
0.89
250
1.36
200
0.89
bag of noodles
191
500 1.62
200
1.05
500
2.62
500
1.62
200
1.05
250
1.31
200
1.62
500
1.62
500 1.62 500
1.62
10
ripe avocado
204
200 0,02
200
1.02
1,02
200
200
1.02
150
1.26 200
0.02
200
0.02
200
0.02 150
0.26
11
squeeze bottle
250
300 0.20
300
1.20
500
2.00
300
0.20
450
1,80
350
1.40
400
0.
40
500
1.00
350
0.40
200
0.20
12
bag of rice
900
500
0.44
1000
1.71
1000
500
500
0.44
Aug Err
1.50
2.30
2.52
1.51
1.35
1.82
1.43
1.53
2.74
1.91
Table Model Mass Estimation: Columns a through represent the mass estimates and respective relative error across different model and prompt configurations. The configurations are as follow: a: gpt-3.5-turbo, b: finetuned gpt-3.5-turbo, c: gpt-4-turbo, d: gpt-3.5-turbo with CoT prompting, e: finetuned gpt-3.5-turbo with CoT prompting. f: gpt-4-turbo with CoT prompting, g: finetuned gpt-3.5-turbo with 1% training set size, h: finetuned gpt-3.5-turbo with 10% training set size, i: finetuned gpt-3.5-turbo with 1% training set size and CoT prompting. j: finetuned gpt-3,5-turbo with 10% training set size and CoT prompting
We also provide the full per-item breakdown mass estimates from the full set of evaluated models.
A.2
Per-Item Mass, Friction Coefficient, and Spring Constant Estimate Comparisons for Four DeliGrasp Variants
Table 7: DeliGrasp Model Comparisons of Estimated Fmin
Ground Truth ID Object Name
DG
DG
D
DG PT
DG FT Col
F
m (g) (N/m)
F
F
m
m Em
F
k
F
F
F
F
m
m
F
m
m
(N)
Erz
(N/m)
(N)
Em
(N)
Em:
(g)
Er: (N)
Erz (B)
Er
Paper Airplane
0.02 0.8. 184,9 00100
4.00 5
6.50
3.91
0.25
1.25
Cup (empty)
0.11 3.6 389.8
0.25
1.27 10
2.78 0.4 50
0.5 3.55
0.25 1.23 10
2.78 0.12 0.11 5
1.39
Dried Yoba
0.16 5.5
157.7
0.39
1.44
3.64 0.5 200
0.5 0.33
1.45
20
3.64 0.20 0.23 10
1.82
Raspberry
0,18 6
61.0
0.06
0.67 5
1.17 0.8
0.2 0.11
0.25
0.36
20
3,33 0.06 0.66
1.17
Hard Taco
0.44 15
188.4
0.98
1.23 50
3.33 0.5 1000
1.5 2:41
0.20
0.55
10
1.33 0.20 0.35 10
1.33
Mandarin
1.65
151.8
1.88
150
2.68
1.25
0.24
0.11
2,68
0.61
50
1.11
Stuffed Toy
2.18 74
192.8
0.61
0.72 50
1.22 0.8 100
1.5 0.31 0.61 0.72 50
1.32 0.25 0.89 20
1.73
Cup (water)
3.12
373.2
4.08 0.31 250
0.04 0.05
1.89
0.21
1.42
Bog (noodles)
5.62 191 4324
12.3 1.79 500
2.62
0.4 300
0.29
0.13 200
1.05
4.91 0.13 200
1.05
10 Avocado
6.00 204 298.7
3.92 0.35 200
1.02
0.5 500
0.83 3.92 0.35 200
1.02
3.92 0.35 200
1.02
11 Spray Bottle
7.36 250 1383.
12.2 0.66 500
2.00
0.4 150
0.32 7.36
0.00
300
1.20
11.0
0.50 450
1.80
12 Bag (rice)
26.5 900 6192.
19.6 0.22 1000
1.17 0.5 200
0.70 0.70 19.6 0.26 1000
1.11 19.6 0.26 1000
1.11
We record LLM estimates for F, the minimum grasping force, the relative error F Err., mass m, and ra- tio of mass overestimation m Err. for 5 different DeliGrasp strategies: DG with the default model without finetuning, the ablated DGD which directly estimates F. DG FT with the model finetuned on the PhysOb- jects dataset, and DG FT Col' for the finetuned model with additional offin-of-thought prompting to elicit PhysObjects- comparisons. We use the same μ and & values across models to isolate the effect of mass estimates grasping. Lower mass, and thus F estimates for DG FT DG FT COT for the bag of noodles and hard-shelled taco explain why resulting policies in 7 have more slip failures.
A.3 Classical Adaptive Grasping Baseline Validation
We first validate these baseline methods on the evaluated objects from their respective studies, span- ning a potato chip, tomato, and paper cup (2g to 150g) for "In Place" grasping and an empty sty- rofoam cup. filled styrofoam cup, empty plastic water bottle, filled plastic water bottle, and filled cereal box (68g to 384g) for "In Motion" grasping. For the "In Motion" strategies, we tune a sepa- rate set of applied force gain and velocity gain for the 0.5N and 1 1.5N contact force strategies. The 0.5N strategy has more aggressive gain terms than the 1.5N strategy, and our current-draw based force sensing used in "In Motion" adaptive grasping is much lower resolution and frequency than dedicated force sensing resistors and slip sensors used in the original baselines.
13

---

Query Object
lzz
kg
ka
(N/m) (Nim) (N/m) (N/m)
200g
Avocado 65.1 143.5 335.4
Tomato 271.6 331.8 606.2 1589.
kt 5ee N/in
given time, which should
mint 2,52
New Avocado
This grasp sets the
week? Tomato
initia force different Initial
Right Avocado
now? Tomato
Farce 0.5 Noucon
Three Avocado
because of requiring gentle prossing to 355e15 ripeness causing
Days? Tomato
When will this go bad? (days)
without
Avocado
3-5
4-6
5-7
Indentations.
(a) Force Adjustment
(b) Spring Constant
(c) LLM Reasoning
Figure DeliGrasp adjusts the grasp force (A) verh of "checking" the avocado, from estimated 3.92 to 0.5 N. Each grasp measures a spring constant (B) without damaging the avocados. Such measurements can be used for downstream LLM--rasoning tasks (C) like picking ripe produce or meal planning.
We note that the "In Motion" strategy, in various implementations, requires, in addition to custom sensor configuration across force, contact, slip, and/or pressure, expert tuning of controller param- eters. DeliGrasp requires no sensing beyond the gripper motor(s) current draw and no parameter tuning. Perhaps the even larger obstacle is that such strategies require motion execute, limiting efficacy on certain geometries and inducing slip uncertainty in majority of grasps, whereas exccutes entirely in-place and is equally effective across object types.
A.4 Sensing with DeliGrasp to Pick Ripe Produce
DeliGrasp explicitly measures spring constant & while performing Algorithm 1: the adaptive grasp controller checks for contact force Fe with each small motion Ar, and k -= These measured (Appendix A.1) show that DeliGrasp estimates of spring constants are incorrect by a factor of 6.5 (2.5 for only the compliant objects) and, more relevantly, be utilized tasks such appraising produce ripeness, correlate of compliance (Figure 6b and 6c). This sensor data can be used for learning tasks, human collaboration tasks, or for higher-level reasoning tasks such as ripeness checking or appropriateness for specific dishes. We perform this measurement on two assortments of produce: avocados and tomatoes. When receiving the "grasp verb" of "check" or "inspect for ripeness", DeliGrasp bypasses the default minimum grasp force value of ma and manually sets lower F. of 0.5 N for avocados and 0.2 for tomatoes.
compliance sensor in
Qualitatively, the relative ordering of the measured k for each item type corresponds with judgments from our own human hand grasps. We query LLM (GPT-3.55 with the item name and compliance data, and show in Table 6c that it is able to reason over and answer abstract, relative questions about when to eat them. Further qualitative questions such as "how" to cat specific item "right now" yield dishes appropriate for the ripeness of the produce (Appendix A.5).
A.5 Ripeness Reasoning with LLMS
1. Asking "how should eat avocado ky" yields: sliced avocado, grilled avocado, salad top- pings, stuffed avocado
2. Asking "how should cat avocado ki" yields: guacamole, avocado toast, sushi rolls, smoothies
3. Asking "how should eat tomato ka yields: sliced for burgers, grilled or roasted, salsa, stuffed
4. Asking "how should eat tomato k yields: caprese salad, gazpacho, pasta sauce
We additionally observe that multimodal models like GPT-4V OWL-ViT alone cannot reason ripeness over visual images. We query GPT-4V with an image of the same items times and request
14

---

ripeness ranking. For the 4 tomatoes, GPT- 4V produces the same ranking each query, from most ripe to least ripe, of: [k4, k2, k3, k.), reasoning entirely on the "redness" of the tomatoes. For the avocados, it refuses to generate a ranking, citing a fundamental lack of sensory capabilities.
OWL-VIT when directly queried for "the ripest" "least ripe" produce arbitrarily detects items (sometimes the same item) depending the angle (and thus lighting, resolution, relative object size) of the captured image.
When we pair the spring constant data with the image in a query to GPT-4V, ignores the visual characteristics and reasons solely on the spring data.: Query:
have labeled avocados
you between 0, 1, 2 provide their average spring constants from an initial ripeness check. Spring constants are inversely proportional to ripeness--a low spring constant means soft object, and high means stiff Combining this information with the image, please do the following tasks:
1. Estimate the ripeness of ripe and unripe avocado.
2. Bank the given avocados by ripeness.
E. Using the estimated ripeness range, estimate when each one will bad (avocados are stored in the open at room temperature).
Spring constant data: Avocado 1: 65 N/m
Avocado 2: 143 N/m
Avocado 3: 335 N/m
Response:
To perform the tasks, we need to use the given spring constants and the visual clues from the image to estimate the ripeness and predict when each avocado will spoil. Typically, ripe avocado has spring constant at the lower end because it is softer, while less ripe or unripe avocado has higher spring constant because is firmer.
0. Eatimate the ripeness of ripe and unripe avocado: ripe avocado typically has low spring constant, indicating softness, perhaps similar to Avocado 1, which has the lowest spring constant
An unripe avocado has high spring constant, as it's stiffer, like Avocado 3, with the highest spring constant
1. Rank the given avocados by ripeness: Avocado 1, with the lowest spring constant range 65, would generally be the ripest.
Avocado 2, with spring constant range 143, would be less ripe avol.
Avocado 3, having the highest spring constant range 335, would be the least ripe.
So the ripeness rank from ripest least ripe would be:
Avocado 1 Avocado 2 Avocado
2. Estimate when each avocado will go bad:
Avocado 1, being the ripest, based its lover spring constant, about 2-3 days stored room temperature. Avocado 2 is less ripe than avol and may take around 4-5 days to spoil at room temperature.
could spoil
Avocado 3 is the least ripe, with quite high spring constant and could take upwards of week to spoil, potentially around 7-10 days at room temperature.
Please note that these estimates are quite rough and actual spoilage times vary due numerous factors such specific
15

---

temperature, humidity, and the avocado's initial condition. The visual clues limited, but from the image, the physical characteristics that usually indicate ripeness (dark skin, slightly yielding to gentle pressure) don't provide significant additional information beyond the spring constant data.
A.6 DeliGrasp Desciptor Prompt
The descriptor takes as imput a "grasp verb", "object deseription", and "object width." It outputs structured summary of the object, how it should be grasped, and physical characteristics of the object.
Control robot gripper with force control and contact information. The gripper's parameters can be adjusted corresponding
the type object that it is trying to grasp as well as the kind of grasp it is attempting to perform.
The gripper has measurable max force of 16N and min force of 0. 15N, maximum aperture of 105mm and minimum aperture of 1mm
Some graspa may be incomplete, intended for observing force information about given object. Describe the grasp strategy using the following form:
[start of description]
This {CHOICE: [is, is not]} a new grasp
In accordance with the user instruction, this grasp should be (GRASP_DESCRIPTION: <str>]
This is {CHOICE: [complete, incomplete]) grasp.
This grasp {CHOICE: [does, does not]} contain multiple grasps This grasp is for an object with (CHOICE: [high, medium, low]) weight.
The object has an approximate mass [PNUM: 0.0] grams This grasp is for an object with {CHOICE: [high, medium, low]}
compliance.
The object has approximate spring constant of [PNUM: 0. Nevtons per meter.
The gripper and object have an approximate friction coefficient [PNUX: 0.
This grasp should set the goal aperture to [PNUM: 0.0] mm.
If the gripper slips, this grasp should close an additional [PNUM:
0. mm.
If the gripper slips, this grasp should increase the output force by [PNUM: 0.0] Newtons.
[optional] Because of [GRASP DESCRIPTION: <str>], this grasp sets the force to be (CHOICE: [lover, higher]) than the default minimum grasp force. [end of description]
Rules:
1. If you phrases like {NUM: default_value), replace the entire phrase with numerical value. you see {PNUM: default_value), replace it with a positive, non-zero numerical value.
2. If you see phrases like {CHOICE: [choicel, choice2, ...]}, it means you should replace the entire phrase with one of the choices listed. Be sure replace sure about the value, just use your best judgement.
of them. you are not
3. If you see phrases like information from the user instruction to provide description of be grasped, including mentioned
default_valuel, use
the grasp or the object physical characteristics or features.
4. Using information from the user instruction about the object and the grasp description, set the initial grasp force either to this default value appropriate value.
16

---

5. If you deviate from the default force value, explain your reasoning using the optional bullet points. It not deviate from the default value.
6. Using knowledge of the object and how compliant it is, estimate the spring constant of the object. This can range broadly from 20 N/m for very soft object to 2000 N/n for very stiff object.
7. Using knowledge of the object and the grasp description, if the grasp slips, first estimate an appropriate increase to the aperture closure, and then the gripper output force.
8. The increase in gripper output force the maximum value of (0.05 N, the product of the estimated aperture closure, the spring constant of the object, and a damping constant 0.1: wwwpettiioal..osueeeee0 0001)).
9. Provide the full description of the grasp plan, even if you may only need to change few lines. Always start the description with [start of description] and end it with [end of description].
10. not add additional descriptions not shown above. Only use the
bullet points given in the template.
11. Make sure to give the full description. Do not skip points if they not optional.
A.7 DeliGrasp Coder Prompt
The coder takes input the structured summary of the grasp and inferred characteristics and gener- ates an grasp policy which implements the adaptive grasping algorithm and compliance sensing.
We have description of gripper's motion and force sensing and want you to turn that into the corresponding program with following class functions of the gripper:
The gripper has measurable max force of 16N and min force of 0. 0.N5N, maximum aperture of 105mm and minimum aperture of 1mm.
.c.
def get_aperture (finger=/oot
finger: which finger get the aperture in of, either 'left', 'right', or 'both', If 'lett' or 'right', returns aperture, or distance, from finger to center. If 'oot'' returns aperture betueen fingers.
def
(aperture, finger='both? record
aperture: the aperture to set the finger (s) to (in mm) finger: which finger set the aperture in mn, of, either 'left', 'right', or 'both',
record whether to record the load at the goal aperture. If true, will return array of (pos, load) tuples
This function will move the finger to the specified goal aperture, and is used close and open the gripper.
Returns a position-load data array of shape (2, n) [[positions], [loads]], average force, and max force after the motion.
def set_compliance (margin, flexibility, finger=
margin: the allorable error betseen the goal and present position (in mm)
flexibility: the compliance slope of torque (value 0-7, higher is more flexible) until reaches the compliance margin
finger: which finger to set compliance for, either 'left', 'right', or 'ooth'
17

---

def set_force (force, finger='bothe
rcr
force:
the maximum force the finger is alloved to apply at contact with an object N), ranging from (0.1 to 16 N)
finger: which finger set compliance for, either 'left', 'right', 'both'
...
def deligrasp
initial_force, additional.closure. additional_forc,, complete.grasp)
...
goal aperturee the goal aperture to grasp the object (in mm) initiallforce: the initial force to apply to the object (in N) ceatiionl_closuree the additional aperture slips (in mm)
close if the gripper
addiiiioallfooree the additional force to apply if the gripper slips (in N)
complete_grasp: whether the grasp is complete or incomplete (True or False)
This function will close the gripper to the goal aperture, apply the initial force, and adjust the force the gripper alips. the graap is incomplete, the gripper will open after the slip check.
def poke (direction, speed, aperture)
...
direction: 'left' or 'right' to poke the left or right finger speed of finger in m/s
apprture: distance to poke mm (of one finger, both)
Example answer code:
from magpie. import Gripper must import the gripper class Gripper
import numpy as # import numpy because we are using it below
coal_aperture {PNUM: goal_aperture
complete_grasp {CHOICE: [True, False]}
Initial force. Convert mass (g) to (kg). The default value of object weight friction coefficient..
initiall {PNUM: {CHOICE: [([PNUM: mass) 9.81) / ((PNUM: mu} 1000), (PNUM: dffferemt_ittallffrree]] }}}
additional_closuu closure (PNUM: additional_closure
Additional force increase. The default value is the product of the object spring constant and the additional,closure with a dampening constant 0.1.
ma.max mao.0001 01, additionsl,closure (PNUM: additional_force mp.may max
spring.constant) * 0.0001])
G. set_geall.pprtuurrooooooooo apeturre additional.closure 2, finger=/both", lecord_load=Falee)
G. set.compliance(1, 3, finger='both") ai._force "both')
G. deliggrapppolll_peruur imitial.foree additional.cloaure additional_force. completescopleee__ress debug=True)
Remember:
Always format the code in code blocks. In your reaponse all five functions above: get_aperture www_-goal_pperuure set_ set_compliance. force, sheck_slip should be used.
18

---

2. Do not invent new functions classes. The only allowed functions you can call the unimplemented code blocks in your reaponse.
listed above. Do not leare
3. The only alloved library is numpy. Do not import or use any other library. If you use np, be sure to import numpy. just your best judge.
4. If you not 5. you see phrases like [REASONING], replace the entire phrase with a code comment explaining the grasp strategy and its relation to the following gripper commands.
sure what value to None for anything.
6. If you with a prediction gripper commands are executed.
phrases like [PREDICTION], replace the entire phrase
the gripper's state after the following
7. If you see phrases like (PNUM: with the corresponding value from the grasp description.
[itault_value), replace the value
8. If you means you should replace the entire phrase with one the choices listed. Be sure to replace all of them. If you are not sure about the value, just use your best judgement.
phrases like {CHOICE: [choicel, choice2, ....}, it
9. Remember to import the gripper class and create Gripper the beginning of your code.
10. Remember to take into account instructions to use different forces than the default values, and to explain your reasoning in the code.
A.8 PhysObjects Finetuning Data Format, CoT Prompt, and CoT Latency Evaluation
Each text caption is adapted to pair-wise relative Q&A conversations across these physical con- cepts concepts, following the structure: Q: Which {has more mass, is more fragile, is more deformable). material, objectı or material2 object?? and A: A material object {has more mass, more fragile, is more deformable).
When evaluating mass estimates from finetuned and/or CoT CoT-prompted models for atypical objects, we query the model 10 times for the mode of estimates (minimum 5) as mass estimates are more varied. We execute these queries in parallel, which is a little bit slower than querying an LLM just once, as the final requests are vate-limited and slower to complete, taking on average 3.73s (n=30).
We add three new mandatory bullet points to the description, starting after the description of whether the grasp contains multiple grasps. These three bullets elicit CoT thinking and leverage PhysObjects finetuning by requesting explicit mass comparisons between heavier, lighter objects and typical vs. the user-described object to be grasped. We modify rule 3. add rule for an example object (rule 4) to further enforce CoT and leverage PhysObjects finetuning.
Control a robot gripper with force control and contact information. The gripper's parameters can be adjusted corresponding to the type of object that it is trying to grasp as well as the kind of graap it is attempting The gripper has measurable max force 16N and min force of 0. 15N, a maximum aperture of 105mm and a minimum aperture of 1mm.
perform
Some grasps may be incomplete, intended for observing force information about given object.
Describe the grasp strategy using the following form:
[start of description]
This {CHOICE: [is, notl]
grasp
-In accordance with the user instruction, this grasp should [GRASP_ DBRAS__DESCRIPTION <str>]
This is (CHOICE: [complete, incompletel) grasp.
This grasp {CHOICE: [does, does not]} contain multiple grasps
19

---

This object has more mass than [example object: <str>], with mass of [PNUM: 0.0] 8, and less with mass of [PNUM: 0.0]
than [example object: <str>],
Typically, this object's mass is approxinately [PNUM: 0.0] g, which is between these two masses.
Because the compared typical, this object has {CHOICE: [greater, leaser, similar]} mass of [PNUM: 0.0] grams.
specified that DBBJECT_DSCCRIPTON:: <str>],
This graap is for an object with {CHOICE: [high, medium, low]} compliance.
The object has approximate spring constant of [PNUM: 0.0]
Newtons per meter.
The gripper and object have an approximate friction coefficient [PNUK: 0.0]
This grasp should set the goal aperture to [PNUM: 0.0] mm.
If the gripper slips, this grasp should close additional [PNUM:
0.0] mm
If the gripper slips, this grasp should increase the output force by [PNUM: 0.0] Newtons.
[optional] Because of [GRASP DESCRIPTION: <str>], this grasp sets force to {CHOICE: [lower, higher]) than the default minimum grasp force. [end of description]
Bules:
1. If you see phrases like (NUM: default_value), replace the entire phrase with numerical value. If you see (PNUM: default_value), replace it with positive, non-zero numerical value.
2. If you see phrases like {CHOICE: [choicel, choice2, ... means you should replace the entire phrase with one the choices listed. Be sure to replace all of then. If you are not sure about the value, just use your best judgement.
3. If you phrases like (GRASP vefault_value] DESJECT_DSSCRIPTION default_value), use information from user instruction to provide a description of the grasp or the object to be grasped, including mentioned physical characteristics or features.
4. If you phrases like [example object: <str>], replace the entire phrase with an appropriate example object that is similar to the object to be grasped.
5. Using information from the user instruction about the object and the grasp description, default value or an appropriate value.
the initial grasp force either this
6. If you deviate from the default force value, explain your reasoning using the optional bullet points. It is not common to deviate from the default value.
7. Using knowledge of the object and how compliant the spring constant of the object. This can range broadly from 20 N/m for a very soft object to 2000 N/m for a very stiff object.
is, estimate
8. Using knowledge of the object and the grasp description, if the grasp slips, first estimate aperture closure, and then the gripper output force. gripper output force the maximum value of (0.05 N,
appropriate increase
the
9. The increase or the product of the estimated aperture closure, the spring constant of the object, and a damping constant 0.1: Araaditiional.cloureee. 0001)).
10. Provide the full description
the grasp plan, even if you may only need to change a few lines. Alvays start the description with [start of description] and end it with [end of description).
11. Do not add additional descriptions not shown above. Only use the bullet points given in the template.
12. Make sure to give the full description. not skip points if they are not optional.
20


Reviews:
[{"review_id": "3971n6B2Kt", "reviewer": "Reviewer_JraE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors introduce DeliGrasp, which utilizes LLM's physical reasoning and code generation capabilities to infer object properties from descriptions and create adaptive grasp policies. DeliGrasp tackles the challenge of grasping diverse and fragile objects like paper airplanes and ripe avocados. A complete pipeline is presented, encompassing object localization, description, and initial grasp point estimation. A dual-prompt LLM interaction generates executable grasping code. Furthermore, estimating object compliance while grasping enables answering context-specific queries about the objects for downstream tasks, such as assessing avocado ripeness for food preparation. The authors demonstrate the effectiveness of DaliGrasp by conducting multiple successful grasping experiments on a dataset of 12 diverse delicate objects.", "summary_of_recommendation": "The authors present a compelling approach for leveraging LLMs in the grasping of delicate objects, showcasing the potential of this methodology. However, the current evaluation strategy raises questions about the generalizability and robustness of their findings. I recommend accepting this paper only if the authors provide additional experimental analysis to address these concerns. Otherwise, I would suggest rejection due to insufficient evidence of the approach's efficacy.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Please explain why the estimated mass varies for different approaches on the same object (e.g., Object ID 11) in Table 2 while the reported error remains consistent. \n- How is the Columbian friction model accounted for while generating LLM responses?\n- What are the actual prompts used to generate the description and code?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lSks8Xfusf", "reviewer": "Reviewer_M14h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper explores the application of large language models (LLMs) to enhance robotic grasping capabilities by leveraging their ability to reason about physical properties and generate code. It describes a novel approach where LLMs are used to infer the mass, friction coefficient, and spring constant of objects from their semantic descriptions. These physical characteristics are then translated into executable, adaptive grasp policies. The study tests these LLM-parameterized grasp policies against traditional adaptive grasp policies and direct LLM-as-code policies using a custom benchmark comprising 12 delicate and deformable items, such as food, produce, and toys, with a wide range of mass and required pick-up force. Results show that LLM-parameterized policies outperform the alternatives. Further improvements in property estimation and grasp performance are achieved through model finetuning on property-based comparisons and the use of chain-of-thought prompting to elicit these comparisons.", "summary_of_recommendation": "While the approach is promising and the evaluation demonstrates potential advantages, the incremental nature of the contributions and the lack of crucial comparative analysis weaken the paper’s impact. If these issues are addressed, particularly by enhancing the novelty and depth of evaluation, the paper could be a strong candidate for acceptance in future submissions. A weak reject is recommended at the current state.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* Considering that DeliGrasp requires querying LLMs ten times to derive simple quantities, what is the total computation time for executing this process? Detailed information on computational efficiency is essential for assessing the practicality of integrating LLMs into real-time robotic operations.\n* What specific improvements are observed from finetuning the LLMs on the PhysObjects dataset? Additionally, why is the GPT-3.5-Turbo model without finetuning not included in the study?\n* Does the PhysObjects dataset used for finetuning the LLM include any of the objects later tested in the evaluation phase?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CXOFeQD11s", "reviewer": "Reviewer_uYAW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose to use an LLM to predict the mass and friction of an object from a high-level object description. These values parameterize an adaptive grasp controller to pick up the object. The authors further vet the potential of LLM-generated parameters by fine-tuning on a dataset of captioned object images.", "summary_of_recommendation": "Overall I recommend acceptance because I think the analysis is interesting, but I think it's important that the authors improve the clarity in the rebuttal period.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- How is success scored in Figure 3? Are the grasps manually labeled as causing damage vs not causing damage?\n- Are the deli grasp results in section 4 all using the fine-tuned model?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Post-Rebuttal Meta Review:
-------------------------------------------
The paper presents an interesting grasping framework. Given that the rebuttal and responses addressed the major concerns of the reviewers, I am happy to recommend the paper for acceptance. We look forward to the discussion and changes being reflected in the camera-ready version. 

Original Meta Review:
-------------------------------------------
This paper proposes DeliGrasp, a grasp controller whose parameter selection is based on semantic information gathered from an LLM.

Strengths: The paper proposes an interesting strategy of integrating common-sense knowledge into an adaptive grasp controller and demonstrates this on a variety of real-world objects. The paper is well-written and presents an entire system in an easy-to-understand way.

Weakness: Given the rise of systems that leverage LLMs to provide common-sense reasoning, the paper could more clearly justify how it differs from previous methods (Reviewer M14h) as well as compare to a wide range of baselines (Reviewer M14h, Reviewer uYAW). It would also strengthen the paper to clarify a few details with respect to the experimental setup (Reviewer JraE), the specification of a few figures and tables (Reviewer JraE, Reviewer uYAW) and the computational expense of the framework (Reviewer M14h).
Decision: Accept