Grasp Adaptation Control with Finger Vision
Verification with Deformable and Fragile Objects◦Akihiko Yamaguchi (CMU) Christopher G. Atkeson (CMU)
1.

Introduction

In order to establish a strategy to learn robotic
grasping behaviors with external vision (e.g. cameras
on a robot head) and tactile perception obtained byFingerVision, we develop a grasp adaptation control
that grasps unknown objects with adequate grasping force. FingerVision proposed by Yamaguchi andAtkeson [1, 2] is a vision-based tactile sensor that gives
robots a tactile sensation and visual information ofnearby objects. When grasping objects, humans arecombining vision and tactile perception. However, useof tactile perception is not considered as essential inrobotics. For example, in the recent work of learning robotic grasping with deep learning [3], robots
learned grasping without tactile sensing. This was
possible because there is a consistent relation between
the state before grasping (visual scene of the object
and the gripper) including the grasping parameters
and the outcome of grasping. Tactile sensing is intermediate information, which is not necessary to use inlearning grasping behavior.
However tactile perception is useful in many manipulation scenarios. When grasping a container whose
contents are unknown, visual information is not sufficient to decide the grasping force, while tactile sensing can handle this problem. Romano et al. createda tactile-based grasping behavior using a PR2 robotthat grasped a range of unknown objects including
deformable ones [4]. From our experience of peelingbanana with a Baxter robot teleoperated by a human,
we noticed that the absence of tactile feedback makes
teleoperation difficult [5]. Since peeling a banana isa dual-arm manipulation task and bananas are deformable, estimating force applied by the other handis not easy, which makes it difficult to avoid slippingor crushing.
We explore combining external vision and tactile
perception in learning grasping behaviors. This approach should make the acquired grasping behavior
more robust, and the required number of samples
might be decreased. We consider a grasping behavior
model consisting of two components: one is a predictor of grasp parameters such as a grasp pose using
external vision, and the other is a controller to adapt
the grasp to hold an object using tactile sensing. We
refer to them as the grasp pose estimator and grasp
adaptation controller respectively.
This paper reports on an implementation of grasp
adaptation controller with FingerVision and the re-Fig.1 Left: Robotiq gripper with FingerVision sensors. Right: 30 objects used in the experiment.

Fig.2 Examples of grasp (cup cake, raw egg, hairyrubber toy, strawberry, tomato).

sults of preliminary experiments. We consider thegrasping task as holding an object without crushingit. Since holding an object is achieving a state where
the object is not slipping in a gripper, controlling thegrasp to avoid slippage would be the most straightforward approach. We use FingerVision [1] to detect
slip, and propose a grasp adaptation controller thatmodifies the grasp to avoid slip.

2.

Related Work

Similar ideas of using slip sensing in grasping areexplored in [4, 6, 7, 8]. There are many papers on slip
detection such as [9]. The major advantage of ourapproach is that since FingerVision provides visionbased slip detection, it does not depend on the force
between the fingers and an object. It can sense slip
even when the object is very lightweight, such as
grasping origami objects. This approach can control
grasping when the force is too small to measure. Sucha grasping strategy is adequate especially for grasping
deformable and fragile objects such as vegetables, raw
eggs, and origami objects.

3.

Grasp Adaptation Controller

We use FingerVision to detect slip. The image sequence obtained by FingerVision is processed using
a background subtraction method. Since the background subtraction perceives both the object move-ment and the background movement (including gripper movement), we need to distinguish the object
from the background. First we build a backgroundmodel, and then we adaptively construct an object
model. Finally we use the object model as a mask to
extract the object movement from the image. Boththe background and the object models are represented
as HSV color histograms. We use an implementation of OpenCV (cv::BackgroundSubtractorMOG2)
for the background subtraction.
The grasp adaptation controller is modeled with afinite state machine. We assume a two-finger parallel
gripper. It has two phases: (1) Moving the object
upward slightly (5 mm) and waiting for a short time
(0.4 s). If slippage is detected, moving the object tothe initial height and closing the gripper slightly (0.7mm). These are repeated until no slippage is detected.
(2) Moving the object upward to a target height (15cm from the initial height). If slippage is detected, wego back to the phase (1). In both phases, a feedback
control is activated: when slip is detected, closing thegripper slightly (0.7 mm).

4.

Preliminary Test of Grasp Adaptation Controller

We conduct a preliminary test of the grasp adaptation controller. We verify that when the grasp pose
estimator gives an adequate pose, the grasp adaptation controller can grasp an object robustly. We leta human operator decide a good grasping pose to agiven object with a joystick controller, and then run
the grasp adaptation controller to pick up the object.
We do not tune the parameters of the controller foreach object.
We tested with 30 deformable and fragile objects
shown in Fig. 1 including vegetables, fruits, origami
objects, and a raw egg. We use a Baxter robot with a
Robotiq 2-Finger Adaptive Robot Gripper where twoFingerVision sensors are installed as shown in Fig. 1.
Initially each object is placed on a table.
We conducted 36 trials: Origami box, Origami
crane, Badminton ball, Hairy rubber toy, Cupcake, Chocolate, Strawberry, Tomato-medium-1, Eggplant@1, Eggplant@2, Zucchini-yellow, Mushroom1@1, Mushroom-1@2, Egg(raw), Pepper-red-1, Oyster mushroom-1, Peach-1, Mushroom-2, Potato1, Kiwi-1, Tomato-medium-2, Broccoli@1, Broccoli@2, Oyster mushroom-2, Green pepper-1, Kiwi2, Pepper-red-2, Tomato-big, Banana-1@1, Banana1@2, Banana-1@3, Green pepper-2@1, Green pepper2@2, Peach-2, Potato-2, Banana-2. A label with @N
denotes an N-th trial of the same object. Examplesof successful grasping are shown in Fig. 2. There wereseveral failures: (1) Dropped after bringing up: Oyster mushroom-1, Potato-1. (2) Slippage could not
be detected due to a computer vision failure: Eggplant@1 (the skin was black), Broccoli@1 and Green

pepper-2@1 (the color was similar to the fingers).
(3) Closing gripper did not stop in Banana-1@1 because detecting the deformation of object as slip.
Since the contact force from the table disappeared
when bringing up the banana, the banana skin wasdeformed slightly. (4) In Banana-1@2, dropped during bringing up, and failed to re-grasp since the fingers
got stuck at the edge of the object, and the passive
joints of the gripper bent. Note that (2) was solved bygrasping the green part (Eggplant@2), helping the object detection manually (Broccoli@2), and just trying
again (Green pepper-2@2).
The issues of (2) and (3) will be solved by improving the computer vision method for: (A) a better object detection and (B) distinguishing slippage and deformation. The issues of (1) and (4) will be solved
by improving the behavior. For example, testing thegrasp stability by shaking the object after grasping
will avoid (1). (4) can be solved by optimizing thetrajectory of fingertip in re-grasping.

5.

ConclusionWe explored a grasp adaptation control that grasps
unknown objects with adequate grasping force without crushing them. The controller is a feedback control of slip where FingerVision [1] is used to detectslip. Since this slip detection is vision-based, it cansense slippage of very lightweight objects such asorigami objects. The results of preliminary experiments showed that although slight improvements arenecessary for the slip detection and the grasp adaptation, this approach is promising toward the goal oflearning grasping a range of objects.
References[1] A. Yamaguchi and C. G. Atkeson: “Combining finger visionand optical tactile sensing: Reducing and handling errors
while cutting vegetables”, Humanoids’16 (2016).
[2] A. Yamaguchi: “Fingervision”, http://akihikoy.net/p/fv.html (2017).
[3] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz and D. Quillen:“Learning hand-eye coordination for robotic grasping withdeep learning and large-scale data collection”, The International Journal of Robotics Research, p. 0278364917710318(2017).
[4] J. M. Romano, K. Hsiao, G. Niemeyer, S. Chitta and K. J.
Kuchenbecker: “Human-inspired robotic grasp control withtactile sensing”, IEEE Transactions on Robotics, 27, 6, pp.
1067–1079 (2011).
[5] A. Yamaguchi: “Baxter peels banana”, https://youtu.be/rEeixPBd3hc (2016). [Online; accessed Oct-28-2016].
[6] M. Kaboli, K. Yao and G. Cheng: “Tactile-based manipulation of deformable objects with dynamic center of mass”,
Humanoids’16 (2016).
[7] A. Ikeda, Y. Kurita, J. Ueda, Y. Matsumoto and T. Ogasawara: “Grip force control for an elastic finger using visionbased incipient slip feedback”, IROS’04 (2004).
[8] D. Gunji, Y. Mizoguchi, S. Teshigawara, A. Ming,A. Namiki, M. Ishikawa, and M. Shimojo: “Grasping force
control of multi-fingered robot hand based on slip detection
using tactile sensor”, ICRA’08 (2008).
[9] M. R. Tremblay and M. R. Cutkosky: “Estimating friction
using incipient slip sensing during a manipulation task”,
ICRA’93 (1993).