Thin plate manipulation by an under-actuated robotic soft gripper
utilizing the environment
Toshihiro Nishimura, Student Member, IEEE, Kaori, Mizushima,
Yosuke Suzuki, Member, IEEE, Tokuo Tsuji, Member, IEEE, and Tetsuyou Watanabe, Member, IEEE

Abstract— This paper presents methodologies for dexterous
manipulation by an underactuated robotic soft gripper with alow DOF. While the softness and passive joints provide a stablegrasp, the manipulations become challenging owing to thecomplexities of modeling and state estimation. To investigate thedrawbacks, target manipulations using a support surface andgravity are defined in this study. By realizing the target
manipulations, methodologies for utilizing both the support
surface and gravity in manipulation are validated. For
manipulation utilizing the support surface, the key states—at
which the type of gripper motion primitive changes—are defined,
and gripper motion at the key states is produced with a DNNbased motion generator. For manipulation utilizing gravity, thefall risk of the object is formulated as a constraint for a criterion
function, and aparameters-online-exploration based
methodology is adopted. The validity of the methodologies isdemonstrated through several experiments.

I. INTRODUCTION
Softness is a key requirement for stable object grasp.
Softness provides low contact impact, safe interaction with theobject and environment, high friction, and adaptation to theobject shape. Another key factor is utilization of anenvironment. Intentional contact with a table deforms thecompliant elements, and shifting of the grasping mode isfeasible. With a low DOF, multiple styles of grasping are
obtained. With these aspects in mind, an under-actuatedrobotic soft gripper was developed [1] (see Fig. 1). A ratchet
was installed at the under-actuated joint; the grasping mode isaltered when the ratchet comes in contact with a supporting
surface such as a table. Parallel gripper, pinching, andenveloping modes are available with only one actuator. The
gripper has a soft surface, which is constructed from a
deformable rubber bag filled with an incompressible fluid anda microgripper inside the fluid [2][3][4]. The microgripper
increases the weight-carrying capacity of robotic hand. The
grasping of a wide variety of objects was realized using thegripper.

The next challenge is to manipulate an object using thisgripper. The soft surfaces and passive joints of the soft gripper
cause uncertainties in state estimation and complexities ofmodeling. These issues are common for soft robotic hands.
Moreover, the gripper has only one actuator. Therefore,
utilization of an environment is key for realizing dexterous
manipulation. In this study, two types of utilizations ofenvironment are considered. One is to utilize a support surface
such as a table, and the other is to utilize gravity. A relevant
example of manipulations incorporating both theseutilizations is illustrated in Fig. 2. This is the target
manipulation inspired by in-hand human manipulations
utilizing the environments [5][6]. First, the gripper picks up
the target, which is a thin object laid flat, in an imbalancedmanner, wherein one finger is bent by a contact with the table
while the other finger is not (Initial grasp). From theimbalanced grasping state, the object is brought to a standing
position via reiterated up and down motions as well as closing
and opening of the gripper (Stand-up manipulation). Lastly,
the object is rotated by utilizing gravitational force such thatthe object is brought to the standing position in thelongitudinal direction (Rotation manipulation). The operation
is aimed at inserting the object into a box. By realizing thetarget manipulation, we will reveal the methodologies forutilizing both the support surface and gravity in manipulation
via soft robotic hands with low DOF.
The main contributions of this study are manipulation
strategies allowing 1) the uncertainties in state estimation andcomplexities of modeling, 2) only visual inputs to generategripper motions, and 3) unexpected object motions, which arerecovered by subsequent motions. At the stand-up
manipulation, a learning-based approach utilizing humandemonstration data (obtained from manual operations of thegripper) was adopted utilizing the concept of key states atwhich the type of motion primitive changed. A strategy forcontrolling the gripper at only the key states, was presented.
By the concept, we developed the framework wherein the
motion generation was repeated until the object attained thetarget posture. At the rotation manipulation, a parametersonline-exploration based approach was adopted, and unknown
parameters were identified by repeated exploring motions
until the object attained the target posture, while completely
avoiding fall risk.

Figure 1. The under-actuated robotic soft gripper [1] whose graspingmode is altered by contact with a support surface* Research partly supported by JSPS KAKENHI Grant Number 16H04298.
T. Nishimura and K. Mizushima are with the Graduated school of Naturalscience and Technology, Kanazawa University, Kakuma-machi, Kanazawa,
9201192 Japan (e-mail: to.nishimura@stu.kanazawa-u.ac.jp).

Y. Suzuki, T. Tsuji, and T. Watanabe are with the Faculty of MechanicalEngineering, Institute of Science and Engineering, Kanazawa University,
Kakuma-machi, Kanazawa, 9201192 Japan (e-mail: te-watanabe@ieee.org)

(a) Schematic illustration of the procedure of the manipulation(b) Example of the generated manipulation
Figure 2. Target manipulation where a target sheet lying flat is picked up for insertion into a box. Initial grasp: the gripper picks up the target object inan imbalanced manner, wherein one finger is bent by a contact with the table, while the other finger is not. Stand-up manipulation: from the imbalanced
grasping state, the object is made to stand-up via reiterated gripper motions of moving up and down, closing, and opening. Rotation manipulation: theobject is rotated utilizing gravity such that the object stands up in the longitudinal direction.

A. Related works
A number of robotic hands have been developed [7].
Recently, softness has been attracting attention owing to its
abovementioned benefits, and robotic hands with softness
have been developed, e.g., ROBO hand [8][9], lightweightunderactuated pneumatic fingers [10], ISR-SoftHand [11] andjamming gripper [12][13], Pisa-IIT Hand [14][15], and Velo
Gripper [16]. A more detailed review is available in [1].
The softness in robotic hands enables high adaptation totarget object and facilitates a stable grasp with a low DOF.
However, the high adaptation renders it challenging to modify
the grasped object’s pose and position. Manipulation using
soft robotic hands is challenging. A solution is to use anenvironment that can be utilized as if it was another DOF.
Object manipulation utilizing environments have been
investigated [17][18][19][20]. Mason and Rodriguez et al.
focused on pivoting manipulation utilizing support surfaces
such as table [21][22][23]. Francisco et al. realized pivoting
task utilizing gravity [24][25]. However, soft robotic hands
were not the subject of these studies. The soft components
renders it challenging to identify the parameters for thepivoting manipulation and to control the pivoting angle. It isnecessary to develop a fresh strategy to address theuncertainties in the application of soft robotic hands.
Another approach to address the uncertainties or softness isthe application of machine learning methodologies. Amongthese methodologies, DNN and Bayesian optimization, which
are closely related to this study, are focused on. DNN is ahighly effective tool to generate complex robot motions, andit has been used to develop a number of motion planners andgenerators [26][27][28][29][30]. However, soft robotic hands
have been studies less. A. Gupta et al. [28] presented a guided
policy search based algorithm to generate a manipulationmotion with the ROBO hand based on human demonstration
data [8][9]. The generation of recovery motion and regrasping

motion were not studied. The proposed methodology is basedon multimodal integration learning method [26][29][30],wherein the autoencoder [31][32] was utilized to connect
sequential robot motions, RGB images, and sound spectrums
and to generate a robot motion from an image sequence. The
main dissimilarity between this method and the proposed
method is the sequence of data. In the latter, only theinfrequent key states were considered to address an
unexpected motion due to the softness and passivity of the
newly developed gripper.
Bayesian optimization [33], [34] has been adopted for
identifying optimal parameter values [35][36][37][38]. A few
studies were focused on robotic hands [35]. Nogeira et al.
defined an unsafe grasp as one wherein the grasp parameters
are highly sensitive to marginal perturbations; they proposeda method for realizing a safe grasp. “Safe” in that study is
dissimilar from “safe” aimed at in this study; the latter implies
the avoidance of falling down. For this purpose, Safe Bayesian
optimization [36][39], which was originally adopted for
identifying control parameters for quadcopters, was adopted
in this study.
To our knowledge, there is no study on the completion ofcomplex manipulation by a soft robotic hand, using two typesof machine learning based methodologies.

Figure 3. Position of cameras

II. SYSTEM FOR MANIPULATIONS
The developed under-actuated robotic soft gripper [1] ismounted on the XYZ motorized liner stage (Oriental motor).
To obtain visual information of the object and gripper, twocameras (Intel RealSense SR300) were installed, one each atthe back and side of the gripper, as illustrated in Fig. 3. The
infrared image from the back camera was utilized during theinitial grasp and stand-up manipulations, while the infrared
image from the side camera was utilized during the rotation
manipulation.
The target object is a thin and deformable rectangular object
such as a PTP packaging sheet exhibited in Fig. 2(b). The size,
shape, texture, gravitational center, and friction of the object
are assumed to be unknown. A packing operation wasconsidered, wherein such a thin object is inserted in a box. For
the insertion, the study targeted a task, wherein the object ismanipulated such that it stands up in the longitudinal direction
from the initial lay-flat posture.
III. STRATEGIES FOR MANIPULATIONS
A. Initial grasp
Human demonstration data was collected for the target
manipulation, and the procedure for the manipulation waslearned. The aim is to pick up the object, which is lying flat on
the edge of the table, in an imbalanced manner wherein one
finger is bent by a contact with the table, while the other finger
is not. The amount of lateral movement is then determinedsuch that the center position of the gripper coincides with theposition of the table edge in the lateral direction. The amountof downward movement is determined by the lock position of
the ratchet gear of the right finger. The closing distance isdetermined by the size of the target object, which can be
estimated from a single image. Therefore, the problem is∈formulated as the derivation of the closing distancefrom a single image captured by the back camera∈(256 × 192 pixels). Here, the when the right fingertip
made contact with the table (0 mm in Fig. 2(a)) was
utilized.
The problem is resolved by learning the relationshipbetweenandfrom the human demonstration data.
The autoencoder [31][32] and GRNN (Generalized
Regression Neural Network) [40] were adopted, as illustrated
in Fig. 4:(1)
First, the image features∈
are extracted using theautoencoder. The autoencoder is a neural network comprising
input, hidden, and output layers. Learning was conducted suchthat the input for the input layer equals the output for the
output layer; moreover, it is an unsupervised learning. The
layer structure is symmetric with respect to the hidden center
layer. By utilizing a smaller number of neurons for the hidden
center layer than the number of the input or the output, lowdimensionalized data can be obtained at the hidden center
layer. This implies that the input is encoded to the lowdimensionalized data while the output is decoded from thelow-dimensionalized data. Therefore, it is feasible for thehidden center layer to contain the low-dimensionalized image
feature. By selecting only the encoder part, the function of
extracting the low-dimensionalized image feature from theimage can be constructed. The function is referred to as image
feature extraction encoder (IFEE) in this study:

(2)
from
is
Next, the closing distance of the gripper
derived using GRNN. GRNN was adopted in order to obtainvarious values ofaccording to . GRNN is constructedusing a radial basis and special linear layers. For the available
training data set (,), (,), ⋯ ,(as,), the output
∑

exp
∑

exp

for the input

is expressed
/ 2

(3)

/ 2

where is a constant parameter. It is noteworthy that for
GRNN, a learning process is not required. If adequate numberof data sets are available, a GRNN that approximates theandcan be constructed.
relationship between

Figure 4. Neural network for deriving the closing distance of the gripper
based on the image from the back camera at initial grasp

Figure 5. Motion generator for stand-up manipulation

B. Stand-up manipulation
We present a gripper motion generator whose input is the
IR images of the back camera, based on the concept of the key
state. In the case of the gripper, the key state isstraightforwardly defined to be the state where the type of thegripper motion command changed. From the sequential dataof images and gripper motion commands obtained fromhuman demonstration, the data at the key states were selected.
The only this selected data was utilized to learn the procedure
for the complex manipulation, wherein the object is made tostand up via reiterated gripper motions. With this approach,the problem was simplified, and the number of learning targets
was reduced substantially.
The problem is to develop a motion generator that

repeatedly provides the gripper motion commands at the key
states according to the image input such that the object finallyattains the targeted standing posture.
The gripper motions were classified into four; moving
down, moving up, closing the gripper, and opening the gripper.
(4)
,
,
,
represents the classified gripper motion, where∈, ,,takes 1 or 0, and ∑1 . Forexample, the gripper-closing motion is expressed by
1 implies that the gripper motion is limited
0,0,1,0 . ∑to one of the four motions at a time; combinations of the
motions were not considered. The key state is then defined as
the state whenchanged, namely, the commanded gripper
motion was completed, and the subsequent command wascommunicated to the gripper. By controlling the gripper atonly the key states, unexpected object motions can beaddressed. According to the states of the object and gripper
after the completion of motion, the subsequent motion isdetermined. If the object motion was an unexpected one, thesubsequent motion could be a recovery motion. Therefore, theobject attains the targeted posture notwithstanding whether thegenerated motions include unexpected ones, and robust
manipulation to error in state estimation was obtained.
The motion generator was constructed mainly using feature
extraction and the output layers, as illustrated in Fig. 5. To
generate sequential motions, the generator utilized the IR
image
and the previous gripper motion type
. By
, the generated subsequent motion can be
feedbacking thedifferent from the previous one, and a recovery from an
unexpected object state can be expected. Therefore, thefeatures for the combination of theand for the state
resulting from the, were extracted. The dimensions ofandexhibit significantly high variations. Then, thefeatures were extracted from the combination ofand ,which is the low-dimensionalized image feature expressed inequation (2):
(5)
∈
∈
was
By utilizing the autoencoder, the feature vector
derived:(6)
where FEE (feature extraction encoder) denotes the encoder
part of the autoencoder.
With the feature vector, three types of networks wereconstructed for the three types of outputs. The first output (endjudgment) network judged whether the object attained thefinal targeted posture. The softmax function was used for the
network:
exp
1,2
(7)
∑ exp
and
are the network parameters, which are
whereobtained through learning. One of the output neurons provides
the probability of completion, while the other provides that ofbeing ongoing. If the judgement was ongoing, then the second

output (motion judgement) network was activated, and an
appropriate type of subsequent gripper motion was determinedaccording to the current states of the object and gripper.
, it was derived
Considering the discreteness of the output
using softmax function. The operation amount for the selected) was derived atgripper motion (indicated by the output
the last network. As a continuous value was to be derived,
GRNN was utilized. Regression was conducted for eachgripper motion type, and then, GRNN for each type (fourGRNNs in total) was prepared. The GRNN whose
corresponding component of the output
was 1 wasselected, and the corresponding operation amount was derived.
The overall network structure is illustrated in Fig. 5. A
noteworthy feature of the gripper motion is that only type
was fed back, while its operation amount was not.
The data for training is expressed by
(8)
, ,
,
is the image input captured by the back camera,
where,
expresses whether the object attained thefinal posture ( 0, 1 ) or not ( 1, 0 ),is the gripper
motion type that provided the state , and is its operation
amount.
C. Rotation manipulation
Utilizing gravity, the object was rotated by 90° around thegrasping points such that the object stands up in thelongitudinal direction. The problem is to explore the optimal
grasping points that provide rotation angle of 90°, avoiding
falling down of the object. After the rotation, by making theobject contact with the table, the object posture got to either
the initial or desired one. Therefore, repeated exploringmanipulations from the same initial object posture can be
adopted, and a parameter-online-exploration based approachis preferable.
Here, the following manipulation was considered:
Step 1: The gripper was closed, and the target object was
grasped and lifted up.
Step 2: The opening action of the gripper was continued until
the object’s rotation was detected. The opening action stopped
when the rotation started. The rotation occurs owing to themoment resulting from the gravitational force and stops owing
to contact friction.
In step 2, the opening action of the gripper could becontinued until the rotation angle reached 90°. However, thismethod does not ensure that the rotation angle reaches 90° (orthe angle close to 90°). On the other hand, different grasping
points provide varying moment and rotation angles, as
illustrated in Fig. 6. We then took the above way. Note thatthe rotation angle was derived from the image captured by theside camera.
Here, we consider the case illustrated in Fig. 6. Let be therotation angle of the object, be the moment of inertia at therotational center, be the length from the rotational center tothe center of gravity,
be the mass of the object, be theacceleration due to gravity, be the normal component of thegrasping force,be the coefficient of viscoelastic frictional
moment, and be the coefficient of frictional moment. Then,the dynamics of the object is represented by [21]:cos(9)
The relationship appears convenient to apply. However,owing to the softness of the gripper and unevenness of theobject surface,and are not constant:Figure 6. Mechanical relationship for the rotation manipulation
Algorithm 1: Safe Bayesian Optimization for the rotation manipulation
Domain: , Constants: , ,Lowest function value:0
Try the manipulation at , and observe the rotation angle
1and evaluate
For : n 2,3, ⋯2Update , , ,
3
∈ |
0
4
∈
′
∈
Initial value & Definitions

′∈ \

5
6
7
8

∈ |
argmax
∈

,

,

′

0

∪

Attempt the manipulation at, and evaluate, observe the rotation angle

End
Figure 7. Algorithm of Safe Bayesian Optimization for exploring thegrasping point maximizing function (Eq. (11)), which provides a rotation
angle approximately equal to 90°, avoiding falling down of the object

Bayesian optimization [33], [34] aims to obtain an input
value that maximizes an unknown function through several
explorations. Bayesian optimization uses a Gaussian process
to approximate the function based on past observation values,
and predicts the input, which is an optimal exploration target
to maximize the function. By repeating exploration with theBayesian optimization, the optimal input, which maximizes
the function with a smaller number of the explorations thanrandom trials, can be identified. In this study, the exploration
domain was defined for the grasping points , and the rotation
angle at the grasping points was utilized to construct thefollowing criterion function:|(11)
90 |90The optimized grasping points , which maximize andprovide a rotation angle close to 90°, is obtained as follows:
Let be an exploration domain for the input vector ∈ .
The i th observation value is expressed by(with noise(= 0.05 in the system inthis study)), which is provided by the image from the sidecamera. After the th observation, the Gaussian processestimates a function output for an arbitrary with a form ofGaussian whose mean isand variance is
:
(12)
(13)
,
is the covariance
where is the covariance function,
vector,is the kernel matrix,
is the vector of theobserved function values, and is the identity matrix:
,

exp

2

,

,…,

,

,

,

,

,…,
The upper and lower confidence bounds are expressed by(14)(a) Initial input(b) Third input: selected by visual information(c) Optimized input

Figure 8. Schematic illustration of Safe Bayesian Optimization

(10)
, , ,
, ,
where denotes the grasping points. It is challenging toderive an explicit equation (10). Therefore, instead of
considering equation (10), theproviding= 90° isidentified by manipulations through exploration. In order toavoid falling down, Safe Bayesian Optimization [36][39] wasadopted. By describing the risk of falling down as a constraint
for a criterion function, the manipulation with the exploration
was repeated until the manipulation succeeded, completely
circumventing the risk of falling down.

(15)
where
0 is the confidence interval. A larger enhances
the exploitation performance, while a smaller enhances the
exploration performance.
Safe Bayesian Optimization limits the exploration domain
to the safe one and prevents the system state failures fromisbeing unsafe. Concretely, the set of the safe region
expressed by
(16)
∈ |
where
is the safe threshold for the function value). In
this study, we set
0 corresponding to no-rotation,
which indicates falling down, and avoided the falling down
event during the exploration. The subsequent exploration was
conducted atargmax
∈

∪

(17)

where
provides the high-probability bounds for
maximizing the function on , andprovides domains
which are newly judged to be safe on . The exploration
procedure is illustrated in Fig. 7, and Fig. 8 illustrates theschematic of the procedure. Note that , ,is the lower
confidence bound calculated using equation (15) based on pastdata and data that is artificially constructed by the input ofand output of.
As the target is a rectangular object and as the gripper isused (see Fig. 9), the y and z coordinate values form the
exploration target and are identical for both the fingertips.
Then, the dimension of the exploration domain was two, andthe grasping points are expressed as, . It isnoteworthy that the domain included only the upper area,
where it was feasible for the gripper to make contact, as
illustrated in Fig. 9. It was not feasible for the gripper to makecontact with the lower area owing to the presence of thefingernail.
In actual situations, the initial grasping points are to bedeliberately selected, and for preventing falling down, they are
not to be the gravitational center. Based on the image from theside camera, which is exhibited in Fig. 10, the initial grasping
was determined bypoints
(18)
1.5
,
where
argmax| |(19)

methodology (Table I). The total number of training data was1173, as can be observed in Table I. The number of image
frames for each manipulation was an average of 600 (= 30 s
20 fps). Of the entire data (1173 / (600 15)), 13 % isrequired; this validated that the proposed methodology
simplified the problem.

Figure 9. The exploration domain and the rotation angle

Figure 10. Initial grasping points∈ ,The other notations are exhibited in Fig. 10.
Additionally, for90°, a contact between the object andthe table could provide
90° if was in the proximity of90°. Although the explored input in this case was not optimal,
the targeted posture was obtained. In this case, we finished themanipulation by judging that the manipulation was completed.
The overall procedure is illustrated in Fig. 11.
IV. EXPERIMENTAL VALIDATIONS
A. Initial grasp and stand-up manipulation
Here, both the initial grasp and stand-up manipulation were
evaluated simultaneously. The objects during the training
were PTP packaging sheet (No. 1 in Fig. 12), the sheets
painted with the four patterns (Nos. 2–5), and the cut half sheet(No. 6). The test objects were the sheet with a dissimilar
pattern (No. 7 in Fig. 12), tablet pillbox (No. 8), another PTP
packaging sheet (No. 9), and business card (No. 10). Table I
presents the number of training data for the stand-up
manipulation. While constructing , the labels Down, Up,
Close, and Open were treated as Ongoing. The parameter
0.3 was set for all GRNNs by adopting grid search to thetraining data. Table II presents the results. Both the types ofmanipulation were successful for all the trained and test
objects except for the card. The failure of the manipulation
was due to the card’s substantially low thickness, whichcaused the misjudgment of completion (Fig. 13). An example
of recovery motions can be observed in the attached video,
wherein additional opening and closing motions were
implemented to obtain an expected object state from an
unexpected object state, which had resulted from wrong
motion due to wrong state estimation. In total, five recovery
motions were observed at all the examinations. The small
number of training data was also one of the features of thisFigure 11. Procedure for the rotation manipulation
TABLE I.

NUMBER OF TRAINING DATA FOR THE STAND-UP
MANIPULATION

Label
Number of data

Down
352

Up
223

Close
478

Open
120

End
166

Figure 12. Target objects for the experimental evaluation of the initial grasp
and the stand-up manipulation. The objects from No. 1 to 6 were for training
while the other objects were for testing.
SUCCESS RATE OF INITIAL GRASP AND STAND-UP

TABLE II.

MANIPULATION

TrainedObjectnumberSuccess
rate

Test

1

2

3

4

5

6

7

8

9

10

3/3

3/3

3/3

3/3

3/3

3/3

3/3

3/3

3/3

1/3

Figure 13. Successful grasping of the PTP packaging sheet and the failure ofgrasping of the thin business card

B. Rotation manipulation
The methodology was evaluated using the box with six
pockets (70 50 16.5 mm, 18.8 g), which is illustrated inFig. 14. The gravitational center was altered by placing aweight (5g) in one of the pocket. The exploration domain was

0
70
(20)
0
30
The interval for the exploration was 2.0 mm for each axis. The
manipulation was completed when |J 90| 5 . The
parameter
7.07 in the covariance function and = 60 in
the confidence bounds were set by adopting grid search at thepractice phase utilizing the PTP sheet (No. 1 in Fig. 12).
The results when the weight was in pocket 2 aresummarized in Table III and Fig. 15. Table IV presents theresults with the weight placed in each of the six pockets, one
pocket at a time. It is observed that the methodology worked
effectively. It is noteworthy that there was no falling during
any of the exploration manipulations.
,C. Combination of all manipulations
The overall manipulation was conducted by combining thethree manipulations. The corresponding result for the table

Tablet pill box
Initial grasp

Stand-up manipulation

Rotation manipulation

TABLE III.

THE OBTAINED PROCESS RESULTS WHEN THE WEIGHT WAS
IN POCKET 2
1
2
3
4
5
6

[mm]
[mm]

52
24
78.0

60
30
73.8

60
18
59.7

50
30
39.0

44
14
21.3

58
24
86.8

Figure 15. The explored grasping points and the corresponding criterion
function values when the weight was in pocket 2
TABLE IV.
Weightposition
[No]
1
2
3
4
5
6

RESULTS OF THE CASES WITH THE WEIGHT IN EACH OF THE
SIX POCKETS, ONE AT A TIME.
Optimized
Trial
Success or
Numbergrasping points
numbernotof trial
[mm]
[mm]
1st
Success
260
28
2ndSuccess
6
58
26
3rdSuccess
2
60
28
1stSuccess
6
58
24
2ndSuccess
6
56
24
3rdSuccess
7
58
24
1stSuccess
1
52
24
2ndSuccess
1
52
24
3rdSuccess
1
52
24
1stSuccess
2
6230
2ndSuccess2
62
28
3rdSuccess
2
62
38
1stSuccess
4
48
30
2ndSuccess
6
6228
3rdSuccess
8
52
30
1stSuccess
3
6230
2ndSuccess
2
62
26
3rdSuccess2
60
28

Motion
Down
Close
Up
Close

1st
2nd
3rd
4th

40
32
70
82

9.6
Operationamount [mm]
16.20
8.30
11.65
4.27
,
[58,24][52,18][52,30][42,26]

Figure 16. The result of the combined manipulation for the tablet pill box
TABLE V.

Success
rate

Figure 14. The target object for evaluating the rotation manipulation: a box
with six pockets for varying the gravitational center.

Closing distance [mm]Operation
number
1st
2nd
3rd
4th

SUCCESS RATE OF THE COMBINED MANIPULATION

PTP
packagingsheet (No. 1)

Tabletpillbox(No. 8)PTP packaging
sheet (No. 9)

Business card
(No. 10)

3/3

3/3

3/3

1/3

pillbox is presented in Fig. 16. Table V presents the results forthe four objects. It is observed that the combined manipulation
was successful for all the objects except for the business card.
The reason for the failure was similar to those presented inTable II. It is noteworthy that the success rate of rotation
manipulation for the business card was 3/3. The attached video
displays the successful combined manipulation of several
objects.
V. CONCLUSION
This study presented novel methodologies for dexterous
manipulation (Fig. 2) using under-actuated robotic soft
gripper with a low DOF developed previously by the authors(Fig. 1). In order to achieve dexterous manipulation, theutilization of an environment is key. The target manipulations
utilizing a support surface and gravity were defined. Throughrealization of the target manipulations, the methodologies forutilizing the support surface and gravity in manipulation by asoft robotic hand were revealed. In order to deal with thesoftness and passivity of the gripper in the target manipulation
utilizing the support surface, the key states where the type ofgripper motion primitive changed were focused on, and astrategy for controlling the gripper at only the key states byusing a DNN based motion generator, was presented. The
motion generator was highly robust, and it was capable ofaddressing unexpected object motion and generating recovery
motion. In the manipulation utilizing gravity, a parametersonline-exploration based approach was adopted to prevent
falling down of the object. By formulating the fall risk as aconstraint for a criterion function, the manipulation with
exploration was repeated until the manipulation was
successful, completely avoiding the fall risk. All themethodologies were combined, and the effectiveness of theobtained
methodologies
wasdemonstratedbyexperimentation.
The targeted manipulation was set considering thefeasibility of conducting complex manipulations by using softrobotic hands with a low DOF, such as the soft gripper
developed previously by the authors, and whether it is feasible
to realize its atomization. The strategy of the approach basedon key state could be adopted to various manipulations byother soft robotic hands, with appropriately defining the key
states. Challenges in other complex manipulations andadaptation of the approach investigated in this study to otherrobotic systems are planned to be explored in future studies by
the authors. Moreover, the authors intend to extend the set ofavailable object types through their future studies, including
the business card, whose manipulation was not completely
successful in this study.
REFERENCE[1]

T. Nishimura, K. Mizushima, Y. Suzuki, T. Tsuji, and T. Watanabe,“Variable-grasping-mode underactuated soft gripper withenvironmental contact-based operation,” IEEE Robot. Autom. Lett., pp.
1–1, 2017.
[2] R. Maruyama, T. Watanabe, and M. Uchida, “Delicate grasping by
robotic gripper with incompressible fluid-based deformablefingertips,” in Proc. 2013 IEEE/RSJ Int. Conf. on Intell. Robots Syst.,
2013, pp. 5469–5474.
[3] R. Adachi, Y. Fujihira, and T. Watanabe, “Identification of danger
state for grasping delicate tofu with fingertips containing viscoelastic
fluid,” in Proc. 2015 IEEE/RSJ Int. Conf. on Intell. Robots Syst., 2015,
pp. 497–503.
[4] T. Nishimura, Y. Fujihira, R. Adachi, and T. Watanabe, “New
Condition for Tofu Stable Grasping with Fluid Fingertips,” in Proc.
2016 IEEE Int. Conf. on Autom. Sci. Eng., 2016, pp. 335-341.
[5] J. M. Elliott and K. J. Connolly, “A CLASSIFICATION OF
MANIPULATIVE HAND MOVEMENTS,” Dev. Med. Child Neurol.,vol. 26, no. 3, pp. 283–296, Nov. 2008.
[6] I. M. Bullock, R. R. Ma, and A. M. Dollar, “A Hand-CentricClassification of Human and Robot Dexterous Manipulation,” IEEE
Trans. Haptics, vol. 6, no. 2, pp. 129–144, Apr. 2013.
[7] M. Controzzi, C. Cipriani, and M. C. Carrozza, “Design of Artificial
Hands: A Review,” 2014, pp. 219–246.
[8] R. Deimel and O. Brock, “A compliant hand based on a novel
pneumatic actuator,” in Proc. 2013 IEEE Int. Conf. on Robot. Autom.,
2013, pp. 2047–2053.
[9] R. Deimel and O. Brock, “A novel type of compliant andunderactuated robotic hand for dexterous grasping,” Int. J. Rob. Res.,
vol. 35, no. 1–3, pp. 161–185, Jan. 2016.
[10] A. Dameitry and H. Tsukagoshi, “Lightweight UnderactuatedPneumatic Fingers Capable of Grasping Various Objects,” in Proc.
2016 IEEE Int. Conf. on Robot. Autom., 2016, pp. 2009–2014.
[11] M. Tavakoli and A. T. de Almeida, “Adaptive under-actuatedanthropomorphic hand: ISR-SoftHand,” in Proc. 2014 IEEE/RSJ Int.
Conf. on Intell. Robots Syst., 2014, pp. 1629–1634.
[12] E. Brown, N. Rodenberg, J. Amend, A. Mozeika, E. Steltz, M. R.
Zakin, H. Lipson, and H. M. Jaeger, “Universal robotic gripper basedon the jamming of granular material,” Proc. Natl. Acad. Sci., vol. 107,
no. 44, pp. 18809–18814, Nov. 2010.
[13] J. R. Amend, E. Brown, N. Rodenberg, H. M. Jaeger, and H. Lipson,“A Positive Pressure Universal Gripper Based on the Jamming of
Granular Material,” IEEE Trans. Robot., vol. 28, no. 2, pp. 341–350,
Apr. 2012.
[14] M. G. Catalano, G. Grioli, E. Farnioli, A. Serio, C. Piazza, and A.
Bicchi, “Adaptive synergies for the design and control of the Pisa/IIT
SoftHand,” Int. J. Rob. Res., vol. 33, no. 5, pp. 768–782, Apr. 2014.
[15] G. Grioli, M. Catalano, E. Silvestro, S. Tono, and A. Bicchi, “Adaptivesynergies: An approach to the design of under-actuated robotic hands,”
in Proc. 2012 IEEE Int. Conf. Intell. Robot. Syst., 2012, pp. 1251–
1256.
[16] M. Ciocarlie, F. M. Hicks, R. Holmberg, J. Hawke, M. Schlicht, J.
Gee, S. Stanford, and R. Bahadur, “The Velo gripper: A versatile
single-actuator design for enveloping, parallel and fingertip grasps,”
Int. J. Rob. Res., vol. 33, no. 5, pp. 753–767, Apr. 2014.
[17] J. C. Trinkle and R. P. Paul, “Planning for Dexterous Manipulation
with Sliding Contacts,” Int. J. Rob. Res., vol. 9, no. 3, pp. 24–48, 1990.

[18] C. Eppner, R. Deimel, J. Álvarez-Ruiz, M. Maertens, and O. Brock,“Exploitation of environmental constraints in human and robotic
grasping,” Int. J. Rob. Res., vol. 34, no. 7, pp. 1021–1038, Jun. 2015.
[19] Y. Aiyama, M. Inaba, and H. Inoue, “Pivoting: A New Method of
Graspless Manipulation of Object by Robot Fingers,” in Proc. 1993
IEEE/RSJ Int. Conf. Intell. Robot. Syst., 1993, pp. 136–143.
[20] P. Tournassoud, T. Lozano-Perez, and E. Mazer, “Regrasping,” inProceedings. 1987 IEEE Int. Conf. on Robot. Autom., 1987, vol. 4, pp.
1924–1928.
[21] A. Holladay, R. Paolini, and M. T. Mason, “A general framework foropen-loop pivoting,” in Proc. 2015 IEEE Int. Conf. on Robot. Autom.,
2015, pp. 3675–3681.
[22] N. Chavan-Dafle and A. Rodriguez, “Prehensile pushing: In-handmanipulation with push-primitives,” in Proc. 2015 IEEE Int. Conf.
Intell. Robot. Syst., 2015, pp. 6215–6222.
[23] N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa, M.
Erdmann, M. T. Mason, I. Lundberg, H. Staab, and T. Fuhlbrigge,“Extrinsic dexterity: In-hand manipulation with external forces,” inProc. 2014 IEEE Int. Conf. on Robot. Autom., 2014, pp. 1578–1585.
[24] F. E. Vina B., Y. Karayiannidis, C. Smith, and D. Kragic, “Adaptive
control for pivoting with visual and tactile feedback,” in Proc. 2016
IEEE Int. Conf. on Robot. Autom., 2016, pp. 399–406.
[25] F. E. Vina B., Y. Karayiannidis, K. Pauwels, C. Smith, and D. Kragic,“In-hand manipulation using gravity and controlled slip,” in Proc.
2015 IEEE/RSJ Int. Conf. on Intell. Robots Syst., 2015, pp. 5636–5641.
[26] K. Noda, H. Arie, Y. Suga, and T. Ogata, “Multimodal integration
learning of robot behavior using deep neural networks,” Rob. Auton.
Syst., vol. 62, no. 6, pp. 721–736, Jun. 2014.
[27] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning HandEye Coordination for Robotic Grasping with Large-Scale Data
Collection,” in Proc. .Int. Sympo. on Exp. Robot., 2017, pp. 173–184.
[28] A. Gupta, C. Eppner, S. Levine, and P. Abbeel, “Learning dexterous
manipulation for a soft robotic hand from human demonstrations,” inProc. 2016 IEEE/RSJ Int. Conf. on Intell. Robots Syst., 2016, pp.
3786–3793.
[29] C. Finn, Xin Yu Tan, Yan Duan, T. Darrell, S. Levine, and P. Abbeel,“Deep spatial autoencoders for visuomotor learning,” in Proc. 2016
IEEE Int. Conf. on Robot. Autom., 2016, pp. 512–519.
[30] X. Yin and Q. Chen, “Deep metric learning autoencoder for nonlinear
temporal alignment of human motion,” in Proc. 2016 IEEE Int. Conf.
on Robot. Autom., 2016, no. 1, pp. 2160–2166.
[31] M. F. Møller, “A scaled conjugate gradient algorithm for fast
supervised learning,” Neural Networks, vol. 6, no. 4, pp. 525–533, Jan.
1993.
[32] B. A. Olshausen and D. J. Field, “Sparse coding with an overcomplete
basis set: A strategy employed by V1?,” Vision Res., vol. 37, no. 23,
pp. 3311–3325, Dec. 1997.
[33] D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient Global
Optimization of Expensive Black-Box Functions,” J. Glob. Optim.,
vol. 13, no. 4, pp. 455–492, 1998.
[34] E. Brochu, V. M. Cora, and N. De Freitas, “A tutorial on Bayesian
optimization of expensive cost functions, with application to active
user modeling and hierarchical reinforcement learning,” ArXiv, p. 49,
2010.
[35] J. Nogueira, R. Martinez-Cantin, A. Bernardino, and L. Jamone,“Unscented Bayesian optimization for safe robot grasping,” in Proc.
2016 IEEE/RSJ Int. Conf. on Intell. Robots Syst., 2016, pp. 1967–1972.
[36] F. Berkenkamp, A. P. Schoellig, and A. Krause, “Safe controller
optimization for quadrotors with Gaussian processes,” in Proc. 2016
IEEE Int. Conf. on Robot. Autom., 2016, pp. 491–496.
[37] S. Bai, J. Wang, F. Chen, and B. Englot, “Information-theoretic
exploration with Bayesian optimization,” in Proc. 2016 IEEE/RSJ Int.
Conf. on Intell. Robots Syst., 2016, pp. 1816–1822.
[38] O. Andersson, M. Wzorek, P. Rudol, and P. Doherty, “Modelpredictive control with stochastic collision avoidance using Bayesianpolicy optimization,” in Proc. 2016 IEEE Int. Conf. on Robot. Autom.,
2016, pp. 4597–4604.
[39] Y. Sui, A. Gotovos, J. Burdick, and A. Krause, “Safe Exploration for
Optimization with Gaussian Processes,” Proc. 32nd Int. Conf. Mach.
Learn., vol. 37, pp. 997–1005, 2015.
[40] Philip D. Wasserman, Advanced Methods in Neural Computing. 1993.