Journal of Robotics, Networking and Artificial Life, Vol. 1, No. 4 (March 2015), 257-260

Vision-Based Grasp Planning and Experiments of a Mobile Manipulator
Yi-Fu Chiu

Institute of Electrical Control Engineering, National Chiao Tung University
1001 Ta Hsueh Road, Hsinchu 30010, Taiwan
Kai-Tai SongInstitute of Electrical Control Engineering, National Chiao Tung University
1001 Ta Hsueh Road, Hsinchu 30010, Taiwan
E-mail: ivan790721.ece01g@nctu.edu.tw, ktsong@mail.nctu.edu.tw

Abstract
In this paper, a motion planner is designed and implemented for of a mobile manipulator to travel to a spot forgrasping of an object. In this work, the probability of successful grasping inside the workspace of the robot arm isused for grasping planning. A vision SLAM system is combined with reachability calculation to figure out thegrasping position. Using a laboratory dual-arm robot, we conducted experiments in different conditions to verify
the effectiveness of the developed system.
Keywords: Mobile manipulation, visual servoing, visual navigation, robot grasping.

1.

Introduction

Mobile manipulation technologies have been developed
rapidly in recent years. It has been expected that variousrobots will come into our everyday life for living aids.
In order to help people with housework, a robot needs tobe equipped with many abilities such as communication,navigation, grasping and object recognition. Therefore,
developing a mobile manipulator which is able to moveand handle objects such that it can support people in ahome setting deserves urgent attention.
Cosero1hasachieved
notablesuccessin
RoboCup@Home contests for its mobile manipulation
performance. It localizes objects and plans path basedon a 2D occupancy grid map constructed from multiple3D scans of the environment2. Collision with obstacles
can be avoid using the map, even though it cannot
perceive an obstacle with current sensor’s view. PR23
processes sensory data from 3D visual sensors in point
clouds. Multilayered 2D costmaps and a layered
representation of the robot body are used to reduce thepossibility of collision4. In the work of Stulp et al.5,
through experience-based learning, the robot first learns
a so-called generalized success model which
distinguishes among positions from which manipulation
may succeed or fail. The model is used to compute a
probability distribution that maps positions to a
predicted probability of successful manipulation, taking
into account the uncertainty of the robot and object’s
positions.
A key factor of mobile manipulation design is that asuitable pose of the mobile robot needs to be obtained
through navigation in order to conduct object
manipulation. One possibility is simply to travel fromthe current position to a position such that the target
object is reachable, as depicted in Fig.1. As shown in
Fig 1(a), the object is placed in front of the table, wherean obstacle exists to influence the grasping. In this case,the object might not be successfully grasped due to thelimited workspace of the robot arm. However, if themobile robot travels to the side of the table as shown in
Fig. 1(b), the object can be easily grasped. It is therefore

Published by Atlantis Press
Copyright: the authors257

desirable to develop a method to plan a pose for therobot such that the target object is easy-to-reach.
Attamimi et al.6 proposed a method to use perception
maps that express the robot arm reachability and robot
platform navigation reachability to a target object. Theauthors also show that by integrating these two maps, aproper position that has maximum probability ofsuccessful grasping is determined. In this study, wedeveloped a motion planning system for determining theproper pose for grasping objects. The robot position isdetermined based on the arm’s reach and ease ofmovement. We adopted the design of Attamimi et al.6 todescribe the limitations of robot workspace, thesituation of environment and target object position. By
constructing the reachability map that represents theease of robot mobile platform motion and thegraspability map that describes the success rate ofgrasping of robot arms in workspace, robot is able todetermine the proper position to move toward andincrease the success rate of mobile manipulation task.
Visual EFK-SLAM and visual servo technics areintegrated into the design for vision-based grasping of
the object.
2.

The Dual Arm Robot

A dual-arm robot is designed and constructed for mobile
manipulation experiments. The 6-DOF dual-arm design
allows the robot to grasp various objects in theenvironment. A Kinect RGBD camera is installed on
top of the robot for visual servo control. The forward
and inverse kinematics of two robot arms are derived
and implemented on the robot controller. Two laser
range finders are equipped on the mobile platform inforward and backward directions for obstacles
detection. An on-board industrial PC (IPC) is used fordata processing and motion control of the robot. The(a)(b)
Fig.1 An example of mobile manipulation is influenced byan obstacle in front of the target.

steer- and-drive mechanism allows the robot to move in
omnidirection7. The implemented EKF-SLAM8 allows
the robot to navigate and localize itself in theenvironment. The visual servo system9 allows the robot
to move and fetch an object without collisions.
A. Visual EKF-SLAM
A visual SLAM system has been developed for theomnidirectional mobile robot8. There are three main
steps of the EKF-SLAM: a measurement step to acquire
the location vector of robot relative to the feature in theenvironment, a prediction step to predict the location ofrobot using motion model and an update step tocalculate Kalman gain and update robot stare according
to above two steps. Then, the updated robot state isdelivered to prediction step to calculate the next
moment state of robot. These steps repeat to achieve thereal time localization and mapping.
B. Visual servo grasping
By using the depth data from Kinect, the input image issegmented into several planes. In this method9, theplane that the target object locates is first labeled. The
labeled plane is featured by SURF algorithm and ismatched with data base to find where the target object
is. Except for the target object, the other objects in theenvironment are obstacle for robot manipulation.
Potential field is then combined gravity and repulsion
with these indices for safe path planning of robot arms.
According to this, the robot is able to move its arms in
the workspace without collisions and to grasp thetargeted object.
3.

Grasp Planning

The proposed system architecture is shown in Fig. 2.
For representation of the environment, we utilize EKFSLAM algorithm and Kinect sensor to construct afeature map of the environment. For grasping planning,
we adopted the method of Attamimi et al.6 in this design.
The feature map is used as input to the mobile platform
to calculate the easiness of traveling to the reachable
region of the robot. On the other hand, the target
location is inputted to arm graspability calculation torepresent the extent of graspable region of robot arms.
We fuse the two representations to decide the goal
location for robot to grasp the target object. This work
uses EKF-SLAM algorithm for robot navigation

Published by Atlantis Press
Copyright: the authors258

proposed system. We first verify that different locationsof target object in the same environment will result indifferent robot navigation behavior. Then we add anobstacle around the target area to verify the adaptationof the robot.

A. Target object in different locations

Fig.2 System architecture

control. First, we extract SURF features that represent
the configuration of the environment. Then, EKF isutilized to improve the localization error and to correct
the coordinates of features. For grasping, we use Kinect
for environment detection so that the robot is able todetect not only the target object but also the obstacle in
the environment and to guide its arms to fetch the target.
The inverse kinematics solution map suggested in thework of Attamimi et al.6 is adopted to calculate
reachable/unreachable points. Then a probability
distribution is given to all reachable points for later
decision making of the grasping position.
To include the description of robot navigation in themotion planning of grasping, we have to consider thenavigation control of the mobile robot in anenvironment. Obstacles in the environment influence
the motion planning of the robot. In this work, a featuremap is built to express the environment by using EKFSLAM algorithm. We adopted the method proposed by
Attamimi et al.6 to construct a reachability map for
motion planning. The closer to obstacles, the lessprobability is given for the robot to reach. The motion
planning of the robot is then determined for the robot tomove to the maximum probability of the integrated
representation of the navigation and arm motion6.
4.

Experiments and discussion

Three interesting experiments have been carried out for
grasp planning to verify the effectiveness of theFig. 3 shows the environment of this experiment,Condition A and Condition B are termed to distinguish
two locations of the object on the table. We first utilize
EKF-SLAM to construct the feature map of theenvironment. Then we replace the features with the sizeof robot. Fig. 4(a) shows the planning result ofcondition A. It is noted that the planned grasping
position is in front of the table, which is depicted indotted line. For condition B, the planned grasp positionis at the side of the table, as shown in Fig. 4(b). Fig. 5
shows the experiment by real robot in Condition A andFig. 6 shows the experiment in Condition B.

B. Grasping planning affected by obstacles
The purpose of this experiment is to verify the proposed
method is able to judge the proper grasp position under
the obstacle situation. Fig. 7 (a) shows the setup of thisexperiment, termed Condition C. The target object is at
the same position as Condition A but is behind theobstacle. The planning result is shown in Fig. 7(b). The
graspable region moves to the side of the table becauseof the graspability affected by the obstacle changes.
Compare with Condition A, the difference is theobstacle effect that causes the different planning fromproposed method.

(a)(b)
Fig. 3 Environment setup for experiments (a) Condition A,
target in the front of the front of the table, (b) Condition B,
target on the side of the table.

Published by Atlantis Press
Copyright: the authors
259

(a)
(b)
Fig. 4 (a) The planning result of Condition A, (b) theplanning result of Condition B.

(a)

(b)

(c)

Fig. 5 Task execution of Condition A on real robot

(a)

(b)

(c)

Fig. 6 Task execution of Condition B on real robot

5.

Conclusions and Future WorkA dual-arm robotic system has been designed andimplemented for mobile manipulation tasks in anunstructured environment. A grasp planning system isdeveloped based on visual SLAM and visual servoing of
the robot. The proposed system has been verified inthree different conditions. The experimental results
show that the robot can adapt to the variation ofenvironment and find a suitable place to grasp theobject. In the future, object fetching in more extensive
environments will be studied.
Acknowledgements
This work was partly supported by Ministry of Scienceand Technology of Taiwan, under grant NSC 102-2221E-009-140.

(a)(b)
Fig. 7 (a) Environment setup of Condition C (b) theplanning result of Condition C.
2. J. Klaess, J. Stuckler and S. Behnke, “Efficient Mobile
Robot Navigation Using 3D Surfel Grid Maps,” in Proc.
of 7th German Conf. on Robotics (ROBOTIK), (Munich,Germany, 2012), pp. 1-4.
3. Sachin Chitta, E. Gil Jones, Matei Ciocarlie and Kaijen
Hsiao, “Mobile Manipulation in Unstructured
Environments: Perception, Planning, and Execution,” inIEEE Robotics & Automation Magazine, 19(2012), 5871.
4. Hornung, A., Phillips, M., Jones, E.G., Bennewitz, M.,
Likhachev, M. and Chitta, S., “Navigation in ThreeDimensional Cluttered Environments for Mobile
Manipulation,” in Proc. of 2012 IEEE International
Conference on Robotics and Automation, (Minnesota,USA, 2012), pp. 423–429.
5. Freek Stulp, Andreas Fedrizzi and Michael Beetz,“Action-Related Place-Based Mobile Manipulation,” inProc. International Conference on Intelligent Robots andSystems, (St. Louis, MO, USA, 2009), pp. 3115-3120.
6. Muhammad Attamimi, Keisuke Ito, Tomoaki Nakamura
and Takayuki Nagai, “A Planning Method for Efficient
Mobile Manipulation Considering Ambiguity,” in Proc.
International Conference on Intelligent Robots andSystems, (Vilamoura-Algarve, Portugal, 2012), pp. 965972.
7. Kai-Tai Song, Sin-Yi Jiang and Chia-Chang Wu, “PoseControl of a Four-Wheel Drive Omni-Directional Mobile
Robot,” in Proc. of 2012 CACS International AutomaticControl Conference, ( Yunlin, Taiwan, 2012).
8. Chien-Hung Liu and Kai-Tai Song, “A New Approach to
Map Joining for Depth-Augmented Visual SLAM,” inProc. of ASCC 2013, (Istanbul, Turkey, 2013).
9. Kai-Tai Song and Shih-Cheng Tsai, “Vision-Based
Adaptive Grasping of a Humanoid Robot Arm,” in Proc.
of IEEE ICAL 2012, (Zhengzhou, China, 2012), pp.160165.

References
1. Jorg Stuckler and Sven Behnke, “Benchmarking Mobile
Manipulation in Everyday Environments,” in Proc. of2012 IEEE Workshop on Advanced Robotics and its
Social Impacts (ARSO), (Munich, Germany, 2012), pp. 16.

Published by Atlantis Press
Copyright: the authors
260