Combining Finger Vision and Optical Tactile Sensing:
Reducing and Handling Errors While Cutting Vegetables
Akihiko Yamaguchi1 and Christopher G. Atkeson1

Abstract— We develop an optical multimodal-sensing skin forfingers of a robotic gripper. This sensor consists of a transparent
skin made with a marked soft elastic outer layer on a hardlayer, and internal RGB cameras. The cameras can see theexternal scene through the skin. This sensor provides a contact
force field estimated by tracking markers in the skin, and visual
information of close or grasped objects. This sensor uses off-theshelf materials, is easy to manufacture, and is robust against
an external force as the force is not applied to the sensing
devices (cameras). We install a prototype of this sensor on
fingers of a Baxter robot, and demonstrate its usefulness witha cutting vegetable task. The sensor gave us force information
to control cutting vegetables and avoiding slippage of the knife.
Accompanying video: https://youtu.be/ifOwQdy9gDg

Fig. 1. Conceptual design of our optical multimodal-sensing skin and theinstallation example on a robotic gripper.

I. I NTRODUCTION
We are exploring whole-body vision as a sensing skin forhuman safety and improved robot control. Different typesof sensors are useful to maintain human safety, such as
proximity sensors and contact sensors. These sensors are also
useful in robust robot control. For example in the DARPA
Robotics Challenge Finals, no robots used railings, walls,
door frames, or obstacles to support the robot body [1].
Whole-body contact sensors are useful to do this. We areproposing to cover the robot body with a transparent soft
material (skin) and put cameras inside it (Fig. 1(a)). Suchwhole-body vision will give us multimodal information suchas a proximity (by stereo vision), visual information (color,
texture), and contact force estimation (by tracking the skindeformation). This project is named “Argus”: Argus Panoptes
was a 100-eyed giant in Greek mythology.
This paper focuses on a sensing skin for robotic grippers(Fig. 1(b)). We design the sensing skin for manipulation tasks
as well as grasping. We focus on a cutting task with a knife.
During cutting, sometimes a strong force is applied to theknife when cutting a hard material, which causes the knife toslip or damage to the fingers. Sensing skin is useful in sucha situation providing contact force estimations for avoiding
slippage and damage to the fingers, and visual information
for slippage detection.
There are many similar approaches to make a tactile sensor
with a transparent skin and an imaging sensor inside (e.g.
[2], [3], [4], [5], [6], [7]), but all of them are focusing on
tactile sensing only, i.e. contact point location and/or contact
force/pressure. Most of them cover the skin surface with
opaque material to shut out external light. We think fullytransparent skin gives us richer information.
1 A. Yamaguchi and C. G. Atkeson are with The Robotics Institute,Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh PA 15213,
United States info@akihikoy.net

Fig. 2. Prototype of the optical multimodal-sensing skin on the fingersof the Baxter robot. Bottom-left: the sensing skin and the fingers. Topleft: grasping a knife with the sensing skin. Top-right: cutting an apple.
Bottom-right: a camera view during cutting where the marker movements
are rendered.

In this paper, we make a prototype of the sensing skin.
It consists of a camera, a transparent hard layer made withacrylic, a transparent soft layer made with silicone rubber,and colored markers embedded near or on the surface ofthe soft layer. The markers are used to help tracking theskin deformation. A computer vision method is developed
for tracking the markers in order to estimate contact forces.
We integrate the sensing skin on the gripper of a Baxter
robot, and investigate its usefulness in a practical situation,
cutting vegetables with a knife (Fig. 2).
The features of our sensing skin are as follows. It is
comparably low cost; it just uses off-the-shelf materials. It
is easy to make. The issue is size reduction; the bottle necks
are cameras and lenses. Recently small cameras are widelydeveloped for various purposes including smart phones. For
example in [8], a human fingertip-size device was made witha tiny camera. Our sensing skin is multimodal and it gives
us high resolution of contact forces and proximity vision. It
is physically robust because the sensing elements (cameras)
are separated from the skin. We assume that the sensing skinis installed on a robot by fixing the hard layer on a rigid partof the robot. Thus the external force is applied to the softand hard layers only, not to the cameras.

application tasks. The applications of optical tactile sensors
are: grasping task (e.g. [9]), estimating the surface shapeor edge of a contacting object (e.g. [13], [18], [14]), andmanipulation (e.g. rolling a cylinder [19]). In [10], the sensor
is used in a physical human-robot interaction task (passing an
object from the robot to a human). In [15], the sensor is usedin an insertion task of small parts (e.g. a USB connector intoa mating hole). We explore a flat surface and its usefulness
in a cutting task.

Related WorkII. O PTICAL M ULTIMODAL -S ENSING S KIN FOR ROBOTF INGERS

The idea of using imaging sensors for tactile sensing isdecades old. An initial attempt was measuring the frustrationof total internal reflection within a waveguide on a sensor
surface caused by contact [2], [3], [9], [10]. The research
trend has shifted to measuring displacement of markers
placed on the sensor surface with computer vision [4], [5],[11], [12], [13], [6], [14], [7]. A reason would be the marker
displacements are proportional to the external force as thedisplacements are directly caused by the external force.
The resolution of the contact force field is decided by thecamera resolution and the marker density. Recently highresolution sensors are proposed (e.g. [15], [7]). Another idea
is markerless localization of a registered object (e.g. [15]).
The localization accuracy is ideally equal to the camera
resolution. A drawback is the requirement of registering
an object. Much of the above work uses a transparent
elastic material between the sensor surface and the base. The
dynamic range of the force measurement can be controlled
by changing the hardness of the elastic material (softer ismore sensitive; cf. [16]).
Our research is close to this approach. An important difference is the transparency of the skin including the surface,which gives us richer and multimodal information. Previous
work uses opaque surfaces to block the external light as it
would affect marker tracking. We solve the marker tracking
problem under natural external lighting and backgrounds bycomputer vision in order to make use of fully transparent
skin.
Another sensor with transparent skin is proposed in [17]where distance sensors (range finders) are used instead of acamera. The idea is measuring distances between the sensors
and an object, and estimating the deformation of the transparent skin from the distance map. Normal contact forces are
estimated from the deformation. If there is no contact withan object, this sensor simply gives the distance to a close
object. Although this sensor and ours have different sensing
modalities and ranges, we would be able to share ideas; e.g.
we can embed distance sensors around the cameras.
Much related research uses a hemisphere shape for fingertip [3], [5], [11], [9], [18], [13], [6], [14], [7], [16], [10],[19]. Their intention is making fingertips of robotic hands orgrippers. More human-like fingertips are also proposed (e.g.
an elliptical shape [12]). The surface shape should be decided
by the task. Although there is research on other shapes
such as a flat shape (e.g. [2], [4], [15]), the surface shaperequirements have not been studied enough in the context of

A. Overview
The conceptual design of the optical multimodal-sensing
skin is shown in Fig. 1. Unlike other research [4], [6], [7], we
do not place an opaque material on the surface. The whole
skin is transparent except for the markers, and the cameras
can see the external scene through the skin.
The markers are captured by the cameras and tracked.
This gives us a 3-axis (x,y,z) force measurement at eachmarker point. By combining multiple marker measurements,
we can estimate torque information. In general, a bigger
marker is easier to detect; the marker size affects the accuracyof tracking. The density of the markers determines theresolution of the contact force field. There is a trade-off
between the resolution and the surface transparency. The
hardness and the thickness of the elastic layer affect themarker movement caused by contact force (a softer layer
is more easily deformed by a small force), and determine
the dynamic range of the contact force measurement. The
hard layer is assumed to be fixed on the gripper so thatexternal force is applied to the elastic and hard layers onlyand does not affect the cameras. The physical robustness ofthe sensing skin is decided by the elastic and the hard layers.
The camera resolution affects the accuracy of the marker
detection and tracking. The camera frame rate affects thesensing frame rate. These properties (the marker size anddensity, the hardness and the thickness of the elastic andthe hard layers, and the camera properties) should reflect the
purpose (task) of each part of the skin.
B. Prototype Module of Optical Multimodal-Sensing Skin
We make a prototype module of the optical multimodalsensing skin for the parallel gripper of a Baxter robot. For
simplicity, we use one camera for each finger.
Fig. 3 shows the prototype module and its installation on
the Baxter gripper. The size of the module is about 40 mm(W) x 47 mm (L) x 30 mm (H) including a camera modulewith a USB interface. In the following, we describe thedetails of making the module.
1) See-through Skin with Markers: Fig. 4 shows a process
to make the see-through skin with markers. First we make amold for the elastic layer. The fingertip edge has a roundedshape; the other part is flat. Then we put markers on thebottom of the mold. We use black micro plastic beads formarkers, that are spheres of around 1 mm diameter. The
markers are placed on a 5 mm grid. We pour silicone

D. Marker Tracking for Contact Force Estimation

Fig. 3. Prototype module of the optical multimodal-sensing skin (a,b) and
its installation on the Baxter gripper.

Fig. 4.

Process of making the see-through skin.

rubber into the mold. We use Silicones Inc. XP-565 that
has A-16 Shore hardness after cure. Degassing the silicone
was important to keep the transparency before pouring thesilicone into the mold. Right after pouring into the silicone,
we put an acrylic plate (40 mm x 40 mm, 2 mm thickness)with screws as shown in Fig. 4(c). These screws are forpreventing the acrylic plate sinking into the silicone, andcreating a space for screw heads when attaching the skinto the camera. The thickness of the silicone layer is 4 mm.
Fig. 4(d) shows the silicone layer with the acrylic plate after
curing. Finally we cover the skin with a thin transparent
plastic film (Glad ClingWrap) to protect the silicone skinfrom dirt. This film is replaceable.
2) Adding a Camera: We attach the camera to the skin.
The camera is ELP Co. USBFHD01M-L180 which is an
RGB camera with a USB interface and a 180 degree fisheyelens. This skin module works as a stand-alone USB sensor.
The lens is adjusted to focus on the markers, which improves
the marker tracking quality. We embed a fixture used for
integrating into the Baxter gripper in the skin. The fixture isa L-shaped aluminum plate whose one edge is fixed on theacrylic plate (see Fig. 3(a)). Fig. 3(b) shows the side and top
views of the module. In the top view, we can see the camera
through the skin; thus the silicone and acrylic layers havegood transparency.

C. Installation to Gripper Fingers of Baxter Robot
The gripper of the Baxter robot is a parallel gripper with
two fingers. We made two prototype modules and attach one
for each finger. We create a simple mount for the sensor as
shown in Fig. 3(c).

We use an existing blob detection method for markers
implemented in OpenCV1 , cv::SimpleBlobDetector,
that detects small blobs from an image. Each detected blob
has a position (x,y) and size in the image.
Since the camera image is distorted due to the fisheyelens, we rectify the input image before detecting the blobs.
We use camera calibration methods from OpenCV in orderto obtain the camera parameters including the distortion by
the fisheye lens.
Our marker tracking algorithm is simple. First we initialize
the marker tracker by obtaining the initial marker positions
and sizes; we refer to this process as marker tracker calibration. In each frame during the sensor use, we compare thecurrent marker positions and sizes with the initial markers.
1) Marker tracker calibration: We use 20 frames formarker tracker calibration where we put a white board on
the skin so that the blob detector detects only the markers.
If there are moving blobs, they are rejected. We take anaverage of the remaining blob positions and sizes and keepthem as the initial markers.
2) Marker tracking: In each frame, we detect blobs froman input image, and compare them with the initial markers.
Since the order of the blobs does not correspond with theinitial markers, we assume the closest blob to a blob in theinitial frame is the same marker. We define thresholds on
position and size differences, and if they are too large, we
reject them. For each marker position and size differences
dx, dy, ds, we estimate the contact force f as:
f = [cx dx, cy ds, cz dy](1)where cx , cy , cz are conversion coefficients (refer to Fig. 3(c)for the coordinate definition). These coefficients are forhuman readability, not for converting to regular units suchas Newtons.
We can use these contact force estimates as a force field
estimate. We can also convert them to an average force andtorque, which would be useful for simple applications. For
this, we first define a torque estimate around the center ofthe skin surface as: τ = cτ r × f , where r is a displacement
vector from the center of the skin surface to the marker, andcτ is a conversion coefficient. For averaging f and τ , we use
the element-wise 80th percentile2 of the absolute value. Thisfilter gives us a robust estimation against the outliers in themarker tracking, and picks up a local force when the external
force is applied to a narrow region of the skin surface.
We programmed the above methods with as many threads
as possible. The whole process including capturing fromcameras is computed in around 30 frames per second withtwo cameras whose resolution is 640x480. The computer has
an Intel Core i7 CPU (2.70 GHz, 4 cores, 8 hyper threads)
and 16 GB RAM.
1 http://opencv.org/
2 Using the value splitting the highest 20% of the data from the lowest
80%.

Fig. 5.
Pushing force by a human. The right two images are views
from the cameras. The marker movements are rendered (the movements
are emphasized).

Fig. 7. Experimental setup of pushing a knife held by the gripper. Pushingdirections are illustrated.

Fig. 6. Trajectories of the average force (x,y,z) of the left (top graph) andthe right (bottom graph) sensing skins during pushing by a human.

III. E XPERIMENTS
We demonstrate our sensing skin in some experiments. In
these experiments we do not apply a temporal filter, whichshould be used to further reduce noise and errors. Refer tothe accompanying video.

Fig. 8. Trajectories of the average force (x,z) and torque (y) of the left
and the right sensing skins during pushing the knife.

A. Pushing Force Directly Applied to The Skin
First, we let a human push the skin horizontally (xand z axes) for each finger respectively. Fig. 5 shows a
captured image (rectified) with a visualization of the marker
movements. We can see that the markers are moved, andcan see the human fingers through the skin. Fig. 6 shows
the trajectories of the average force (x,y,z) of the left andthe right sensing skins. In both sides, the horizontal (x andz) forces are changing towards pushed directions. During
pushing, the vertical (y) force is changing slightly, but isnoisy. This is because the vertical force is estimated fromthe marker size change and the size does not change muchcompared to the horizontal marker displacement. The reason
why the value of the left sensor is changing around 20 sec
is that the human finger is contacting the skin.
B. Pushing Force Applied to Holding Knife
We made the robot hold an knife and had a humanpush and pull the knife in various directions as shown in
Fig. 7. Fig. 8 shows the trajectories of the average force(x,z) and torque (y) of the left and the right sensing skins.
Each sensor value has an offset value since the gripper
is holding the knife. The horizontal pushing (tx+, tx−)
are captured by the x-force value of the left sensor, while

Fig. 9. Marker tracking result in pushing in the ry− direction (verticalrotation). The left image is a view from an external camera.

the second tx− (around 14 sec) is hardly detected by theright sensor. The reason would be that the surfaces of thesensors were not completely parallel due to an inaccurate
construction, which probably caused an asymmetric force
distribution. The vertical rotational-pushing (ry+, ry−) aredetected clearly by the x-force values of both sensors whichhave opposite directions because of the torque applied to theknife. Fig. 9 shows the marker tracking result in ry−. The ytorque value captures the horizontal rotational-pushing (rz+,rz−), however it seems to have a large hysteresis.
C. Force Profile of Cutting Motion by Human
Next, we let a human hold a knife with the sensing skins.
For this purpose, we detach the fingers from the robot andlet the human hold the entire finger with the knife as shownin Fig. 10. The human cuts a banana and an apple. Fig. 11

Fig. 10. Setup of the cutting motion by a human. The human holds thesensors with a knife, and cuts materials. The left view shows an example ofcutting an apple, and the right two images are views of the sensing skins.

Fig. 11. Trajectories of the average force (x,z) and torque (y) of the left
and the right sensing skins during cutting a banana and an apple by thehuman. In the first short period, the human grasped the knife, and then cut
the banana and the apple.

Fig. 12.

Difficulties in cutting.

Fig. 13. Trajectories of the average force (x,z) and torque (y) of the left
and the right sensing skins during cutting a banana by the robot. There wasa single cutting motion around the peak of left x-force.

shows the trajectories of the average force (x,z) and torque(y) of the left and the right sensing skins. The first half is
cutting a banana several times, and the last half is cutting
an apple. There is a significant difference between cutting
the apple and banana. There are subtle sensor value changes
during cutting the banana because of the softness of banana.
The human cut the apple with “sawing” while the banana
was cut by just pushing the knife down.
D. Robotic Cutting Motion with Sensing Skin
We implemented an automatic cutting motion with thesensing skin. Through our preliminary experiments where we
controlled the robot manually with a joy stick controller, wefound these difficulties: (1) When cutting a hard material,a strong vertical force was applied to the knife, which
deformed the grasp by moving the fingers (cf. Fig. 12(a)).
(2) When a force was applied to the knife horizontally,
the knife slipped in the gripper (cf. Fig. 12(b)). In thisexperiment, we show that using our sensing skins, we caneasily make a controller to avoid these difficulties. Note thateven if (1) or (2) happens, the visual views through the skintell us something happened. For example, look at the camera
view of Fig. 12(b); we can find that the angle of the knife isdifferent from its initial grasp position.
We created a cutting controller which (a) starts from astate where the knife held by the gripper is put above thematerial, (b) moves the knife downward (cutting vertically),
and then (c) slightly pulls the knife (cutting horizontally).

Fig. 14. Trajectories of the average force (x,z) and torque (y) of the left
and the right sensing skins during cutting an apple by the robot. The robot
performed the cutting motion four times to cut the apple completely. The
peaks of left and right x-force correspond with the cutting motion.

The controller moves the knife to the initial position in orderto repeat the motion several times. The difficulties (1) and(2) might happen in the step (b). To avoid them, we consider
conditions to stop the motion: (A) if −(fLx − fLx0 )(fRx −fRx0 ) > 10, or (B) if |τLy | + |τRy | > 4, where fLx and fRx
indicate the x-value of the average force of the left and theright sensors, fLx0 and fRx0 indicate their initial values (rightbefore cutting), and τLy and τRy indicate the y-value of theaverage torque of the left and the right sensors. The condition(A) is defined according to the result of ry− in Section IIIB where the vertical force was applied to the knife. The
condition (B) helps to avoid rotational knife slip.
Fig. 13 and Fig. 14 shows the sensor values during cuttingFig. 15. Marker tracking result in cutting an apple by the robot. The left
image is a view from an external camera.

a banana and an apple respectively where the trajectories of
the average force (x,z) and torque (y) of the left and the rightsensing skins are shown. Since cutting a banana requires onlya small force, the robot cut it with a single trial, while therobot took four trials in cutting an apple. In the banana case,the robot stopped moving the knife because the condition (A)was satisfied as the knife hit the cutting board. In the apple
case, the condition (A) was mainly satisfied by the pressure
from the cut edge of the apple fresh. In these graphs, thecondition (B) was not satisfied, but it was useful when theinitial knife orientation was not perpendicular. Fig. 15 showsa marker tracking result in cutting an apple. Although the
cutting motion is simple, it was capable to cut materials while
avoiding the above difficulties (1), (2).
IV. D ISCUSSION
1) Should we convert the force estimate to an engineering
unit (e.g. Newtons)?: This depends on the application. We
are planning to use machine learning methods to learn dynamical models (e.g. relation between input gripper motion
and output force changes) for example by using neuralnetworks [20]. In such a case, obtaining contact force information in engineering units is not necessary. If the force
estimates are consistent enough, we could learn mappingsfrom grasping-action parameters to force field, and fromforce field to grasping-quality measurements. These models
would be useful to reason about grasping actions. Howeverusing an engineering unit will be generalizable to othersituations and other robots, so it is still beneficial to consider.
2) How does our sensing skin improve manipulation?:
From our experiments, we found there are at least two usefulcases. One is using the force estimates during manipulation.
We implemented the autonomous cutting behavior based on
the force estimates. Stopping cutting when the knife reaches
the cutting board would be hard to implement without force
estimates, since the knife is occluded by food. Our cutting
behavior could be improved by learning physical phenomena
captured by the sensing skin as described above. The otheris the proximity vision during manipulation. Although we
did not use it systematically, slip detection through thevision could help automate failure detection in cutting. An
alternative way would be using cameras located externally oron the robot head, but the perception programming will bedifficult. Since both the knife and gripper are moving during
cutting, distinguishing the knife movement from the gripper(slip) is not simple. The proximity vision through the skinmakes this problem much easier to handle.
3) Accuracy, reliability, and hysteresis: Although we did
not conduct a controlled evaluation, the horizontal force
seems to be reliable. The marker movement in a horizontal
direction is easier to track. The vertical force seems to be
more difficult to detect as the changes of the marker visualsize are comparably smaller. Perhaps we can use the verticalforce estimate as an on/off signal. A way to improve thevertical force accuracy is increasing the thickness of theelastic layer, although it would make the skin heavier.
In the experiments, there were some false detections of
the markers. These were mostly due to the external scene.
Increasing the number of markers is helpful for removing
outliers, although it will reduce the transparency accordingly.
Installing an internal light source would be helpful in darkscenes.
We found hysteresis, especially when a strong force wasapplied. In the cutting vegetable experiments by the robot,
the force estimate changed before and after cutting a hardmaterial. This would be because the deformation of the soft
layer remained. This was mostly reset after releasing theknife.
4) Calibration frequency: We did the camera calibration
once before assembling the sensor. During the experimentsof the previous section, we did the marker tracker calibration
only twice. The first one was the initialization. The second
one was necessary due to the accidental separation of thesoft layer and the acrylic base.
5) Physical robustness: A weak part of the sensor isthe adhesion between the soft skin and the acrylic base.
Currently we rely on the sticky property of the silicone
rubber. It was strong enough against horizontal forces anddownward forces from above, but the skin can be easily
peeled by upward forces from below. This issue would be
solved by an adhesive between the layers. Other way would
be covering the acrylic base with the silicone rubber fromboth sides, or making holes on the acrylic base and pouring
the silicone rubber into them3 .
Other than that, the sensor strength is decided by theacrylic base hardness and the soft skin. For heavier tasks
where larger contact forces will be applied, we would needa thicker acrylic base. Although the soft skin is weak against
something with thin tips (e.g. needle) or sharp edges (e.g.
knife), the hardness is close to that of the human skin. Thus
we think it is strong enough for daily manipulations.
V. C ONCLUSION
We developed an optical multimodal-sensing skin for robot
fingers. The sensor structure is simple: a transparent skinmade with a marked soft elastic outer layer on a hard layer,
and internal RGB cameras. The important feature is the seethrough skin that provides visual information of a close
object. We installed a prototype of the sensor on fingers
of a Baxter robot, and used it in a practical task, cutting
vegetables. The sensor gave us a force estimate to control
cutting vegetables and avoiding slippage of the knife.
3 We appreciate a reviewer who gave us this idea.

R EFERENCES[1] C. Atkeson et al., “NO FALLS, NO RESETS: Reliable humanoid
behavior in the DARPA Robotics Challenge,” in the 15th IEEE-RAS
International Conference on Humanoid Robots (Humanoids’15), 2015.
[2] S. Begej, “Planar and finger-shaped optical tactile sensors for robotic
applications,” IEEE Journal on Robotics and Automation, vol. 4, no. 5,
pp. 472–484, 1988.
[3] H. Maekawa, K. Tanie, K. Komoriya, M. Kaneko, C. Horiguchi,
and T. Sugawara, “Development of a finger-shaped tactile sensor andits evaluation by active touch,” in Robotics and Automation, 1992.
Proceedings., 1992 IEEE International Conference on, vol. 2, 1992,
pp. 1327–1334.
[4] K. Kamiyama, H. Kajimoto, N. Kawakami, and S. Tachi, “Evaluationof a vision-based tactile sensor,” in Robotics and Automation, 2004.
Proceedings. ICRA ’04. 2004 IEEE International Conference on,vol. 2, 2004, pp. 1542–1547.
[5] J. Ueda, Y. Ishida, M. Kondo, and T. Ogasawara, “Developmentof the NAIST-Hand with vision-based tactile fingertip sensor,” inProceedings of the 2005 IEEE International Conference on Robotics
and Automation, 2005, pp. 2332–2337.
[6] Y. Ito, Y. Kim, and G. Obinata, “Robust slippage degree estimation
based on reference update of vision-based tactile sensor,” IEEE
Sensors Journal, vol. 11, no. 9, pp. 2037–2047, 2011.
[7] N. F. Lepora and B. Ward-Cherrier, “Superresolution with an optical tactile sensor,” in Intelligent Robots and Systems (IROS), 2015
IEEE/RSJ International Conference on, 2015, pp. 2686–2691.
[8] X.-D. Yang, T. Grossman, D. Wigdor, and G. Fitzmaurice, “Magic
finger: Always-available input through finger instrumentation,” inProceedings of the 25th Annual ACM Symposium on User Interface
Software and Technology, 2012, pp. 147–156.
[9] H. Yussof, J. Wada, and M. Ohka, “Sensorization of robotic hand using
optical three-axis tactile sensor: Evaluation with grasping and twisting
motions,” Journal of Computer Science, vol. 6, no. 8, pp. 955–962,
2010.
[10] T. Ikai, S. Kamiya, and M. Ohka, “Robot control using natural

instructions via visual and tactile sensations,” Journal of Computer
Science, vol. 12, no. 5, pp. 246–254, 2016.
[11] C. Chorley, C. Melhuish, T. Pipe, and J. Rossiter, “Development ofa tactile sensor based on biologically inspired edge encoding,” inAdvanced Robotics, 2009. ICAR 2009. International Conference on,
2009, pp. 1–6.
[12] D. Hristu, N. Ferrier, and R. W. Brockett, “The performance of adeformable-membrane tactile sensor: basic results on geometricallydefined tasks,” in Robotics and Automation, 2000. Proceedings. ICRA
’00. IEEE International Conference on, vol. 1, 2000, pp. 508–513.
[13] Y. Ito, Y. Kim, C. Nagai, and G. Obinata, “Shape sensing by visionbased tactile sensor for dexterous handling of robot hands,” in 2010
IEEE International Conference on Automation Science and Engineering, 2010, pp. 574–579.
[14] T. Assaf, C. Roke, J. Rossiter, T. Pipe, and C. Melhuish, “Seeing bytouch: Evaluation of a soft biologically-inspired artificial fingertip in
real-time active touch,” Sensors, vol. 14, no. 2, p. 2561, 2014.
[15] R. Li, R. Platt, W. Yuan, A. ten Pas, N. Roscup, M. A. Srinivasan,
and E. Adelson, “Localization and manipulation of small parts using
gelsight tactile sensing,” in 2014 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, 2014, pp. 3988–3993.
[16] OptoForce Co., “White paper: Optical force sensors — introduction
to the technology,” Tech. Rep., January 2015. [Online]. Available:
http://optoforce.com/[17] R. Patel and N. Correll, “Integrated force and distance sensing usingelastomer-embedded commodity proximity sensors,” in Proceedingsof Robotics: Science and Systems, 2016.
[18] S. C. Abdullah, J. Wada, M. Ohka, and H. Yussof, “Object exploration
using a three-axis tactile sensing information,” Journal of Computer
Science, vol. 7, no. 4, pp. 499–504, 2011.
[19] L. Cramphorn, B. Ward-Cherrier, and N. F. Lepora, “Tactile manipulation with biomimetic active touch,” in 2016 IEEE International
Conference on Robotics and Automation (ICRA), 2016, pp. 123–129.
[20] A. Yamaguchi and C. G. Atkeson, “Neural networks and differential dynamic programming for reinforcement learning problems,”
in the IEEE International Conference on Robotics and Automation(ICRA’16), 2016.