In-Hand Gravitational Pivoting Using Tactile Sensing
Authors: Jason Toskov
Rhys Newbury
Mustafa Mukadam
Dana Kulic
Akansel Cosgun
Venue: CoRL 2022 Poster (2022)
Keywords: Tactile Pose Sensing
In Hand Manipulation

Abstract: We study gravitational pivoting, a constrained version of in-hand manipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To achieve this, instead of controlling the gripper to avoid slip, we \emph{embrace slip} to allow the object to rotate in-hand. We collect two real-world datasets, a static tracking dataset and a controller-in-the-loop dataset, both annotated with object angle and angular velocity labels. Both datasets contain force-based tactile information on ten different household objects. We train an LSTM model to predict the angular position and velocity of the held object from purely tactile data. We integrate this model with a controller that opens and closes the gripper allowing the object to rotate to desired relative angles. We conduct real-world experiments where the robot is tasked to achieve a relative target angle. We show that our approach outperforms a sliding-window based MLP in a zero-shot generalization setting with unseen objects. Furthermore, we show a 16.6\% improvement in performance when the LSTM model is fine-tuned on a small set of data collected with both the LSTM model and the controller in-the-loop. Code and videos are available at https://rhys-newbury.github.io/projects/pivoting/.

In-Hand Gravitational Pivoting Using Tactile Sensing
Jason Toskov
Rhys Newbury Monash University
Mustafa Mukadam
Monash University
Meta Al
jtos00039student. monath. edu
rhys www.uurymnonnsh.....mmmmmmmm. edu
mukadamffb.co
Dana Kulić
Akansel Cosgun
Monash University
Deakin University
dana. kul i comonash.. edu
akan. wwwa..ccsnnddeekin.ccmm .dw.needuueerr au
Abstract: We study gravitational pivoting, constrained version of in-hand ma- nipulation, where we aim to control the rotation of an object around the grip point of a parallel gripper. To achieve this, instead of controlling the grip- per to avoid slip, we embrace slip to allow the object to rotate in-hand. We collect two real-world datasets, static tracking dataset and controller-in-the- loop dataset, both annotated with object angle and angular velocity labels. Both datasets contain force-based tactile information jects. We train an LSTM model to predict the angular position and velocity the held object from purely tactile data. We integrate this model with troller that opens and closes the gripper allowing the object to rotate to de- sired relative angles. We conduct real-world experiments where the robot is tasked to achieve a relative target angle. We show that our approach outper- forms sliding-window based MLP in zero-shot generalization setting with unseen objects. Furthermore, we show a 16.6% improvement in performance when the LSTM model is fine-tuned on small of data collected with both the LSTM model and the controller in-the-loop. Code and videos are available at haaps::/rrhss-eeeee
ten different household ob-
gi thub. httpprojecta/ppvottttooo
Keywords: Tactile Pose Sensing, In Hand Manipulation
Gripper Width Control
Grip Controller
Angle and
Velocity of object
Tactile
LSTM
Data
Figure 1: We study hand manipulation to rotate object with gravitational pivoting. We design an LSTM model to predict the position and velocity of an object purely from tactile sensing. Our grip controller uses this prediction to modulate the width of the gripper and achieve target angle.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Introduction
The majority of past works in robotic manipulation either assume fixed grasps [1. 2] aim to avoid any slipping during manipulation [3]. However, we focus on embracing slip, using object slip to increase the dexterity of simple grippers. This idea was explored by Chen et al. [4] where the grip on the object is loosened to allow an object to slip downwards with gravity. This paper explores a method of inducing rotation for in- in-hand manipulation using gravity. Such manipulation, known as pivoting, is key to performing tasks requiring an object to be at a specific relative angle to a gripper, such stacking shelves [5].
This paper proposes a method for robot with a parallel gripper to rotate long object grasped away from its center of mass to a desired final relative orientation. This aims to allow parallel grippers to robustly reorient objects into a desired orientation without having to regrasp the object. To achieve this task, we focus addressing two challenges: tracking the position of the object and controlling the gripper to allow for gravitational pivoting towards the target angle.
Assion-based methods to track the object often make use of an eye-in-hand camera. However, the gripper will often occlude the object, making it difficult estimate the angle of the object accu- rately [6]. An alternative is to use an externally placed camera. However, this necessitates the robot moving to a fixed position in front of the camera for each manipulation. We use purely tactile infor- mation to track the object to avoid these issues. We design an LSTM-based neural network model, RSE-LSTM, which uses tactile information to predict a held objects' relative angular position and angular velocity.
Previous approaches to controlling the gripper often used model-based approaches, which required information about the object, such as shape, mass, and friction. In contrast, we design a simple gripper controller that assumes no a priori knowledge about the object parameters to allow for gen- cralization to unseen objects.
We collect real-world force-based tactile dataset, ten household objects. This dataset is anno- tated with both angular position and velocity measurements. RSE-LSTM is trained on this dataset, and the results reported with respect both unseen data and unseen objects. We further validate our approach experimentally on unseen objects.
The contributions of our paper are threefold:
An annotated dataset containing gravitational pivoting with 10 household objects.
A LSTM-based neural network which can predict both the velocity and angle of an object using
only tactile information.
A grip controller, which can adjust the width of the gripper to allow an object to pivot in-hand to achieve required relative angle.
2 Related Works
Slip measurement. Slip detection is often framed as a binary problem, with machine learning models predicting either slip or no slip. This is achieved with the use of visual sensors [7], force- based sensors [8]. or optical sensors [9]. Various machine learning techniques have been used including: Support Vector Machines [10, 91. MLPs [11] and LSTM models [12]. Alternatively, Convolutional Neural Networks (CNN) have been used to both detect and classify the type of slip as either translational rotational [13]. LSTM models have been used to determine the direction
of rotational slip [14), or the overall direction of the combination of rotational and translational slip [15]. The domain of quantitative slip mcasurement is comparatively underexplored. Previous works mea- sure the amount of translational slip using image-based tactile sensors [16]. Alternatively, visual gel-based tactile sensors have been used to measure the rotation angle using a model-based ap- proach [17]. However, to knowledge, the use of force-based tactile sensing has not been ex- plored, which is the focus of this paper.
Induced rotation. To induce rotation in held object, previous work has made use of the external environment to apply a torque or force on the held object [18, 19, 20J. However, rotation can also be induced without any interaction with external objects. For example, the robot can perform swinging motion using the end-effector, where the velocity of the swing aims to bring an object to a desired angle [21, 22, 23].

---

Alternatively, by loosening the grip on the object, gravity can be used to induce a rotation in the object [24, 25, 26, 27, 28, 29]. These approaches are model-based and rely on prior knowledge of important parameters of the system, such as shape, mass and friction of the held object. Our work assumes no prior knowledge about such parameters to allow for generalization unseen objects.
Induced translational slip. Shi et al. [30] design model-based approach to induce translational slip an object by accelerating the gripper causing the object slide in a desired way in-hand. Chen et al. [4] train MLP to predict the velocity of an object which is undergoing translational slip. They feed the MLP the previous one second of observations to predict the sliding velocity of the object. A controller is then designed to achieve a target sliding velocity for the object. Our work extends this to the rotational case and makes use of an LSTM rather than providing fixed length history. The allows the network to learn an encoding for the history, which may be more informative.
3 Problem Definition
Gravitational pivoting
form of in-hand manipulation, where an object rotated in hand by loosening the grip on the object. Consider static gripper which holds an object away from its center-of-mass. If the gripper closed tightly enough, the object should remain stationary inside the gripper. However, if the grip the object is loosened slightly, gravity will induce torque the object, causing it to pivot inside the gripper.
Figure 2: Gravitational Pivoting occurs when a parallel-jaw gripper loosens grip on object such that gravity induces torque on the object. This torque will then cause rotation to occur.
The coordinate system for this problem is defined in Figure 2. Gravity is defined to work in the negative y-direction in the global frame. We also define rotating object-centric coordinate frame, where the y-axis is aligned with the long axis of the object.
Specifically, consider pivoting tasks as follows. The robot starts with an object in hand, grasped away from the center-of-mass. The task is to rotate the object by a relative angle around the object-centric z-axis. We only consider achieving relative angles (a) in the range [o, 180] degrees, as we only consider allowing the ob- ject fall and rotate. We assume that the robot has no a priori knowledge about the object and only has access to tactile information from the fingertips. This aims to allow generalizability by assuming no knowl- edge of the grasped objects. We constrain the type of objects to ones that have prism-like shape, where one dimension, length, is much larger than the other two, width and depth. When these objects are held away from the center-of-mass, the torque induced on the ob- ject by gravity will be much larger than the downwards force at the contact points. Therefore, the objects are more likely to undergo rotational slip when the grip is loosened, and the translational slip is assumed be negligible. Prism-like objects include common household objects, such as bottles, boxes, and tools (such hammers).
UR5 Robotic Arm
Parallel Jaw Gripper
OAK-D
Camera
Figure 3: The hardware setup for both data collection and experiments.

---

Data Collection
4.1 Hardware Setup
The robot hand consists of a Robotiq 21-85 parallel-jaw gripper and a table-mounted URS robot arm. On each jaw of the gripper is PapillArray tactile sensor by Contactile [31]. The parallel jaw gripper has a width of 85mm and can be controlled 256 increments, for a resolution 0.33mm per increment. The PapillArray sensors consist of 9 pillars, arranged in 3 by 3 square, where each pillar provides a force and displacement measurement in each direction, and whether the pillar is in contact with the object. The sensor also provides global forces and torques in each direction, for a total of 142 measurements over the two sensors. addition, an OAK-D camera positioned to provide side-on view of the robotic arm and held object, which will be used record the ground truth angle (more details of the ground truth collection is provided in Section 4.3). The hardware setup for both data collection and experiments is shown in Figure
4.2
Object set
We consider the pivoting task for set of 10 different household objects. Similar to the object set used by Chen et al. [4], we consider two classes of objects: box-like objects and cylinder- like objects, with five objects in each class. The set of objects used shown in Figure 4.
4.3 Methodology
A systematic methodology is used to collect dataset of gravitational pivoting, outlined in the supplementary material. We use two different methodologies for controlling the gripper dur- ing rotation of the object:
Box
Cylindrical
Rotate To Stop: The gripper opened fixed amount and the object is allowed to ro- tate until the object comes to a stationary po- sition.
Angle Goal: The controller (described in Section 5.2) is tasked to stop the object at an angle (Okstop) from set of angles (A stop). The controller receives ground truth angle readings for the purposes of data collection.
Figure 4: Objects used in this dataset. We dis- tinguish between two classes objects, box-like object and cylinder-like objects. We refer to the names of these objects throughout paper. The names are from left to right (back row): Tooth- paste, Earbud, Breadboard, Magnet, Deodorant, Spray2, Shampoo, Spray Pill. The object the front is labeled as Toothbrush.
To create larger variation of friction proper- ties, we collect data both with and without layer of masking tape added to each object sur- face. In total there were 595 'Rotate To Stop' and 971 I 'Angle Goal' sequences collected, after filtering invalid datapoints. The dataset is attached the supplementary material.
4.4 Ground Truth Annotation
To measure the ground truth rotation of the grasped object, distinctly colored blobs have been attached to each object. An external camera observes these blobs, and we then define the object- centrie y-axis between the centroid of the two blobs. The orientation change between the initial object-centric y-axis and the current object-centric y-axis is used as the ground-truth angles. The position and angular velocity are filtered ensure they follow smooth signal. The details of the filter are in the supplementary material.
Proposed Approach
Our approach consists of two main components. A Rotational Slip Estimator LSTM (RSE-LSTM) and a Grip Controller. The system diagram is shown in Figure 1. From purely tactile information, the RSE-LSTM estimates the relative angle change between the initial and current object-centric y-axis. The Grip Controller uses the estimation and gravitational pivoting, aiming to reach desired rotation relative to the initial angle.

---

5.1 Rotational Slip Estimator
The RSE-LSTM uses measurements from the tactile sensors and predicts both the current angular velocity (w) and relative angle of the object (a) the outputs. We found that calculating both and w improved the results of the model (Section 6.3).
The RSE-LSTM model consists of an LSTM, the outputs of which are passed to an MLP. Instead of using a sliding window (similar to [4]), using the hidden states the LSTM could allow the model to use longer history more effectively by learning more feature-rich hidden state. The model runs 60 due to hardware limitations training data collection.
5.2 Grip Controller
Algorithm Grip Controller
goal + Sstop
error < relattve_ggeee ggoal
while error to do m + Observation()
or w + -SS-LSTM(())
if w Wmin and teurr Eprev > wait then increase.gripe purrass-gripper.openingg
tpree teurr end if
Or + fp(a, w,
error + or goall end while fully wwwseeoooee.ccc gripper()
The grip controller algorithm described Alg. 1. The grip controller uses the position and velocity estimated by the LSTM to achieve a target angle. While the object is not moving, the controller slightly opens the gripper. Once the object reaches the target angle, the gripper then closes. However, this alone constantly overshoots the target due to the delay in closing the gripper. To account for this, the velocity estimate used to forward predict the angle (denoted by fp in Alg. 1), by a fixed delay (d) to stop the gripper early. The forward prediction equation is: aF d X w.
Pose Estimate Model Evaluation
wx(2/x) a('))
750
Initial State
Dynamic Range
Steady State
500
250
0
100
0.5
1
1.5
2
2.5
time (s)
Figure 5: We split gravitational pivoting in to three sections, denoted by the three colors in the graph
6.1 Baseline
Chen al. [4] found promising results using a sliding-window window MLP similar task, where they predict the velocity of an object undergoing translational slip. Therefore, as a baseline, we consider the use sliding window MLP. which takes the last 15 steps of tactile data and predicts the angle and angular velocity at the next timestep. The parameters of this model were optimized using sweep to maximize the performance on the test set, during random split experiments. In Section 9.10 present results ablation study on the length of the sliding window.
5

---

6.2 Experiments
We split the trajectory in to three sections (shown in Figure 5).
1. Initial State (IS): The object is held still, before any movement. The networks should be able predict both for the initial relative angle and 0 for velocity.
2. Dynamic Range (DR): The object is currently rotating. The models should track both position and velocity.
3. Steady State (SS): The object has finished rotating. The network should be able to predict the correct final angle and 0 velocity.
The dynamic range commonly lasts less than a second, thus necessitating learned estimation and control work in real-time. The objects also commonly reach velocities of up 750%/s, making ac- curately predicting the velocities of the object difficult. The fast rotation of the object also increases the need for both the perception and control to be real-time, as any time delay in the system will magnify any errors.
We conduct three experiments, firstly experimenting unseen data (Section 6.3), secondly experi- menting on unseen objects (Section 6.4) and finally experimenting on unseen classes. The training and network parameters detailed in the supplementary material with the unseen classes experi- ment results.
120
Ground Truth MLP
Angular Velocity (%/ee)
500
90
LSTM
400
Angle (9)
300
60
200
30
100
L
Ja..
0
1
2
3
1
2
3
time (s)
time (s)
Figure 6: Example Tracking Performance. The LSTM produces a much smoother output compared
to the MLP. The models display a consistent undershooting of the peak angular velocity.
6.3 Rotation Estimation
We train the previously described LSTM and MLP the collected dataset. We varied the output of both models, to predict only a, only w or both w and a. the model only outputs w, a will be recovered by integrating the signal. If the model calculates only a, w will be calculated by taking the derivative. An 80/20 train/test split of the data was used for training and the mean absolute error (MAF) is reported in Table
Angular Error (9)
Angular Velocity Error (*/8)
IS
DR
SS
IS
DR
SS
LSTM Both
0.29+0.06
4.3940.18
7.23+0.2
0.6420.22
32.3842.6
1.47+0)42
MLP Both
0.6740.58
10.58+0.57
19.85+1.07
0.76+0.27
41.38+1.64
1.37+0.3
LSTM (w-only)
0.27+0.03
7.4+2.1
13.79+4.36
0.71+0.20
31.9742.36
0.8140.26
MLP (w-only)
0.3940.11
12+1.14
22.21+2.35
1.38+0.5
36.93+2.01
1.92+0.57
LSTM (a-only) 0.3140.03
4.5240.21
7.2340.26
1.7440.32
40.13+1.88
4.8+0.25
MLP (a-only)
1.48+0.55
10.41+0.67
19.79+1.11
3.52+1.28
65.07+1.62
8.31
IS, DR, SS represent the three sections of motion, Initial State, Dynamic Range and Steady State, respectively. represents the angular velocity, and a represents the angular position.
Table 1: Testing results on unseen data. Each network is trained times.

---

LSTM Both out out-performed the other models in terms of the position error. slightly outperforming LSTM (a-only) terms of the position error in all three stages. LSTM Both also performed similarly to LSTM w-only in terms of the velocity error. The MLP performance was lower than the LSTM for all stages for both velocity and position emror.
An additional benefit of the LSTM over the MLP is that the LSTM produced much smoother results than the MLP. This can be seen in Figure where the MLP predictions are more noisy compared to LSTM. This is problematic when paired with the Grip Controller, making it challenging for the controller to achieve precise angles and also more prone to early failure.
6.4 Unseen Objects
For the next experiments, we only con- sider models that output both and w, as they were the best performing models. We
MLP
LSTM
investigate the performance of the mod-
els when generalized to unseen objects. We trained the LSTM and MLP on ob-
35
Angle Error
30
jects and tested on the remaining 1 unseen object. The performance of the models decreased compared to training ously seen objects. This is most likely due to the model struggling to generalize to ob- jects that have unique properties compared to the other objects in the training set. An example is the 'Toothpaste' object, where the position error is much larger for both the LSTM and MLP, as shown in Figure 7.
25
20
previ-
Breadboard
Decdarant
Earbod
Mag
PiNl
Shargoo
Sprayl
Spray
Teothimsh
Teothpaste
Figure 7: Distribution of position emrors for each unseen object.
7
Controller Evaluation
7.1 Baselines
We consider two baselines:
Oracle: Ground truth angle and angular velocity measurements for the Grip Controller are taken from visual data. This provides upper bound on the performance of the gravitational pivoting system.
MLP: The MLP described in Section 6.1
7.2
Experimental Setup
The experimental methodology followed was same as the 'Angle Goal' data collection process de- scribed in Section 4.
We experimentally tested the performance on unseen objects using the models from Section 6.4. During experiments both the ground truth and predicted a/w were recorded. These measurements were used compute the MAE of the model within the sections as defined in Section 6.2 (IS, DR and SS).
The most important error in the controller experiments is the error in the final angle achieved by the controller, denoted as target error (TE). We also report the failure rate (FR) which is the percentage of trials for which the controller either dropped the object or became stuck, predicting an > Wmin while the object is stationary.
To address the model generalization to the real-world setup, further collected a small amount of AAngle Goal data with the controller receiving angle measurements from the trained RSE--LTM model instead of the Oracle to test the effects of fine-tuning the controller the desired scenario. The data collection parameters are detailed in the supplementary material.
7

---

7.3 Results
The results of the real-world experiments shown in Table 9. While the LSTM outperformed the MLP for all metrics, the tracking error much larger than the er-
MLP Finetuned
Finetured
MLP
Angle Error (*)
LSTM
ror seen on the test set. This is likely be-
cause the training dataset does not repre- sent all scenarios seen in real-world test- ing such as overshooting the target angle and the gripper not closing.
5
Furthermore, the target error was much larger than the Oracle baselines. This can be attributed to the previously stated track- ing inaccuracies, particularly in dynamic range, resulting in the model predicting the target angle was reached either too late or too early.
Bresdtuand
Drestrrnt
Latbol
Magert
Spoayd
Toothoruth
Figure 8: The target-error for all unseen object exper- iments. The fine-tuned LSTM outperformed all other methods for 8 out of the 10 objects.
As shown in Figure 8, fine-tuning improved the results on most objects. The distribution of the box plot follows similar pattern Figure 7. This suggests some objects are harder for our approach to generalize to. This could be further improved by collecting data on a wider range of objects.
Failure Rate (%)
Target Error (9)
Oracle
0
3.95+3.02
LSTM
3.3
15.2+4.69
MIP LSTM-finetund
6.7
23.08=8.31
4.44
12.67+5.31
MLP-finetuned
10.00
23.10=8.17
Table 2: Results of fine-tuned models with experimental setup. The the other models in terms of the target error
outperforms
8 Limitations and Future Work
One of the main challenges of gravitational pivoting is that to avoid extra motion of the end-effector, the object can only rotate in one direction. Hence, it is hard to recover from errors when the object moves past the target angle. To minimize the effect of the issue, we aim rotate the object slowly, by slowly loosening the grip until some velocity is observed. However, the object can still rotate in the magnitude of 500%s. The fast speeds necessitate accurate stopping conditions, which makes accurate target reaching very difficult. Furthermore, there is a large delay in commanding the gripper rest, at around 0.83s. The stopping condition uses forward
to close and the object coming prediction proportional to both the delay and velocity. therefore, any errors in velocity prediction are magnified by a large time delay. A potential solution for both of these issues would be to slow the object down as it is pivoting. This could be achieved using learning-based controller, which could make smarter decisions on when to speed up slow down pivoting. Another potential solution would be to increase the speed of the cola-colleetion process and in turn allow RSE-LSTM to operate at a higher frequency. This would allow more reactive control to fast changes in position and velocity. Another limitation of our work is that the performance degrades when working with unknown objects. This could be improved with collecting more varied data on a wider range of objects. Further investigation is needed at looking at the effect of out of distribution testing objects, for examples lighter objects will likely be harder information is sufficient for this problem. While it does show promising results, further work needs be done to improve the results, to make both the estimation and the controller more robust.
rotate. Our work aims to show that tactile
8

---

Acknowledgments
We would like to thank Lily Tran for help editing the manuscript.
References
[1] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-driven grasp synthesis- survey. IEEE Transactions Robotics, 30(2):89-339, 2014. doi: 100.10000000022022299000000
[2] R. Newbury, M. Gu, L. Chumbley, A. Mousavian, C. Eppner, J. Leitner, J. Bohg, A. Morales,
T. Asfour, D. Kragic, D. Fox, and A. Cosgun. Deep learning approaches grasp synthesis: A review. arXiv preprint arXiv:2207.022. 22.5566,, 2022.
[3] W. Chen, H. Khamis, I. Birznieks, N. F. Lepora, and S. J. Redmond. Tactile sensors for friction estimation and incipient slip eetection-tower dexterous robotic manipulation: A review. IEEE Sensors Journal, (8(2))99049-999) 909499906444 2018. 109/ISEN20/ 2.268330..
[4] Y. Chen, C. Prepscius, Lee, and D. D. Lee. Tactile velocity estimation for controlled in- grasp sliding. IEEE Robotics and Automation Letters, 6(2):1614-6211, 2021. doi:10.1109/ IRA.2021.305893.
[5] M. Costanzo, G. De Maria, G. Lettera, and C. Natale, Can robots refill supermarket shelf?: Motion planning and grasp control. IEEE Robotics Automation Magazine, 28(2):61-73, 2021. doi: 10.1 109/MRA .3064754.
[6] T. Anzai and K. Takahashi. Deep gated multi-modal learning: In-hand object pose changes esti- mation using tactile and image data, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9361-9368 2020. doi: 100110.1100/R0055443320000000000
[7] Dong, D. Ma, E. Donlon, and Rodriguez. Maintaining grasps within slipping bounds by monitoring incipient slip. In 2019 International Conference on Robotics and Automation (ICRA), pages 3818-3824, 2019.
[8] T. Narita, S. Nagakari, W. Conus, T. Tsuboi, and K. Nagasaka. Theoretical derivation and realization of adaptive grasping based on rotational incipient slip detection. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 531-537, 2020. doi: 10. N0.100//C/4409455 010.44..220.999660
[9] J. W. James, N. Pestell, and N. F. Lepora. Slip detection with a biomimetic tactile sensor. IEEE Robotics and Automation Letters, 3(4):3340-3344 2018.
[10] L. Li. F. Sun, B. Fang, Z. Huang, C. Yang, and M. Jing. Learning to detect slip for stable grasping. In 2017 IEEE International Conference Robotics and Biomimetics (ROBIO), pages 430-435, 2017. 1oi:10.1109/ROBB (00.2200778325555
[11] 7. Su, K. Hausman, Y. Chebotar, A. Molchanov, G. F. Loeb, S. Sukhatme, and S. Schaal. Force estimation and slip detection/cllassffcation for grip control using a biomimetic tactile sensor. In 2015 RES 15th International Conference on Humanoid Robots (Humanoids), pages 297-303, 2015. doi: HOL.L0//HAANNIISS000........
[12] K. Van Wyk and J. Falco. Calibration and analysis of tactile sensors as slip detectors. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7744-275 2018. doi:10.7 200.1109//CRA.2000 8.8811117
[13] M. Meier, F. Patzelt, R. Haschke, and H. J. Ritter. Tactile convolutional networks for online slip and rotation detection. In ICANN, 2016.
[14] J. Li. Rotational direction detection using tactile sensor and external camera. In 2019 JEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pages 1456-1461, 2019. 10i:10.1 100.1190....01..888222222
[15] B. S. Zapata-Impate P. Gil, and Torres. Learning spatio temporal tactile features with convistm for the direction of slip detection. Sensors, 19(3):523, 2019.

---

[16] X. Li, K. Zhao, C. Lu, and Y. Wang. Quantitative motion detection of in-hand objects for robotic grasp manipulation. International Journal of Advanced Robotic Systems, 16(3): 1729881419846333, 2019. 40f:100177/177998811999466 117/472998141996633
URL hattps://don otg/io..g11...
[17] R. Kolamuri, Z. Si, Y. Zhang, A. Agarwal, and W. Yuan. Improving grasp stability with rotation measurement from tactile sensing. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6809-6816, 2021. doi: 1e111.111/1ROSSS 2021..0.222.... 96364888.
[18] N. Chavan-Datte and A. Rodriguez. Prehensile pushing: In-hand manipulation with push- primitives. In 2015 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6222, 2015. 100110..119/1100.001....77
[19] N. Chavan-Dafle and A. Rodriguez. Sampling-bast based planning of in-hand manipulation with external pushes. In International Symposium of Robotic Research, pages 523-539, 01 2017. 1001/0. 00-000000------0
[20] Y. Hou, 7. Jia, and M. T. Mason. Reorienting objects in 3d space using pivoting. arXiv preprint arXiv: 19/2.022/ 2019.
[21] S. Cruciani and C. Smith. In-hand manipulation using ttree-stages open loop pivoting. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1244-1251, 2017. 10t::0..11//RRS.... 10/0.100000ROR.....70000000 2017.202299.
[22] R. Antonova, S. Cruciani, C. Smith, and D. Kragic. Reinforcement learning for pivoting task. arXiv preprint arXiv: 10XXvv1703.0000 004000
[23] A. Sintov and A. Shapiro. Swing-up regrasping algorithm using energy control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4888 4888-4893, 2016. doi: 1tw.11100/AAA2211..7...
[24] F. E. Viña B., Y. Karayiannidis, K. Pauwels, C. Smith, and D. Kragic. In-hand manipulation using gravity and controlled slip. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5636-5641, 2015. doi:10.1 109/IROS (009//ROS.20157355444 2015.7354777...
[25] E. Viña B., Karayiannidis, Smith, and D. Kragic. Adaptive control for pivoting with vi- sual and tactile feedback. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 399-406, 2016. doi: 10.1 109//CRA.2016..44599 159
[26] M. Costanzo.
Control of robotic object pivoting based on tactile sensing. Mechatron- ISSN 0957-4158. bboittps://doi.rg/10001100 URL https:/ //tp:///www... www.:sseenneddret.coom http:///ttcceecccccccece///ttttt
ics, 16:1025,,, 2021.
2021.102545. $095741582100043X.
[27] M. Costanzo, G. D. Maria, and C. Natale. Control of sliding velocity in robotic ob- ject pivoting based on tactile sensing=*this work was supported by the european commis-
sion within the h2020 refills project, id 731590. PFAC-PapersOnLLine, 53(2):9950- 2020. ISSN 2405-8963. //www. www.ssciencedrectt. http:///cciiceeet......////////a World Congress.
101600066...... 000. URL https:
21st IFAC
[28] A. Cavallo, M. Costanzo, G. De Maria, and C. Natale. Modeling and slipping control of planar slider. Automatica, 115:10 2020. ISSN 0005-1098, doichttps:/doi..pg http:////ooi.//00.1010100 1016/ L.automatiaa.2200.000 10887.. URL https: //ww://wwwwwwww sciencedirect. http:/rrttt.cooccceeeeeeeeeeeeee com/sccience//atticll// p11/$0005109820300733.
[29] A. Holladay, R. Paolini, and M. Mason. A general framework for open-loop pivoting. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 3675-3681, 2015. doi: 1e.11...11999 (009//CAA.201..71999999
[30] J. Shi, J. Z. Woodruff, P. B. Umbanhowar, and K. M. Lynch. Dynamic in-hand sliding ma- nipulation. JEEE Transactions on Robotics, 33(4):778-799 2017. 2693391.
[31] H. Khamis, B. Xia, and Redmond. Real-time friction estimation for grip force control. In IEEE International Conference on Robotics and Automation (ICRA), pages 1608-1614, 2021.
10


Reviews:
[{"review_id": "awrPwnpD0o4", "reviewer": "Reviewer_ynxK", "summary": "This paper leverages gravity to achieve in-hand manipulation of prism-like objects via pivoting objects to a desired angle. The proposed solution has two components: (1) a Rotational Slip Estimator LSTM (RSE-LSTM) that estimates the relative angle change and velocity of the object based on tactile information and (2) a grip controller that uses RSE-LSTM to open the gripper until the target angle for the object is achieved. Since the control is based on the feedback of a learned tactile module, the system assumes no knowledge of object parameters, other than the fact that the object is prism-like and will thus experience minimal translational slip relative to the rotational motion. For the RSE, an LSTM-based model outperforms an MLP-based model on unseen objects. It is also shown that it improves performance to predict angle and velocity. The LSTM model leads to better controller performance, (compared to MLP), although even with fine-tuning both lagged behind the oracle's performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper prevents a novel method for pivoting an object in-hand in order to modify the object's orientation. In contrast to many other in-hand manipulation systems, the system does not assume any knowledge of object properties, which should lead to increased generalization. The issue of accurate and very dynamic manipulation, as tackled in this paper, is a tricky one. As stated in the limitations, the relatively simple controller serves as a good jumping off point for future work.\n\nThe paper is well-written, with the structure clearly laying out each contribution. Section 3 was particularly helpful in clearly stating the problem and the guiding assumptions. The experimental evaluation section was comparatively extensive. Given that the closest related work used an MLP in the context of translational slip, it made sense to compare LSTM and MLP.\n\nAs stated above, accurate and dynamic manipulation is tricky. However, it would strengthen the paper to better understand what could be done to improve the generalization to new objects (particularly since generalization is a strength of the approach). Is this primarily a weakness of the estimator or the controller? The paper mentions that some objects are harder to generalize to. What are the properties of these objects that contribute to this?\n\nIt would also be great to include some videos of the system in action as part of the supplementary material. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am weakly recommending the acceptance of this paper. The paper presents an interesting in-hand manipulation method that leverages tactile sensing to perform dynamic pivoting. The formulation does not rely on any object properties (aside from being prism-shaped) and thus can generalize to many objects, although there may be some more tweaks or training needed to fully fulfill this promise. The paper is well-organized and provides appropriate in-hand manipulation demonstrations.  \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "As stated above it would be great to have a bit more discussion (perhaps in the appendix, since the paper is space constrained) about how to improve the generalization to new objects (or what is a limiting factor preventing this). This could include discussion of what properties of objects make generalization difficult. For example, there is a comment in Section 6.4 that error for the toothpaste is higher. What about the toothpaste causes this?\n\nIn collecting the dataset, there were two methodologies for controlling the gripper, \"Rotate to Stop\" and \"Angle Goal\". Was there a significant difference between the data collected by these two methodologies? Would there be an impact in using data gathered by one over the other? Additionally, I believe the \"Angle Goal\" controller leverages the RSE-LSTM, which is trained based on data collected by this process. It would be helpful to clarify this.\n\nIn algorithm 1, is there any mechanism that prevents the gripper from opening past the point where the object would fall out? Would it be possible to integrate such a mechanism?\n\nIf possible, videos of pivoting the various objects (particularly the unseen objects) would strengthen the paper. \n\nThese two related papers might be worth discussing and citing: \n- Sintov, Avishai, and Amir Shapiro. \"Swing-up regrasping algorithm using energy control.\" In ICRA, pp. 4888-4893. IEEE, 2016.\n- Holladay, Anne, Robert Paolini, and Matthew T. Mason. \"A general framework for open-loop pivoting.\" In ICRA, pp. 3675-3681. IEEE, 2015."}}, {"review_id": "PL4sHIW_afX", "reviewer": "Reviewer_R21W", "summary": "The paper presents an approach for controlled pivoting of objects using tactile sensing. The outcomes of an LSTM approach are compared to an MLP based approach and shown to be superior. Vision is used as a ground truth.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe problem of controlled gravity based pivoting has been addressed by several other papers recently and so is a current topic of interest in robotics research..\n\nWeaknesses\n\nThe MLP approach that the system is compared against is not well described and motivated in the main body. Though the parameters of the MLP are given in the supplemental material, it is not clear how these were determined and why the approach of [5] was selected as inspiration. Given the key antagonistic role the MLP approach plays in this work, the authors must convince the reader that they didn't intentionally design / tune a system to have poor performance. \n\nThe paper is unclear at times and it took me some backtracking to make sense of the results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is generally interesting and the arguments for the reason for investigating the problem are compelling. My major qualm with the paper comes from the MLP comparison method, which, as mentioned in the weaknesses, I think needs more text to justify why it was selected (compared to the approaches of oether papers) and how it was tuned. Until this is done, the outperformance claim is somewhat weak. \n\nI have listed some minor issues in the following section which affect paper clarity. I mention this here as there were quite a few of these, which contributed to a sense of frustration by the time I finished reading the manuscript.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* In the first sentence of the introduction - it seems that such general claims on robotic manipulation would be better served by citing review papers rather than these specific pieces of work. \n\n* It seems that the following paper from Bristol would be a relevant addition to the literature review - 'Slip Detection With a Biomimetic Tactile Sensor' 2018.\n\n* The motivation for adding masking tape to the objects was not explained - we are simply told that friction is augmented. I assume this is to make the surface friction consistant across the objects (which simplifies the pivoting problem somewhat). Could the authors also make it clear that this was applied to all trials. I first read this as the tape being a variable.\n\n* Why are two of the objects covered in green paper but the others are not?\n\n* Are the objects full or empty? For example the deoderant and pills.\n\n* We are not told how much control is possible over the gripper, in terms of grasp aperture resolution. It would be interesting to know what is the smallest amount you can reliable move one of these grippers and how much your controller has to move it. \n\n* How are the 2x9 papillae of the tactile sensors combined to get a force output? Or are all 18 ouputs fed into the models?\n\n*  I could not make sense of Table 1 for sometime until I read in Section 7.2 that the headings were defined in section 6.2. Though I had read section 6.2, I did not recall the described labels when seeing the acroynm, as there was quite a bit of space between 6.2 and the table. There is no harm in reminding the reader in the caption of the table and using the same acroynms elsewhere, such as on Figure 5. \n\n* The benefits of slower rotational motion is argued in Section 8, but a counter-argument would be that this would slow down any processes that the robot is engaged in. An example is given earlier in the paper of shelf stacking, which would be a repetitive industrial task where speed is prized. \n\n* I think a current limitation of the method is the reliance on the tape to normalise the frictional properties of the objects and would like to see this discussed in the limitations section.  "}}, {"review_id": "ETRvAFh22mU", "reviewer": "Reviewer_Wnm1", "summary": "This work studied pivoting objects using gravity with tactile sensors. It proposed a learning-based method to track the angle and velocity of the object in hand using force-based tactile sensors. The tracking model is used in a closed-loop control to pivot the object to a target angle. The proposed LSTM model outperforms the baseline MLP models in the tracking and closed-loop control tasks. In addition, the experiments show promises of transferring to unseen objects.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It is an important topic of in-hand manipulation that can benefit from tactile sensors.\n- The proposed models achieve promising results and outperform baselines in perception and control tasks.\n- The held-out experiments also show promises to generalization to unseen objects.\nThe paper is well written and easy to follow.\n\nWeaknesses:\n- The comparison between MLP and LSTM seems unfair. The MLP only has the last 15 timestamps, but LSTM has access to longer sequences. It is unclear whether more data or the better model causes the improvement. It will be clearer if there are experiments when the LSTM model also uses the last 15 timestamps.\n- As the author mentioned in the Limitation section, the controller is relatively simple. The model requires relatively large movements to observe, which will introduce rather large errors. The author could consider comparing different controllers to achieve better performances.\n- The physical properties of the objects should be described, like sizes, mass, center of mass, and surface roughness. This helps to interpret the data distribution, and see the generalization performance to different physical properties.\n- The author should also discuss the range of suitable objects for this method. Light objects might be challenging to pivot based only on gravity.\n- The work can be improved with video demonstrations to show tactile signals, real-time tracking, and the control policy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work provided a learning-based method for gravitational pivoting with real-time tactile feedback. The proposed models showed promising results and outperformed the baseline model. The perception result was further evaluated with a controller for application. While overall is good, it could further be improved by exploring more advanced controllers, models, fair comparison between baselines, and more details about the generalization.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "As discussed in Weaknesses.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "4jzvl9TF91-", "reviewer": "Reviewer_Yu3a", "summary": "This paper looks at the in-hand manipulation of objects with a two fingered gripper by relaxing the grip to allow the object to rotate under gravity. A simple 2 state controller is used (slightly open / hold) depending on the object's angle and rate of change of angle, aiming to reach a desired angle of rotation. These are estimated using a MLP and LSTM on tactile data from two PapillArray sensors mounted on a Robotiq parallel-haw gripper. The networks are trained on several objects and tested on novel objects. Overall, the LSTM works best and is effective with an angle error of 5-10deg. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n- Fairly novel task for tactile sensing. Could be used as a benchmark to compare sensors, grippers etc.\n- Nice combination of machine learning and control.\n- I learnt that you also need to use the rate of change of angle when trying to reach a desired angle, as a correction on the latency in the system.\n\nWeaknesses\n- The controller has just two preset states (close or slightly open). How is that tuned to the different object sizes? Is that done manually? As you have a tactile sensor it would have been nice to use the sensor to set how closed and open you need to be.\n- Prediction of object angle is a basic capability that there are many ways of doing with different sensors (e.g. using image moments etc). What made your particular sensor well suited for this? I had no sense of how the tactile data was interpreted by the ML method to make the prediction of angle.\n- The scope seemed limited. Once the method was shown to work, it would have been good to see some other demonstration of its capability or more explanation of aspects of it to broaden the scope. E.g. rather than gravity you could use translate the gripper so the object is turned by knocking against something.\n- You could have been more complete in your referencing - e.g. there are plenty more on slip and related topics (including a review article by the inventors of the sensor in this paper). Not a major weakness, but I don't think there is any limit on referencing at this conference.\n- How suitable is the sensor for this task. As I understand it, it has a relatively low dimensional output (3 x 3 array) that is well suited for slip detection. However, you are using it for object pose estimation, where I would have thought more detailed sensors e.g. the DIGIT would be more effective. I would have liked to have seen some benefits of the tactile sensor you are using coming through in this study. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I like the paper, but veer towards a weak accept because the scope is narrow and the problem is somewhat straightforward to solve. I also did not learn much about how the tactile sensor data is intererpreted to allow the task to be succesful, which would have been interesting.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see comments above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)