TactileVAD: Geometric Aliasing-Aware Dynamics for High-Resolution Tactile Control
Authors: Miquel Oller
Dmitry Berenson
Nima Fazeli
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
tactile control
high-resolution tactile sensors

Abstract: Touch-based control is a promising approach to dexterous manipulation. However, existing tactile control methods often overlook tactile geometric aliasing which can compromise control performance and reliability. This type of aliasing occurs when different contact locations yield similar tactile signatures. To address this, we propose TactileVAD, a generative decoder-only linear latent dynamics formulation compatible with standard control methods that is capable of resolving geometric aliasing. We evaluate TactileVAD on two mechanically-distinct tactile sensors, SoftBubbles (pointcloud data) and Gelslim 3.0 (RGB data), showcasing its effectiveness in handling different sensing modalities. Additionally, we introduce the tactile cartpole, a novel benchmarking setup to evaluate the ability of a control method to respond to disturbances based on tactile input. Evaluations comparing TactileVAD to baselines suggest that our method is better able to achieve goal tactile configurations and hand poses.

TactileVAD: Geometric Aliasing-Aware Dynamics for High-Resolution Tactile Control
Miquel Oller
Dmitry Berenson
Nima Fazeli
Department Robotics, University of Michigan Ann Arbor, MI 48109 109, United States foller, dmitryb, n #izzicumcch... edu https: httppp://www..o www.wmam.n...mmmmmmmmm.mmmmmm... com/tactile-vad
Abstract: Touch-based control is promising approach to dexterous manipula- tion. However, existing tactile control methods often overlook tactile geometric aliasing which can compromise control performance and reliability. This type of aliasing occurs when different contact locations yield similar tactile signatures. To address this, we propose TactileVAD, a generative decoder-only linear latent dynamics formulation compatible with standard control methods that is capable of resolving geometric aliasing. We evaluate TactileVAD on two mechanically- distinct tactile sensors, SoftBubbles (pointcloud data) and Gelslim 3.0 (RGB data), showcasing its effectiveness in handling different sensing modalities. Addi- tionally, we introduce the tactile cartpole, novel benchmarking setup to evaluate the ability of control method to respond to disturbances based on tactile input. Evaluations comparing Tactile to baselines suggest that our method better able to achieve goal tactile configurations and hand poses.
Keywords: Manipulation, tactile control, high-resolution tactile sensors
Introduction
Tactile sensing is key enabler for dexterous robotic manipulation, allowing robots to per- ceive the objects they interact with and con- trol their behavior. The ability to sense and re- spond to external forces and changes in grasped object poses during manipulation is essential for a broad range of tasks including grasping, pushing, sliding, and tool-use. These tactile perception-action loops provide accuracy and robustness in the face of uncertainty while com- plementing vision by providing feedback de- spite occlusions from the robot or environment.
Tactile Sensors
Cartpole
In recent years, there has been growing inter- est in developing high-resolution, vision-based and collocated tactile sensors for robotic ma- nipulation [1, 2, 3, 4, 5]. These sensors convert tactile signatures to visual formats (e.g., RGB or point clouds) and have enabled a variety of perception tasks including in-hand pose estima- tion [6, 7, 8), object identification [9, 10, 11, 12]. and slip detection [13, 14]. However, these high-resolution tactile sensors, by virtue of their mechanical structure, introduce complex dynamics between the robot, grasped object, and the environment that must be accounted for in the control
Figure 1: Tactile Cartpole: Our proposed bench- mark task evaluates the robustness of tool grasp- ing control subject to disturbances applied by the cart.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

of the robot [15]. Current state-of ofate-of-the-arr art tactile control methods rely on black-box modeling of the sensor-ohjeet-environne interaction coupled with sample-based control [16, 17, 18] or reinforce- ment learning [19, 20].
While these approaches have shown promise in leveraging tactile feedback for control, they often overlook critical factor: tactile geometric aliasing. This aliasing occurs when different configura- tions or contact geometries produce indistinguishable tactile signals, leading to ambiguity in inter- preting the underlying interactions. Neglecting this effect can hinder planning and control perfor- mance, as we show in this paper. To address these limitations, we introduce a tactile representation and controller that is able to disambiguate geometric aliasing. Our contributions are:
VAD, a generative decoder-only linear latent dynamics formulation for tactile control with high-resolution sensors that is robust geometric aliasing,
Demonstration this method for two mechanically-distoe distinct tactile sensors with different sensing modalities: SoftBubbles [1] (point cloud data) and Gelslim 3.0 [2] (RGB data),
Tactile cartpole, novel benchmarking setup for evaluating the stability of tool grasping control in the presence of external perturbations.
2 Related Work
Tactile Control In recent years, there has been growing interest in the application of tactile sen- sors for various robotic manipulation tasks, including insertion [21], cloth manipulation [22], cable manipulation She et al. [18]. or tool manipulation [23, 15]. Tactile sensors provide valuable informa- tion for controlling the interaction between the robot and a grasped object. Some existing methods focus on learning the dynamics of sensor-object interactions [16, 3] to utilize them for model-based control. However, the high dimensionality of the tactile signals poses a challenge in modeling these dynamics and using them for controls. To address this, a common approach is to project them into space [15, 19]. Despite the potential shown by these approaches, they frequently neglect the presence of ambiguity in tactile signals arising from geometric aliasing. In our work, address this issue by constructing latent spaces capable of disambiguating aliased observations.
Latent-Space Dynamics The identification of system dynamics from raw observations plays a cru- cial role in the development of effective control strategies. Brunton et al. [24] proposed to identify the underlying dynamics of a non-linear system through sparse regression. Since observations are space [25]. Many of these approaches employ encoder-decoder architectures, and one notable framework is Embed-to-Comtrol (E2C) [26]. E2C formulates the latent as Variational Auto-encoder with locally-linear latent space dynamics over conditional distributions. This work inspired subsequent research efforts. Jaques et al. [27] explored a more constrained parametrization of the latent dynam- ies, resulting in improved learning of the latent space for PD) control laws. Furthermore, Okumura al. [28] extended this approach to incorporate tactile sensing by learning a latent space projec- tion for both tactile and visual inputs. Despite the success of encoder-decoder architectures, they introduce aliasing in the latent space when faced with aliased observations. To overcome this limitation, our approach employs decoder-only architecture and updates the latent space through back-propagation, effectively mitigating the adverse effects of aliasing.
often high-dimensional, is common to represent the system dynamics in
Decoder-Only Architectures Decoder-only models have gained attention as an alternative to tradi- tional encoder-decoder architectures. These models eliminate the need for an encoder by leverag- ing gradient-basee based optimization to infer the latent space representation directly from the observa- tions [29]. Decoder-only architectures have exhibited promising results in learning structured latent spaces for high-dimensional data generation [30, 31, 32, 33J. A notable example of this is the Vari- ational Auto-decoder (VAD) [34], which extends the well-knoww Variational Auto-encoder (VAE) [35] formulation to a decoder-only scenario. VAD incorporates a codebook that stores conditional latent distribution parameters, which are jointly optimized along with the decoder weights. This formulation regularizes and encourages a more structured and smooth latent space. In our work, we extend the VAD formulation by introducing dynamics through linear transitions in the latent space. Our proposed approach, VAD, enables the generation of temporal trajectories of states.

---

ENCODER-DECODER
83
S
X
2
x3
$3
81
S
X12
21
Z
22
82
24
X1
S4
X44
State Space
Observation Space
Latent Space
DECODER-ONLY
$4
X4
S
X
X3
Z
$1
Z1
Z3
22
X1
X2
X4
Space
Space
Space
Figure 2: Geometric Aliasing (Left) illustrates the geometric aliasing problem of the tactile sensor reading while grasping an object with geometric symmetries. Observe that two different states $1 and 52 corresponding different grasping locations result in the same sensor mcasurement X X2- However, the same action u applied both of them results in very different next-state observations. (Top Right) As consequence of the aliasing in the observations, the mapping X Z introduces ill-posed latent space. (Bottom Right) modelling only the & 2 X' results in better latent spaces.
3 Problem Formulation
In this paper, we consider the problem of controlling a system with unknown dynamics of the form 8 = /4yyn(s; u.). Here, S, S C R= represents the system state at time consisting of both robot and environment configurations. The control actions u C R correspond to the robot's end-effector motions. We assume that the true system state s, not directly measurable, but we have access observations x, obs(s:) X' R= Specifically, in our study, these observations X, obtained using based tactile sensors and take the form of point clouds RGB images. We further assume that may suffer from observation aliasing, where different system states $2 E R"",, * S2 can result in the same observations xı obs(s:) = X2 obs(s2). This aliasing can occur due to geometric symmetries and limited sensor measurement range. This situation can lead to an ill-posed inference problem, as the indistinguishable observations x1 and X2 result in ambiguity in determining the underlying dynamics. Fig. (Left) illustrates this. The goal of this problem is to determine sequence of actions uj.... ....... uy that can drive the system from some initial So desired system state Sy E S in the presence of observation aliasing.
4 Methods
4.1
Variational Auto-Decoder Linear Latent Dynamies
The key idea of our method to learn a model of the underlying system dynamics in a lower- dimensional latent space Z such that: (i) observations x, are reconstructed; (ii) the latent space dynamics linear, smooth, and structured; and (iii) aliased states are identified and disambiguated.
One common approach is to learn mapping function ó X + Z that projects high-dimensioonll observations x R= to a lower-dimensional latent space z E I= where n. & nz. The dynamics in this latent space are globally-
and given by Z++1 = fiatayn [Zc, u,) = Az, Bu, where T xn. This approach typically employs an encoder-decodee architecture observe that formulating the mapping from observation space to latent
A E R xx=
[26, 19, 25J. However, space ø X + Z introduces ill-defined latent spaces when aliased observations present in the dynamics. With reference to Fig. 2 (Right), consider two distinct states s * that result

---

in aliased observations X1 = obs(s1) X2 = obs(s2). Additionally, assume the existence of an action u E 4 that leads to non-aliased states: 83 = (s) u) and 8, = faya(s) u) where 83 7 St. Let X3 = (bs(ss) and x4 = obs(s1) be the corresponding observations, such that X3 7 X4- The resultant latent representations are denoted as 2, - ((x:). such cases, the mapping will force Z1 Z2 since X1 X2. and therefore they are compelled to produce the same resultant dynamics 2: Simldyn(2 (21, u) dyn (22, u) 2. However, the dynamics should yield distinct outcomes: /taadda (Z1, u) 23 + 24 Jimadya( u). This situation leads to an ill-posed problem, as the indistinguishable observations and X2 result in ambiguity in determining the underlying dynamics. Figure (Top Right) visualizes this effect.
To address this, we propose an alternative approach that focuses exclusively on modeling the map- ping from the latent space to observation space, denoted as ib Z + X. By adopting this surjective mapping s/, we have multiple latent vectors mapped to the same observation Each of these will evolve differently when action is taken. Once the corresponding outcomes are perceived, we can use an inference via optimization procedure (Sec. 4.3) to disentangle the aliased observations. Fig. 2 (Bottom Right) provides an illustration of this concept. Our objective is to jointly learn the mapping from latent space to observation space, 1b, and the latent space dynamics fiuldyn from state observation transitions (x1, U1, x2....; X7-1, -T XT)) To this end, we employ a decoder-only architecture. Additionally, we choose model the latent space as conditional distributions, which leads to more structured and smooth learned latent spaces. Consequently, our model takes the form of a VAD with globally-linear latent dynamics, in contrast to E2C, which is formulated as VAE. The latent space dynamics are of the form:
Z₁ ~ ((z)) = -N E,); Z ~ Q(Z++1 u.) = -A + Bu, AE,AT)
We have chosen this parametrization of latent dynamics to ensure compatibility with standard control methods like LQR. Section 4.4 describes our control formulation.
In our method, we eliminate the need for an encoder by maintaining a codebook of size C com- posed of normal distribution parameters {14, E.i........ The size of the codebook is given by the number of states in the training data, since for each training state, we initialize a codebook ele- ment. These parameters are optimized jointly with the decoder's, and the dynamics parameters via back-propagation. The training process is described in detail in Section 4.2. During the inference, when mapping from observation space latent space, employ optimization-bant as explained in 4.3. This inference procedure enables us to accurately determine the latent space representation based on the observed data. Furthermore, it allows us to generate multiple likely can- didates for an aliased observation, providing a more comprehensive understanding of the underlying latent dynamics.
approach,
4.2
Tactile VAD Training
We train our model from dataset of N trajectories of T state-action transitions of the form D {xx,, un X..... XT, ur, (T+x)) Our goal is to jointly optimize the latent space parameters
Xa
Xa+1
Adc Auttre
Pi
X
3
KL
E..
AT
Figure 3: VAD Latent Linear Dynamics Model (Left) Tac for trajectory generation of length T via latent space linear dynamics propagation. (Right) single-step training with reconstruc- tion, prediction, and latent space consistency.

---

(pg: Ei..=I..((T+1)))) the transition dynamics A and B, as well as the parameters that define the mapping from latent space to observation space fase- The training procedure is outlined in Algorithm 1. To jointly learn this, we use a loss function composed of four weighted terms:
C Cree (xe; qt) + Olpred (x+ 1x++1, 0r+1) + Clonss (Ct+1 qt+11 + Creg (4ts (t+1)
where (i) free imposes that are able to reconstruct the current observation from the current latent distribution, (ii) Eped imposes that we are able to generate the next observation from the predicted distribution dynamics given by the control action u, (iti) Leons imposes that the predicted latent distribution matches the distribution, and (iv) Leeg imposes a prior distribution on the latent space which regularizes the latent space. See Appendix A.I for more details about the training and the loss formulation.
4.3 Embedding via Inference Search
Decoder-only architectures exclusively model the mapping from latent space to observation space v 2 - X. Unlike encoder-decoder architectures, which can infer the latent representation 2, by forward-passing the encoder, decoder-only architectures do not directly model the mapping ø X + 2. Instead, they employ an inference-based encoding approach to obtaín a set of latent vectors corresponding to a given observation. Here, infer z values by fixing the decoder weights 0 and performing gradient descent the training loss to compute the most likely values that produced x, similar to the procedure proposed in [30].
2 arg min Lame ( (S (z), x)
Although this inference approach is more computationally expensive than a forward encoding pass, it offers two main advantages: (i) better generalization to unseen data and distribution shifts and (ii) generation of multiple candidate latent states that result in the same observation. The ability to generate multiple candidates helps disambiguate aliased observations by capturing the inherent uncertainty in the latent space representation. To generate multiple candidates, we initialize a set of values for the latent elements to be optimized. this work, we combine random sampling with priors. Specifically, given the first tactile observation, we initialize the candidates by uniformly sam- pling the latent space and the top n elements in our codebook that are most similar to the current observed state. We add Gaussian noise to the codebook values to obtain a diverse set of initial can- didates. During optimization, these latent elements will converge to various local minima, resulting in diverse set of latent candidates. To maintain temporal consistency over the trajectory, we prop- agate candidates using the latent space dynamics with additive Gaussian noise. Generating multiple candidates is easily parallelized, resulting in only minor addition to computation (see Sec B.1.)).
4.4 Low-level Control with VAD Dynamics Model
The goal of the controller is to drive the system state S; to a desired target state Sg. Given that we only have access to an aliased observations X, of the system state st, we formulate our controller in the learned latent space Z since it does not suffer from aliasing and has the added convenience of linear transition dynamics. Given a quadratic cost, we formulate discrete time LQR controller:
(P1) min
2 QZ: + u, Ru,
1=1
s.t.
Z
Z₁ Zg
t= 1=..,,,,0 00,
= Az, + Bu, t= 1,..., 0C
with Q = O, R > 0. The optimal controller is computed by solving the discrete-time algebraic Riccati equation (DARE), yielding u, = - K [36]. Here, we choose = nu When ny >> flu the resulting system dynamics are underactuated and the controller may not have a stable solution. Algorithm summarizes our controller formulation.
One consideration is that approach generate multiple latent candidates for given observed state. To integrate this capability with controls, maintain a history of candidates, initialized as
5

---

described in section 4.3. At each step, we propagate the dynamics on the candidate set and evaluate the similarity of their decoded state observations with the actual observed sequence. We select the candidate with the lowest state trajectory loss against the observed state observation trajectory. This gives the current latent value 2₁ that the controller takes to produce the optimal action.
5 Experiments and Results
We demonstrate the effectiveness our proposed approach to tactile control in tasks: an aliased-rich simulation task with discrete states, where many of them exhibit aliasing, (ii) real- world tactile control for static tool grasping with the Soft Bubbles tactile sensors and (iii) Gelslims 3.0 sensors, and (iv) the tactile cartpole: novel quasi-dynamic tactile control bench-marking task for tool grasping under external perturbations inspired by the classic cartpole control task.
5.1 Baselines
We benchmark method with 3 baselines, including 2 encoder-decoder approaches and one decoder-only approach. All methods have linear latent dynamics. See Supp. B. for details.
Auto-encoder Dynamics (AE) This baseline is a common approach that uses auto-encoders to shape the latent space [25]. The encoder projects high-dimensional observations into the latent vector space where dynamics are propagated. Predictions are then projected back to the observation space using the decoder. Controls are formulated the low-dimensional latent space.
Embed Control (E2C) Embed to control [26] is popular framework latent space dynamics that extends the auto-encoder formulation the variational setting. The resulting latent spaces distributions over vector spaces and have been shown to perform well for control tasks. For consistent evaluation, use the global-E2C formation (globally linear dynamics).
Auto-decoder Dynamies (AD) This decoder-only framework optimizes a latent codebook simulta- neously with the dynamics parameters and the decoder parameters. The main difference from our proposed approach is in the latent space, which is not a distribution. Here, the optimized codebook not distribution parameters, but the latent vectors directly.
5.2 Moving Block Control
State (7.6)
State Observation
State (5,2)
State Observation
This task aims to demonstrate the im- pact of aliasing when modeling the dynamics on an aliased-rich task with discrete states, Fig. 5 provides a vi- sualization of this task. The objec- tive to control the planar position of the rectangular object (in yellow) from image observations, where the object undergoes 2D translation tions. The states s, represent the ob- ject's position in the 2D plane, while the observations X, capture only a partial view of the full state (depicted as the blue subpart around the center). Fig. also shows an example of observation aliasing, where two different states lead to the same observation due to the small sensed area.
(2244)
State (7.4)
State (5.0)
Observation
State Observation
Figure 4: Moving Block Setup On the left, two different block locations result in the same observation. However, the same action applied to both states resulted in different next states (right).
We benchmark approach and compare to the baselines. The goal is to control the block position to reach desired target location. We evaluate the reconstruction and prediction quality as well as the LQR-controlled performance of the resulting dynamics models. Tab. summarizes the results. We note that the decoder-only models (AD and VAD) result much lower reconstruction error and prediction errors. This can be explained because the learned latent space from decoder-only models does not suffer from aliasing. particular, our approach (VAD) consistently obtains the best scores long-horizon predictions, which suggests that our method results in better-structured latent spaces (see Appendix C.I). Furthermore, approach results in better controlled achieved pose (Tab. first column). See Appendix B.2 for more details.

---

Bubbles Target
Gelslim Target Tactile Signature
Tactile Signature
(A)
(
Figure 5: Tactile Rod Grasping (A) Rod grasping with Soft Bubbles where tactile signatures are encoded depthmaps. (B) Rod grasping with Gelslims 3.0 where tactile signatures encoded as RGB images.
5.3 Tactile Rod Grasping
LQR Pose Errors [px]
Prediction Error
(-10-2)
Method
Reconstruction Error
The goal of this task is to con- trol the relative pose between the robot end-effector and grasped object from only tactile data. The object being grasped is 3D 3D-printed rod designed to sim- ulate a tool handle. This rod fixed in the scene to facilitate easy reset and data collection. It important
Mean 1
Std
step
step
step
AE
2.13
4.36
2.21
10
21.19
318.4
1045.3
1.01
3.02
7.07
10
102.4
110.6
107.7
AD VAD (ours)
2.90
3.22
2.41
10-1-10-7
2.231
40.57
78.77
0.53
2.37
1.14
10
0.726
30.74
62.96
Table I: Moving Block Evaluation: LQR Pose Errors = error
between an unaliased target pose and the pose achieved after 10 control steps following LQR control law. Reconstruction Error and Prediction Error for 500 test data trajectories propagating la- tent dynamics.
that the con- troller does not have access the robot pose with respect to the grasped object; it solely relies on tactile measurements. The control actions are the changes in position and orientation of the robot end-effector. The control actions are constrained to a box, requiring the robot to perform multiple grasps to achieve desired relative pose.
We evaluate our methods two tactile sensors with distinct mechanical properties and sensing modalities [15]: (i) the Soft Bubbles are composed of two inflated membranes that deform when in contact. This sensor exhibits large deformations that are sensed via depth camera. We also evaluate our methods on the Gelslims 3.0 (ii) which are more rigid than the boft-bubbles and the sensor deformations are sensed via RGB images.
To train the models, collect 500 trajectories of robot-rod interactions under random actions for each sensor. The length of these trajectories is 10. The rod used in these interactions has cylindrical dimension of 100mm in length and a diameter of 20mm. In the case of the Soft Bubbles, the observed states are encoded deformation maps of the sensor, representing the difference between the current sensor depth map and a reference undeformed state. On the other hand, for the Gelslims sensor, the observations are grayscale image differences between the current state and reference undeformed state.
To assess the performance of the learned methods, execute 20 trajectories using LQR troller, starting from random configurations of the robot grasping the rod. The objective is to achieve a specific robot-rod configuration, where the rod is grasped at the center of the sensor and aligned with the end-effector. This goal configuration provided as a non-aliased tactile observation. The evaluation was conducted using three different rods: the same 20mm rod used during training, as well as two novel rods of sizes 15mm and 30mm, assess the generalization capability of the methods varying geometries. The pose error between the final achieved configuration and the tar- get configuration was computed as measure of performance. Detailed information the metrics computation can be found in Appendix B.3. Tables and 6 present a summary of these metrics.
7

---

Bubbles Rod Pose Errors [mm]
Gelslim Rod Pose Ervors [mm]
Method
20mm
Rod
(Train)
15mm
Rod
30mm
Rod
20mm
Rod
(Train)
15mm
Rod
30mm
Rod
Mean
Std
Mean
Std
Mean]
Std
Mcan
Std 4
Mcan 4
Std 1
Mean.
Std]
AE
41.79
37.69
27.27
36.92
53.86
52.44
16.77
22.67
27.27
36.92
11.66
16.29
E2C
25.29
31.65
77.11
58.14
31.08
23.82
14.45
15.15
18.22
10.26
16.01
9.69
58.05
36.27
65.46
31.37
62.26
39.04
64.13
18.00
40.61
36.96
80.22
11.74
VAD (ours) 5.23
4.06
10.91
3.99
7.68
2.67
8.54
16.01
8.98
7.27
10.39
19.83
Table 2: Tactile Rod Grasping Evaluation: Measuring the error between the planar pose achieved
for 20 control steps and the target (rod vertical). For each sensor, we test on the training rod (20mm diameter) as well unseen rods (15mm and 30mm diameter). Statistics are reported over 20 trials per method and rod.
Notably, our method consistently yielded lower errors in achieving the desired tactile configuration and pose, demonstrating its effectiveness for both known and unknown objects, as well as across different sensor modalities.
5.4 Tactile Cartpole
Number of Steps
Rod Angular Mean
Method
Meant
Std 4
Error [rad]
The goal of this task is to demonstrate the utility of our method as a low-level controller for tool manipulation tasks. To this end, we combine our learned tactile low-level controller to cor- rect the relative robot-tool position with a high- level controller that controls tool orientation.
AE
11.84
7.42
0.29419
E2C
8.12
3.17
0,2682
AD VAD (ours)
11.73
7.95
0.2057
73.06
23.25
0.09267
Table 3: Cartpole Control Evaluation: Report-
ing the number of perturbations steps without let- ting the tool slip out of hand or dropping the tool and the rod angle error with respect to the target vertical rod. Statistics are reported over 20 trials per method.
The setup is composed of cylindrical rod con- nected via a revolute joint to an actuated cart on a rail. The rod is grasped by the robot and simu- lates the tool handle. Fig. 1 illustrates this task. By moving the cart, we perturb the grasped tool pose, serving as an automated mechanism. The goal of the robot is to keep the rod vertical and aligned with its end-effector. Therefore, the cartpole motions force the robot to correct and adapt to the resulting perturbed tool pose.
We compare our method with the baselines, all sharing the same high-level controller, differing only on the low-level robot-rod align motion. We evaluate the time the robot is able to keep the rod vertical without dropping it as well as the average angle error, Tab. Our method considerably outperforms the baselines and is able to maintain the tool vertical for longer periods of time and track the desired relative orientation significantly more successfully. This suggests that approach is able to better generalize to novel tasks.
Discussion and Limitations
One limitation of decoder-only architectures is that obtaining the latent representation requires mul- tiple passes through the decoder using inference-via-onn via-ooiimiazatiinn, which can be slower compared to encoder-decoder architectures that only require a single forward pass (see Appendix B. 1.1). Another limitation of our approach is its limited exposure to cylindrical rods. Since tools can have various handle geometries such as rectangular or curved handles, generalizing the learned model becomes a challenge. To address this, we propose learning separate models for each primitive handle geometry. At runtime, handle geometry-awan classifier can be used to select the appropriate controller based on the current grasped tool geometry, enabling better adaptability different handle shapes.
Finally, both the Gelslims and Soft Bubbles tactile sensors are smooth elastomers despite their differ- ent mechanical properties. In our approach, we employ decoder architectures based on convolutional neural networks (CNNs) to generate the sensor data. However, is worth noting that these CNN- based decoders may not fully capture the inherent characteristics of the sensors, like smoothness and continuity. To overcome this limitation, alternative methods can be explored, which specifically incorporate smoothness constraints on the sensor geometry. By adopting such methods that better capture the three dimensioil geometry of the sensors, more accurate predictions can be achieved, thereby improving the overall performance and reliability of the system.
8

---

Acknowledgments
This research project supported by Toyota Research Institute under the University Research Pro- gram (URP) 2.0 and "la Caixa" Fellowship Program (ID 100010434).
References
[1] A. Alspach, K. Hashimoto, N. Kuppuswamy, and R. Tedrake. Soft-bubble: A highly compliant dense geometry tactile sensor robot manipulation. In 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft), pages 597-604, IEEE, 2019.
[2] Taylor, S. Dong, and Rodriguez. Gelslim3. O: High-resolution mcasurement of shape, foree and slip in compact seniilg finger. arXiv preprint arXiv:2103. 12269, 2021.
[3] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, 5(3):3838-3845, 2020.
[4] A. Yamaguchi. Fingervision tactile behaviors, manipulation, and haptic feedhack teleoperation. In the 4th IEEJ international workshop on Sensing, Actuation, Motion Control, and Optimization (SAM- CON2018), 2018.
[5] W. K. Do and M. Kennedy. Densetact: Optical tactile sensor for dense shape reconstruction. In 2022 International Conference on Robotics and Automation (ICRA), pages 6188 6194, IEEE, 2022.
[6] M. Bauza, A. Bronars, and A. Rodriguez. Tac2pose: Tactile object pose estimation from the first touch. arXiv preprint arXiv:2204. 11711, 2022.
[7] N. Kuppuswamy, A. Castro, C. Phillips-Grafflin, A. Alspach, and Tedrake. Fast model-based contact patch and pose estimation for highly deformable dense-geometry tactile sensors. IEFE Robotics and Automation Letters, 112::181118188 2020. doi:10. 10..10.11 (t..9/.RA.001.2911000
[8] T. Kelestemur, R. Platt, and T. Padir. Tactile pose estimation and policy learning for unknown object manipulation. arXiv preprint ar arXiv:2203. 10685, 2022.
[9] M. Bauza, E. Valls, B. Lim, T. Sechopoulos, and A. Rodriguez. Tactile object pose estimation from the first touch with geometric contact rendering. arXiv preprint arXiv:2012.05205, 2020.
[10] J. Kerr, H. Huang. A. Wilcox, R. Hoque, J. Ichnowski, R. Calandra, and K. Goldberg. Learning self- supervised representations from vision and touch for active sliding perception of deformable surfaces. arXiv preprint 220: 13042, 2022.
[11] J. Lin, R. Calandra, and S. Levine. Learning to identify object instances by touch: Tactile recognition via multimodal matching. In 2019 International Conference Robotics and Automation (ICRA), pages 3644-3650. IEEE, 2019.
[12] J. Zhao, M. Bauza, and E. Н. Adelson. Fingerslam: Closed-loop unknown object localization and recon- struction from visuo-tactile feedback. arXiv preprint arXiv:2303.0777 0779977 2023.
[13] S. Dong, W. Yuan, and E. H. Adelson. Improved gelsight tactile sensor for measuring geometry and slip. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 137-144. JEEE, 2017.
[14] S. Dong, D. Ma, E. Donlon, and A. Rodriguez. Maintaining grasps within slipping bounds by monitoring incipient slip. In 2019 International Conference on Robotics and Automation (ICRA), pages 3818 3824. IEFE, 2019.
[15] M. Oller, M. P. Lisbona, D. Berenson, and Fazeli. Manipulation via membranes: High-resolution and highly deformable tactile sensing and control. Conference on Robot Learning, pages 1850-1859. PMLR, 2023.
[16] S. Tian, F. Ebert, D. Jayaraman. M. Mudigonda, C. Finn, R. Calandra, and Levine. Manipulation by feel: Touch- control with deep predictive models. In 2019 International Conference on Robotics and Automation (ICRA). pages 818-824. IEEE, 2019.
[17] C. Wang. S. Wang. B. Romero, F. Veiga, and E. Adelson. Swingbot: Learning physical features from in- hand tactile exploration for dynamic swing-up up manipulation. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5633-5640. IEEF, 2020.

---

[18] Y. She, S. Wang. S. Dong, N. Sunil, A. Rodriguez. and E. Adelson. Cable manipulation with tactile- reactive gripper. The International Journal of Robotics Research, 40112-14)1385-1440,, 2021.
[19] H. Van Hoof, N. Chen, M. Karl, P. van der Smagt, and J. Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 TEFE/RSJ international conference intelligent robots and systems (IROS), pages 3928-3934. IEFE, 2016.
[20] Y. Zheng, F. F. Veiga. J. Peters, and V. J. Santos. Autonomous learning of page flipping movements via tactile feedback. IEEE Transactions on Robotics, 38(5):2734-23) 2749, 2022.
[21] S. Dong, D. K. Jha, D. Romeres, S. Kim, D. Nikovski, and A. Rodriguez. Tactile-rl for insertion: Gen- eralization to objects of unknown geometry. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 6437-6443. IEEE, 2021.
[22] N. Sunil, S. Wang, Y. She, F. Adelson, and A. R. Garcia. Visuotactile affordances for cloth manipulation with local control. In Conference Robot Learning, pages 1606.1606. PMLR, 2023.
[23] H. T. Suh, N. Kuppuswamy, T. Pang. P. Mitiguy, A. Alspach, and R. Tedrake. Seed: Series elastic end effectors in 6d for visuotactile tool use. In 2022 JEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pages 4684 4691. IEEE, 2022.
[24] S. L. Brunton, J. L.. Proctor, and J. N. Kutz. Discovering governing equations from data by sparse iden- tification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113(15): 3932-3937, 2016.
[25] B. Lusch, J. N. Kutz, and S. L. Brunton. Deep learning for universal lincar embeddings of nonlinear dynamics. Nature communications, 9(1):4950, 2018
[26] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
[27] M. Jaques, M. Burke, and T. M. Hospedales. Newtonianvac: Proportional control and goal identification from pixels via physical latent spaces. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4454 4463, 2021.
[28] R. Okumura, N. Nishio, and T. Taniguchi. connector socket insertion. arXiv preprint arXiv:2203.05955. 2022.
newtonianvae for high- accuracy industrial
[29] S. Tan and M. L. Mayrovoumiotis. Reducing data dimensionality through optimizing neural network inputs. AICHE Journal, (166:1447)-448) 1995.
[30] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[31] J. Mu. W. Qiu, A. Kortylewski, A. Yuille, N. Vasconcelos, and X. Wang. A-sdf: Learning disentangled signed distance functions for articulated shape representation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 13001-13011, 2021.
[32] Y. Wi, A. Zeng, P. Florence, and N. Fazeli. Virdo++: Real-world, visuo-tactile dynamics and perception of deformable objects. arXiv preprint arXiv:2210.0370 2022.
2022.
[33] M. Van der Merwe, Y. Wi, D. Berenson, and N. Fazeli. Integrated object deformation and contact patch estimation from visuo-tactile foedback. arXiv preprint arXiv:2305. 144700 2023.
[34] A. Zadeh, Y.C. Lim, Liang, and L.-.. Morency. Variational auto-decoder: A method neural generative modeling from incomplete data. arXiv preprint arXiv:1903.00840, 2019.
[35] D. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61114 2013.
[36] T. Pappas, A. Laub, and N. Sandell. On the numerical solution of the discrete-time algebraic riccati equation. IEEE Transactions on Automatic Control, 25(4))631-641, 1980.
[37] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695. 2022.
[38] R. Lopez, J. Regier, M. L. Jordan, and N. Yosef. Information constraints on auto-encoding variational bayes. Advances in neural information processing systems, 31, 2018
10

---

[39] I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statisticx, pages 2207- 2217. PMLR, 2020.

---

Appendix A Implementation Details
A.1 Tactile VAD Implementation and Training Details
In this section, we describe how we train the TactileVAD. Algorithm 1 describes in detail the training proce- dure. Note that the algorithu is similar the VAD training algorithu but with latent space dynamics trajectories. As result, the TactileVAD training optimizes (i) the decoder parameters 8, (ii) the latent dynam- ies parameters A, B, and (iii) the latent conditioned distribution parameters ja and E. The optimization cost function is composed of 4 terms, and it is described in detail in Scctions 4.2 and B.1.
The loss function is composed of four weighted terms:
C Cree (Xe, qt) Gped Q-11) + acoas ($1+1: qe+1) + (9: Each of the losses is formulated as follows:
Lree (Xe; qe) = Bex [- log Po(xx](s)] 2
Lame (X,, fdee faco (2(()))
m=1
M
pred (x (x++1, (o+1) Bar- m I- log Po (x+++ |2+11]
1 (244))
m=1
Lonnd Comm(@rt: (1 91+1) (L[
Lieg (Mt, (u41) (L(aill((O,L)) -KL(@-/\W(O.))
(4)
Algorithm 1: Tactile Training Algorithm
Input:
K: Number of samples
T: Number of timesteps
(uo; u1.... -T-1): Initial control sequence
funddyn {A (0) ( < Initialization;
Sdec 10, (0)
Initialization;
q: {up
(0) (0)
Initialization;
4+0;
while not converged do
[2e] ~ (Ze Hi (k)
(x))): ):
((2++13 P4 2
(8) (k)
N(A Bu, AΣ,T);;
[2+1]] a( Pi+1 i-1)
(1)
xx] [fec
10
x++1 [fdee (2++1) 1)
11
Lec Ez, Ez,nge log Po (x mz
12
ppd E2 :1 -
log Po (x++1 11+1));
Lunns KL(g pc,
(k) Σ. (K) I| M Σ (11) )):
(K)
4a KL(g(Zei M
(K)
Σ (K)
241 ((()))
15
1 = arechree + ppod + Reonssoons + onlu
(0, A, B, u4, Et, p441, E141 (x+1) (k+1)
grad step E..
(1);
17
k +1;
A.2 Tactile VAD Control Details
In this section, expand Section 4.4 describe the integration details of the TactileVAD model with the controller. Given that designed the latent space dynamics latent dynamics with the LQR control law directly. Algorithm summarizes the control method employed. Note that the controller is formulated in the latent space Z and therefore needs to project the observations to their respective latent values compute optimal actions (step in Algorithm 2).
linear, we integrate TactileVAD's learned
12

---

Algorithm 2: Tactile Control Algorithm
Input:
xg Goal observation representing
Q: quadratic state cost
R: quadratic action cost
number of inference iterations
K f LQRSolution( Q, ;
2g t- inference, faz: (x,)=
while not converged do
X < get.observation()
f inference Jar (x,)
ur f- ((2 Zg);
send to actuator(u;))
k k 1;
Appendix B Experimental Details
B.1 Baselines Details
All four benchmarked models share the same decoder architectures and latent dynamics parametrization. The encoder-decodde approaches (AE and E2C) share the same encoder structure. However, the key difference is in the latent elements. The non-variational approaches (AE and AD) latent elements take the form of vectors, i.e. IR*,, where the AE encoder computes the latent vector directly. The variational approaches (E2C and VAD) employ conditional distribution as latent representations. Therefore, latent elements z are composed of the mean and covariance parameters of the conditional normal distribution: z = (jt, diag Σ)) where qi =
N N(w. E.). Therefore. the E2C encoder produces normal distribution parameters and diag [). similar to VAE. For the encoder-decoder approaches (AE and E2C) the latent elements obtained via forward call encoder. Decoder-only approaches (AD and VAD) optimize a codebook latent vectors z time, they obtain the latent elements using inference via search as described section 4.3. Fig. B.3 shows the schematic for AE over trajectories. Fig. B.2 shows the schematic for E2C over trajectories. Fig. B,3 shows the auto-decoder architecture.
run-
The baselines, as well as proposed model, are trained with a loss composed of four terms:
L Clres Lree (xx, 24) + Cppred (X++1, (Xe+1
2c+1) Clcans (2:+1, Ze+1) + cKreg Crag (Zt, 2x+1)
For encoder-decoder approaches (AE and E2C) the latent elements obtained via forward call of the coder. Decoder-only approaches (AD and VAD) optimize codebook of latent vectors and run-time, they obtain the latent elements using inference via search as described in section 4.3.
Next, we will describe each term equation
X
X
X3
X
x7+1
MSR
MB
Sd MSB
10g
1gg"
T
Figure B.1: AE Model
13

---

1. Les Reconstruction Loss: Incentivizes the latent space to reconstruct the observed state i.e. vb(z)) and is formulated in the state observation space.
Lis (Xe; Las (Xe, (b(2))
For encoder-decoder approaches (AE and E2C), the latent element is obtained via an encoder call. There- fore, this can also be expressed as:
Lee 2+) (Xa, ((ze)) = (Xe, ((((x)))
Decoder-only approaches (AD and VAD) optimize the codebook via backpropagation.
Lped Prediction Loss: Incentivizes the next state prediction to reconstruct the expected next observed
state. It is formulated in the state observation space.
Lped 2.-1) Lats (X @ ayn (Ze, u+)))
For encoder-docoder approaches (AE and E2C), the latent element is obtained via an encoder call. There- fore, this can also be expressed as:
(xt+1 2c+1) = (xe, (b(()) = Luhs (x,-1, v (6(x)), uz)))
Decoder-only approaches (AD and VAD) optimize the codebook 2, via backpropagation.
Lcons Consistency Loss: Incentivizes the latent space be consistent with the imposed dynamics.
For variational approaches (E2C and VAD) this term takes the form of KL divergence between the predicted latent distributions and the expected next latent distribution:
Loass (2e+1, Zt+1) KL (z++1)
For non- variational approaches (AE and AD), this term is the MSE between the expected next latent vector Ze+1 and the estimated next latent vector 2.+1:
Lexes (Ze41, Z(41) ( Zc+1)
Lag Regularization Loss: Regularizes the latent elements they not arbitrary large.
For variational approaches (E2C VAD) this term takes the form of KL divergence between latent distributions and the standard normal distribution (0, I):
Lung (Zt, 2.+1) KL(2, (O, + (L
For non-variational approaches (AF. and AD), this term is the 1.2 norm of the latent vectors:
Lveg (Ze, Ze+1)
|22 + ||z++1||2
B.1.1 Encoder-Decodee vs Decoder-Only Inference
The AE and E2C haselines are instances of encoder-decoder architectures, while our approach (VAD) and the AD baseline are instances of decoder-only models. As seen in section 4, encoder-decoder models are sensitive geometric aliasing. while decoder-only models robust to this effect. In exchange for this ability, decoder-only models need to embedded via optimization search described section 4.3, which more computationally expensive than the forward encoder call required in incoder-decode models. this section, we quantify this difference in our setup for the Moving Block simulated task, described in Section 5.2. We measure the average time obtain batch of B embeddings Z1, ... Zg corresponding Xa. Our setup is composed of Nvidia RTX 2080 GPU and AMD Ryzen 9 3950X CPU. ...,... x.....>
series of observed
states X1, report average computation time to obtain batch of [1, 10, 100, 1000, 10000) averaged over samples. Table 4 summarizes the results.
1
X
X
x
R
XL
KL
KL
KL
W
12
Uy
X1
X2
X3
XT
XT41
Figure B.2: E2C Model
14

---

Average Embedding Time [s]
Method
Batch Size
10
100
1000
10000
Encoder-Decodee (AE)
5.67 10-5
5.66
10-5
5.66
10
5.68
10
5.69
10
Decoder-only (AD)
0.390
0.415
0.465
0.480
0.483
Table 4: Encoder-decoder Embedding Computational Cost: We compare the average time for obtaining an embedding between an encoder-decoder architecture (AE) and decoder-only archi- tecture (AD). The former encodes via forward encoder call which faster than the embedding- via-inference that the decoder-only architecture (AD) requires.
B.1.2 Auto-Decoder (AD) vs Variational Auto-Decoder (VAD) Latent Space Structure
The auto-decoder (AD) baseline as well as our proposed methad are instances decoder-only architectures. By construction, both methods are robust against the geometric aliasing effects (sce Sec. 4). However, the main difference between these approaches is the topology of the resulting embedding spaces. Prior work in image embedding and reconstruction [35, 37, 38] and learning latent space dynamics [26, 27] has shown how manifold of embedding space of auto-encoder models non-convex, takes on irregular geometry, contains holes, is often not smooth. F Fig. B.4 illustrates examples of these pathologies even for the simplest of dynamical systems. The key obscrvation is the "jumbled" latent space of the AE model vs the much more well rell-structured and regular latent space of the variational approach. Paths within the latent space of the latter are better connected, regular, and inference within this space significantly better posed. The consequence AE pathologies is that trajectories in the latent space can easily become out of distribution and cumulative error leads to catastrophic failure. Further, learning latent space dynamics from the "jumble" and performing controls is significantly complicated. Beyond our setup, in the context of image compression, these failures correspond poor and nonsensical reconstructions [35. 37, 38]. For dynamics,
even presence heavy regularization,
these issues result in severe divergences and instability [26. 27].
contrast, variational approaches impose a much more explicit structure on the latent space, imposing a con- ditional normal distribution. This structure avoids many the pathologies of vanilla auto-encoder approaches and has become one of the dominant approaches in encoding [35, 391. In addition, the variational approach allows for principled sampling, out-of-distribution detection, and more uniform distribution of latent states that more amenable dynamics propagation.
To evaluate the difference between an AD and VAD embedding spaces, we perform inference known latent elements from codebook and compare inferred states to the ground truth ones. The infer- ence is initialized with the known latent space embedding zmm with additive Gaussian noise. To account for the difference in latent space scale, we define the noise level proportional to the latent space dispersion, obtained via the stored codebook. The latent space is given by Zeth Rm for the AD and Zgth (M, Σ) for the VAD. We evaluate two metries: i) the L2 distance between the inferred latent state and the expected latent state Zih. and ii) the decoded observed state x" = (x22)) similarity with the expected observed state Xgth For a fair comparison between the different latent spaces, we evaluate the similarity over the normalized latent elements so scales are comparable. Table 5 summarizes the results. The results show that the AD is more sensitive to the initial search values than the VAD suggesting that the latent space of the VAD is more smooth and well structured.
of
X1
X2
XT+
Booiong
22
B
Figure B.3: Auto-Decoder Latent Linear Dynamics Model (AD)
15

---

Auto Decoder (AD) Moving Block Latent Space
Space
Space
Boch pociion
X peation
Variational Auto Decoder (VAD) Moving Block Latent Space
15.5
Figure B.4: AD VAD Latent Space Inference: (Top) AD learned latent space and inferred values for 0.1 noise level. (Bottom) VAD learned latent space and inferred values for 0. noise level. We plot the mean of the latent conditional distributions for the VAD. Color encodes the true state space location (left). We note pathological latent space AD latent space which leads to poorer inference results. The key observation is how wool-p the latent space topology is (in terms of regularity, smoothness, and coverage) for VAD model. This more structured latent space results in better inference results since the inferred latent space values (right) match closer the expected latent values (center).
Latent Similarity Score
State Similarity Score
Noise Level
AD
VAD
AD
VAD
0.001
3.963 10
2.108
10-4
6.146
10-4
1.485
10-3
0.01
5.013
10
2.358
10-4
7.875
10-4
1.487
10-3
0.02
1.379
10
3.0685
10
1.220
10-2
1.493
10-3
0.05
9.765
10-3
7.886
10
0.3274
1,334 10-3
0.1
4.052 10-2
2.601 10-3
2.4107
1.729- 10-3
0.2
0.1693
0.01035
7.317
0.09373
0.5
1.107
0.0691
16.35
2.473
Table 5: VAD AD Latent Space Evaluation: We evaluate the noise effect on the initial latent
state estimation for the inferred latent similarity and the resultant decoded state.
B.2 Moving Block Control Details
The moving block task defined by 20 20 grid that contains 3 block. The sensed arca is x and centered in middle the space. The true block state is defined by block top corner coordinates (r, y) The block motion defined by limited on box of size = (8r, 6y) E l S 3, 3] x [-3,]]. States are encoded binary masks The pose error is computed the Manhattan distance. The models were trained 500 random trajectories of length 20 steps. The reconstruction and prediction loss observations computed as BCE against the ground truth states.
B.3 Tactile Rod Grasping Details
The rod grasping task is defined over SEE(2 space of robot-rod configurations. The space limits are S C [-20, 20)
[- 20, 50] [- 1 (mm mm xrad). Rohot actions constrained within box defined
(ōx, dy. 50) E 24 C [-6, 6) x [-5, 5] x [-0.09m, 0.09m] (mm rad)
16

---

Bubbles Final Imprint Errors (-10-") [m]
Gelslim Final Impeint Emors
Method
20mm Rod (Train)
15mm Rod
30mm
Rod
20mm Rod (Train)
15mm Rod
30mm Rod
Mcan 4
Std 4
Mean]
Std
Mcan
Mean
Mcan..
Std
AE
8.6062
4.3312
2.3818
2.1589
15.269
12.121
30.009
9.90
27.22
4.17
18.84
8.319
F2C
5,9678
5.2829
4.4046
4.6345
3.9664
2.0997
25.28
7.576
29.20
7.27
19.05
5.79
AD
6.5889
3.4163
3.3810
21.076
3.75
15.98
1.51
VAD (ours)
2.3723
1,09266
0.6295
0.2842
3.1057
2.67
19,631
12.252
23.91
9.44
14,52
6.34
Table 6: Tactile Rod Grasping Evaluation: (Tactile similarity)
For the bubbles sensors. states are encoded as deformation depth maps of size (2, 25, 20). encoded as color differences encoded grayscale of size (2, 20, 20)
Gelslim data is
Pose Score(
1/2
MSE(21, MSE( U2) M,E(( 02)
01
02
where ry is the radius of gyration of the rod.
Appendix C
Additional Experiment Results
C.1
Moving Block Control
this section, we provide further experiments for the moving block control simulation task. Figure C.1 shows instances the reconstructed and predicted states our method and the baselines. Note that encoder-based baselines suffer from aliasing.
Reconstruction
Predictions
Ground Truth
VAD (ours)
AD
E2C
AE
time
Figure C.I: Moving Block Evaluation: Reconstructed (orange) and predicted (blue) states from latent space inference and dynamic propagation. E2C and AE methods suffer from aliasing resulting on vertical uncertainty of the block position prediction.
C.2 Tactile Rod Grasping
this section, we provide further experiment results for the tactile rod grasping task. particular, we evaluate how similar are the achieved states in terms of lactile observation similarity to the desired ones. Table summarizes these results. To sum up, VAD produces better similarity results compared to baselines.
17


Reviews:
[{"review_id": "xj6QXCDPedj", "reviewer": "Reviewer_bqGm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes TactileVAD, a dynamics model that takes a decoder-only approach to reduce geometric aliasing. This is particularly important for tactile manipulation applications where aliasing is a significant challenge as tactile sensors can usually only make contact with a fraction of an object. When combined with learned linear dynamics and LQR control, the method is tested on a simple simulated setting of moving block control, a real-world tactile rod grasping task, and a real-world tactile cartpole task where the robot holds onto a tool handle attached to an actuated cart, and tries to keep the rod vertical. Experiments show that this approach outperforms auto-encoder and E2C based dynamics. ", "summary_of_recommendation": "I think this work is well-motivated and has convincing experimental setups. However, it’s still not entirely clear to me how the particular way the VAD model is used is addressing the aliasing problem, and the auto-decoder baseline/ablation is confusing. Thus I don’t think this version of the manuscript should be accepted, but I would be happy to raise my score to accept, if these concerns are addressed and the paper revised.\n\n**Update after rebuttal**: The authors have addressed my main concerns by expanding upon critical details about the method and by providing additional understanding for the performance of the baselines in the new Appendix B.1.1, B.1.2, and Figure B.4. Thus I am inclined to increase my score, as while I think that the experiments could still be more thorough, the idea of using an auto-decoder only model for a tactile control application is quite interesting.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- I’m a bit confused about the auto-decoder dynamics baseline – my understanding is that the tactileVAD codebook optimizes the parameters of a normal distribution for each timestep in the training dataset. If this is the case, what does the AD baseline do? Does it optimize a single z vector for each timestep in the training dataset? If so, why is the performance of AD so much worse compared to the full method? It seems that such an approach should also inherit the anti-aliasing properties of the full method, as these representations would be optimized such that they can predict dynamics accurately. Additionally, in Section 4.1 it’s stated that “ Additionally, we choose to model the latent space as conditional distributions, which leads to more structured and smooth learned latent spaces.” but this statement does not appear to be justified by any experimental results. \n- In section 4.3 “Embedding via Inference Search”, it’s stated that the inference approach offers the ability “to generate multiple candidate latent states that result in the same observation”, but it’s not discussed how this is performed – is gradient descent performed from varying initializations? Once multiple candidates are found, how are they used in the LQR controller? This is also not discussed in section 4.4.\n\n\nNitpicks:\n- The variable $N$ is used in Section 4.1 (L137) but not defined prior.\n- Figure 3 is slightly confusing – I am assuming that the left figure is showing the “full” training picture since it is being trained on multiple timesteps at the same time, while the right side shows the additional losses but on a single timestep for less clutter. But it’s not that clear that this is the case as the caption for (left) just says “trajectory generation of length T” when it seems like it is training (as backprop happens to infer $\\mu$ and $\\Sigma$ at every timestep)\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The problem is well-motivated – geometric aliasing is an important problem in tactile dexterous manipulation, and the proposed method is a promising approach to addressing this aliasing problem.\n- The experimental settings are very well designed: the simulated toy example is excellent for illustrating the problem without introducing any confounding factors, and multiple real-world experimental setups make the results more convincing.\n\nWeaknesses:\n- While an important advantage of the authors’ method is to be able to generate multiple candidate latent vectors at inference time (which was the main weakness of encoder-based methods in the first place), it’s not discussed how they actually perform this, or how it is integrated into the control system. Without this, it’s not intuitive that the method will actually improve over methods like the VAE encoder that suffer from the aliasing issue.\n- I’m confused about the results of the auto-decoder dynamics baseline which seems like it should be much more competitive with the full tactileVAD method. Please see questions for additional details.\n- As the authors themselves mention, this approach causes inference of the latent vectors to be significantly more computationally expensive than using a learned encoder. While I appreciate the authors’ transparency about this limitation, I would appreciate it if they quantified this difference so that readers could fully understand the properties of tactileVAD.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "ecACAYCWbSN", "reviewer": "Reviewer_rJSx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper uses a Variational Auto-Decoder model to learn linear latent dynamics models using tactile observation data from vision-based tactile sensors. The paper claims that unlike existing control approaches that involve learning a latent dynamics model, VADs overcome the problem of tactile geometric aliasing, ie. when different configurations or contact geometries result in the same tactile signal. The learned latent linear dynamics model is then used with an LQR controller on 1 simulated and 2 real-world tasks and compared with other alternatives for learning latent dynamics models.", "summary_of_recommendation": "While the idea in the paper seems interesting and they demonstrate their method in real robot experimental settings, comparative evaluations are poorly presented and lacking details making it difficult to conclusively believe the extent to which the presented method improves on existing methods.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the weaknesses described in the main review. In general, shedding some light on why and where the baseline methods fail, providing more details on the training of baseline methods, and adding units to the evaluation metrics would be quite useful in more convincingly demonstrating that TactileVAD is better than baseline methods.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths**\n\n- The method is described quite clearly and succinctly \n- The paper’s original claim about better VAD performance in the presence of aliasing is well-supported by the moving block simulation experiment, and further validated by real-world experiments on tactile rod grasping and tactile cartpole.\n\n**Weaknesses**\n\n- Data diversity and quantity could be better evaluated. 500 trajectories seems like a large quantity of data. \n- Control problems evaluated involve relatively easy control tasks, ie. static interaction or a 1-dimensional control task. Some analysis and insight into generalization to harder tasks would be useful\n- Implementation details of baseline methods are missing. TactileVAD uses regularization terms like the L_KL loss term. Analysis on the criticality of this term as well as details of regularization for baseline methods will help improve confidence in the comparison numbers presented in the paper.\n- Baseline methods seem significantly worse than the proposed method. Providing some intuition or explanation for this would be helpful. If not, adding more details, such as comparison of the individual loss terms would be useful and make the results more convincing. For instance, in table 2, using AD seems to do almost 10 times worse than VAD. Is this due to poor reconstruction, poor dynamics prediction, or is it just the model being underfit?\n- Often, aliasing can be avoided by adding temporal information to the observation, for instance, by stacking frames. Comparisons to such approaches would provide some insight into whether aliasing is truly the problem being addressed by the use of VAD.\n- The authors are requested to proofread the manuscript. There are a number of errors and typos, some of which are as follows:\n    - None of the tables have units, which make the numbers difficult to understand\n    - Line 246: Length of trajectories is “10”?\n    - Figure 4 and Figure 6 have the same captions\n    - There are also a number of other smaller typos", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "PlH42A-VtR", "reviewer": "Reviewer_pV8K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper provides a tactile representation and controller to disambiguate geometric aliasing, and demonstrate it on two different sensing modalities. A benchmarking setup is also proposed to evaluate the stability of the control during perturbations.", "summary_of_recommendation": "Overall the paper is clear and gives a acceptable method to imporve the geometric reconstruction and prediction during grasping. Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. There is a lack of variation in the selection of objects. Authors only provides cylindrical object for grasping and do not try other objects such as prisms, cones, spheres or irregular shapes. Is it possible to increase the types of objects?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is well presented and makes its points clealy.\n- Authors use decoder-only model giving better latent spaces compared with encoder-decoder model.\n- Two different sensing modalities, Bubbles ( point cloud) and Gelslim (RGB), are both evaluated with the proposed method, and the results are acceptable.\n\nWeaknesses:\n- There is a lack of variation in the selection of objects. Authors only provides cylindrical object for grasping and do not try other objects such as prisms, cones, spheres or irregular shapes.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method that alleviates the issue of geometric aliasing in tactile manipulation. The idea is to use the decoder only. Because the latent space can be learned so that there is no aliasing, it is possible to learn the decorder even when there is tactile aliasing issues.

The reviewers raised a common concern. First, the experiment only contains a cylinderical object. Even though the authors argued that it is sufficient to demonstrate their contribution, I do not believe so because there are different challenges associated with the object shape. A diverse set of symmetrical objects will show the limitations of this method and what has to studied further.

I believe that the video should be significantly improved. The video should be self-explanatory. Now I do not see what the experiment is about.

The strength of this paper is that the main idea is simple but has significant impacts to grasping tasks. The authors rebuttal makes it clear about this point.

Even though this paper has a clear weakness, no reviewers are against it and I am willing to accept this paper.
Decision: Accept (Poster)