Enhancing Tactile-based Reinforcement Learning for Robotic Control
Authors: Elle Miller
Trevor McInroe
David Abel
Oisin Mac Aodha
Sethu Vijayakumar
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement learning
tactile sensing
representation learning
robotic manipulation
Primary area: reinforcement_learning

Abstract: Achieving safe, reliable real-world robotic manipulation requires agents to evolve beyond vision and incorporate tactile sensing to overcome sensory deficits and reliance on idealised state information. Despite its potential, the efficacy of tactile sensing in reinforcement learning (RL) remains inconsistent. We address this by developing self-supervised learning (SSL) methodologies to more effectively harness tactile observations, focusing on a scalable setup of proprioception and sparse binary contacts. We empirically demonstrate that sparse binary tactile signals are critical for dexterity, particularly for interactions that proprioceptive control errors do not register, such as decoupled robot-object motions. Our agents achieve superhuman dexterity in complex contact tasks (ball bouncing and Baoding ball rotation). Furthermore, we find that decoupling the SSL memory from the on-policy memory can improve performance. We release the Robot Tactile Olympiad ($\texttt{RoTO}$) benchmark to standardise and promote future research in tactile-based manipulation. Project page: https://elle-miller.github.io/tactile_rl.

Enhancing Tactile-based Reinforcement Learning for Robotic Control
Elle Miller
Trevor McInroe
David Abel
Oisin Mac Aodha
Sethu Vijayakumar
University of Edinburgh
Abstract
Achieving safe, reliable real-world robotic manipulation requires agents to evolve beyond vision and incorporate tactile sensing overcome sensory deficits and reliance on idealised state information. Despite its potential, the efficacy of tactile sensing in reinforcement learning (RL) remains inconsistent. We address this by developing self-supervised learning (SSL) methodologies to more effectively harness tactile observations, focusing on a scalable setup of proprioception and sparse binary contacts. We empirically demonstrate that sparse binary tactile signals are critical for dexterity, particularly for interactions that proprioceptive control errors do not register, such as decoupled robot-object motions. Our agents achieve superhuman dexterity in complex contact tasks (ball bouncing and Baoding ball rotation). Furthermore, we find that decoupling the SSL memory from the on- policy memory can improve performance. We release the Robot Tactile Olympiad (RoTO) benchmark to standardise and promote future research in tactile-based manipulation. Project page: https://elleemiii .wwerrgtttt..b... wwt.tiiiiteee......cc
Introduction
Imagine humanoid robot gently lifting an elderly person out of their bed with the same delicacy as human caregiver, person with severe motor impairments using re-enabling robotic arm to brush their teeth. These scenarios represent critical pathways toward enhancing human independence, dignity, and quality of life. For this future to safely materialise, robots must evolve beyond merely seeing their environment; they must possess some capacity to feel it.
Due to the scalability challenges associated with collecting dexterous demonstrations with tactile data, reinforcement learning (RL) is primary candidate for enabling tactile-based manipulation. While RL has facilitated breakthroughs in locomotion, yielding robust agents that can traverse complex terrains [49], manipulation lags significantly behind. This deficit stems from multiple challenges, including inaccuracies in contact simulation, the need for complex reward functions, and critically, heavy reliance on idealised state information (e.g., prior work required 19 cameras for object and fingertip localisation [5). In this work, we tackle this reliance by suggesting that tactile data can effectively replace idealised state information. Given that manipulation is fundamentally defined by control through selective contact (37), we argue that tactile sensing provides sufficiently rich information stream to localise and control objects, paving the way for more scalable robotie systems.
Despite a decade of focused interest in tactile-based RL 33 breakthrough manipulation capabilities have yet fully materialise. We posit that progress this domain remains stymied by two critical, interrelated challenges. First, a lack of hardware and data convergence (sensor type, placement strategy, information representation) has hindered systematic progress. Second, the published literature provides conflicting evidence regarding the utility of tactile feedback in RL. Some studies report marginal performance gains when incorporating tactile information to the observations 40 41) while others do not [24
61
68 the hoped-for
For instance, [45] 54]
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The Rooot Tacti le clypplad (RoTO)
shatiz hand sensing
Baoding
Foog
propp
prop
π
Xpt....
obaesvation
abservaticn represencation
binery contacts
og
V
V(o.)
(0. Mossan
history
troined wita HL selE-auperviiion
trainec with
Figure I: Tactile-based RL with self-supervision. Our agents achieve superhuman dexterity in our RoTO benchmark using only history of proprioception and binary contacts (i.e., with vision privileged information). By jointly training the observation encoder with self-supervision, the agent transforms complex sensory input into representations that capture object positions and velocities.
achieve impressive in-hand manipulation using proprioceptive history alone, with 46 claiming that binary contacts are already implicitly contained in proprioceptive history.
We hypothesise that the reported performance discrepancies in tactile reinforcement learning arise from the unique data characteristics that tactile feedback presents to deep RL. Tactile measurements often sparse (only present during contact) and non-smooth, potentially leading to unbalanced datasets and learning instabilities. Consequently. a deep RL agent may struggle to extract a useful representation from this data, causing agents converge prematurely suboptimal policies reliant on continuous proprioceptive signals. To alleviate the significant burden simultaneously learning observation representation, policy, and value function, we propose using Self-Supervised Learning (SSL) to assist with learning the observation representation (25). While existing tactile SSL objectives have shown some utility (e.g., pixel augmentation [19 24 masked reconstruction (53)) we posit that they often fail to encourage the encoding of critical features from state transitions, such as object velocity or friction, that are essential for more complex control.
Therefore, the central objective of this work is to develop general-purge purposs SSL methodologies that effectively leverage tactile observations across diverse robotic control tasks in RL. To harness the power of tactile sensing and avoid interference from complementary modalities, we deliberately do not study pixel depth observations. Furthermore, maximise future utility and real-world scalability, we focus on the simplest, lowest-cost tactile setup: sparse binary contacts (e.g.. Figure []] which conveniently avoids the sim-to-real challenges associated with continuous measurements 71. Finally, guided by the intuition that robust blind manipulation should be possible, we forgo the traditional reliance on teacher-student networks and instead focus on learning powerful representations directly from the sensory information using self-supervision. Our approach proves highly effective, demonstrating that direct sensory learning provides the necessary control capacity without ever necessitating idealised state information. Our main contributions are as follows:
We provide empirical evidence that sparse binary tactile signals critical for dexterity, showing they can significantly improve performance beyond what is achievable with proprioceptive history alone. We propose that explicit tactile information essential only where proprioceptive joint control errors fail to reliably register all environment dynamics, e.g., decoupled object-robot dynamics, low-inertia. objects, contact spatial ambiguity, and multi-contact resolution.
We demonstrate new level simulated superhuman dexterity in complex contact tasks using limited sensory setup of only proprioception and 17 binary contacts.

---

We propuse and analyse four SSL objectives for tactile agents (tactile reconstruction. full recon- struction, forward dynamics, and tactile forward dynamics), demonstrating superior performance to RI--only approaches. We find that forward dynamics is the most effective objective, creating representations that encode object positions and velocities.
We provide empirical evidence that decoupling the SSL training data from the on-policy mem- ory can improve performance, demonstrating possibilities for leveraging off off-policy experience.
We introduce the Robot Tactile 0lympiad (RoTO) benchmark, comprising three challeng- ing Isaac Lab environments (Find, Bounce, and Baoding), with tuned baselines and integrated hyperparameter optimisation to standardise and inspire future research in tactile manipulation.
2 Related work
In-hand manipulation with reinforcement learning (RL), Most prior work on in- in-hand object
manipulation relies on either privileged information with teacher-student networks [45] 46 [9].
visual input (RGB-D) 8 46 731, or explicit object pose estimators [23 43 57 67]. For Baoding
ball rotation, 43 tracking cameras for ball positions, and 73 use pointclouds with privileged
information distillation. To our knowledge, the most advanced dexterous 'blind' RL demos are object rotation (54 71 and half-rotation of Baoding balls (67] In terms of speed, the fastest robot policies in both simulation and the real world achieve a maximum of 3 complete rotations in 10 seconds [73 In contrast, our blind simulated agents achieve up 25 rotations over the same time interval.
Pixel representation learning in RL. RL agents often struggle to learn effective representations from high nigh-dimensional inputs like pixels. While early pixel-based RL methods did not specifically target this problem, most contemporary work leverages auxiliary objectives to guide representation learning 225 Reconstruction is common objective, explored in both model-free 288 29 70) and
model-based paradigms (20 21 22 32 63 66 However, achieving pixel-perfect reconstruction can
retain information irrelevant to control (55). Consequently, many modern auxiliary objectives operate
in the latent space, including forward dynamics I6 39 [1], disentanglement [11 13), contrastive
approaches 31 (56). and iofommation-thoreetee objectives 34 38). In this work, we examine how
these techniques can be applied to the proprioceptive and tactile robotic sensory modalities.
Multimodal representation learning in RL. Recent works have aimed to produce generalisable auxiliary objectives that operate over all modalities. 77) propose maximising mutual information (MI) between unimodal and multimodal representations to align the latent spaces. Noting this does not filter irrelevant information, 772 suggest using an information bottleneck [59] that maximises forward dynamics information. 6 propose that self- supervied objectives should be modality seeeliit---peiiii,,, e.g.. reconstruction for proprioception and contrastive losses for images. We do not propose a generalisable multimodal auxiliary objective, but auxiliary objectives to better leverage the tactile modality.
Tactile representation learning in RL. Few works leverage self- supe-supervieed objectives to improve nettlle-bsedd RL. Early work used variational autoencoder with forward dynamics objective for stabilisation task [60]. More recent approaches have explored pixel-based objectives like masked autoencoding [53] and augmentation [18 24], or contrastive learning to maximise similarity between unimodal visual and tactile representations 10] Other tactile-specific objectives include predicting
the presence of contacts ( (3). Outside of mode-free RL, tactile dynamics models have been learned for estimating object features 664] and model predictive control (MPC) [2] [58.. Similar to these works and [60], we identify dynamics a promising objective for tactile information and are the first to study multi-step forward dynamics with tactile data in model-free RL, focusing on binary contacts.
Tactile-tailored RI. Given that tactile interactions are both important and sparse, some research focuses on modifying the RL setup. Prior work has proposed increasing the sampling probability of contact-rich episodes in off-policy algorithms [62] only updating the tactile encoder under contact 24]. To the best of our knowledge, we are the first in tactile-based on-policy RL to modify the dataset used for the self-supervised objective, deviating from the typical on-policy rollout.
Analysing representations in RL. Despite extensive research into representation learning in RL, relatively little work focuses on understanding the properties of the learned representations themselves The most common analysis technique is linear probing, used to assess how well the latent space encodes environment features 4 47 74. For our setting, we evaluate the representation by its ability to encode the true underlying environment state (36], quantifying this relationship using mutual information estimation [15].
15 65
3

---

tactile recons truction (IR)
full zeconstruction (FR)
d
O
BCE ( it luck)
Cya = LYn (S (
Rg--11
R
dynamics (ED)
L MSE(((()tt)) #(tk)
02
Zt
2++
Mc-2
CTD 1
C211
tactile forward dynamics (TED)
Oc+1
eT
d
LrD Lypt
2 B tatt
Figure 2: Proposed self-supervised objectives. Reconstruction losses (TR, FR) force the encoder to preserve tactile information within Zt by using binary classification decoder to reconstruct the original of aact, Forward dynamics losses (FD, TFD) ensures the encoder extracts the necessary information from for accurate forecasting of the representation 2, several timesteps into the future.
3 Method
3.1 Problem setting
We study a specialised Partially-Obae Observable Markov Decision Process (POMDP) [26] parameterised by the tplle (8, O, F,A, R) R.
The state space s S, action space E A. transition kernel T: S x Ax S > [0, 1]. reward function R: S x - R, and discount factor E (0, 1) are standard. The state is unobservable; instead, the agent receives composite observation defined as k-timestep history of multi-modal readings. The observation function F S + O is deterministic, with partial observability arising from perceptual aliasing (i.e., where distinct states $1 * 82 can produce the same observation F(s) F(s2) due to sensing limitations). The agent interacts via policy क : O ((A) that parameterises distribution over actions, seeking the optimal policy
O, which is
arg max, [> 1 R(st, at)].
We solve our POMDP using RI., implemented as follows (Figure TT. The agent receives the observa- tion Og, which is concatenation of the k-length history for proprioceptive (o.rop; joint angles 0, joint velocities Ó, last action at and if applicable, the end-effector pose XEE, QεE) and tactile (ofact, bi- nary contacts b E {0, 1] Nuneer) modalities. The agent is comprised of three MLPs: an observation en- coder policy n, and value function The observation encoder (o, - 1024 512 256 2+) learns a compact representation Zt, which π and v are conditioned on (π 2, - 128 - 64 + a Zt + 128 + 64 + V(o))). We train using the clip variant of Proximal Policy Optimisation (PPO) [50]. augmented with a self-supervised auxiliary loss Laux for representation learning. The total loss C is minimised via backpropagation:
(ppo Oz 0.) = - CLIP 02) Lv 0,) + Centoons (0e, 0x),
(1)
L (e. 0.) + caux Caux (0e, 0 Oawx).
The PPO update (optimising e, π, and v) is performed using separate optimisers with a shared learning rate and gradient clipping 1.0. The auxiliary loss uses a separate optimiser (optimising and auxiliary networks) with a different learning rate Iraux and is applied without gradient clipping.

---

3.2 Self-supervised objectives
The self-supervised loss Lan is computed using auxiliary networks (mxx) that are discarded post- training. We propose four distinct auxiliary learning objectives, Laux {LTR, LTR, LTD, LTTD J. described below, which are used to train e and the auxiliary networks (architectures in Appendix F.)
Tactile Reconstruction (TR). To mitigate the observed tendency of gradient-based optimisation to prematurely favor features from the stable proprioceptive history, thereby neglecting the complex tactile input, we introduce reconstruction objective to exclusively decode the tactile observation of from the learned multimodal representation Z, with decoder (Figure 2 top). Since the input binary classification problem using a Binary Cross- Ctoss-Entropy (BCE)
is binary, TR is implemented loss on the logits. Given the inherent sparsity of binary contact data, we employ positive weighting P. 10 to penalise false negatives (missed contacts) more heavily. The objective minimises the difference between the predicted and actual observed tactile distribution:
LTR B .o, .oact) -P. o.act log (6tact) + (1 (1-o qaet)) log (1 -1-00 6 )).
(3)
Full Reconstruction (FR), For comparison, we implement a standard Full Reconstruction (FR) baseline, which attempts decode both the tactile and proprioceptive observations simultaneously from zt- FR minimises the sum of the TR loss and a Mean Squared Error (MSE) loss for the continuous proprioceptive observation of oprop.
LFR = LTR + MSE (6 orroP).
Forward Dynamies (FD). Next, we propose multi-step forward dynamics an promising alternate objective to distill the observations into predictive representations. This is achieved by encouraging the encoder to extract information from the current observation og that is essential for accurately forecasting the observation representation Zt several timesteps into the future. A sequence of length n (Ot, a....., . Ot+n-1 at+n-1) is sampled from a memory, ensuring no episode-terminating transitions are included. Given the current latent state Zt and action a, a forward model f predicts the next latent state, - f f(z,,a) This prediction then used autoregressively to forecast the entire sequence (Figure 2 bottom). To provide a stable learning signal, the prediction is compared against target latent state z (T(+)))) where er is target encoder maintained as an exponential moving average (EMA) of the actual encoder e. To decouple the dynamics learning from the prediction task, the prediction loss calculated between a nonlinear projection of the predicted latent state p(2 (2+-1) and the target latent state 2 2T similar to [17
LTD Σ MSE 2(411s),2++1)
Tactile Forward Dynamics (TFD). To ensure the learned dynamics latent space also models tactile dynamics, we introduce a novel objective. This combined loss optimises the encoder to predict future latent states (LFD) from which future tactile observations can be reconstructed (via the TR loss Lrn). Specifically, the predicted latent states ž i are decoded back into the next-step tactile space atact and the resulting loss is added to the dynamics term:
LTD = LFD +
BCE
tact tact
0c+1
3.3 Separated auxiliary memory
When training on-policy RL with an auxiliary objective, the default is to jointly optimise these objectives on the same rollout data, and then permanently discard this data (e.g.. [I]] 30 48) We observe that the abrupt changes training data cause small spikes in the auxiliary loss throughout training (Appendix B). We propose to separate out and increase the size of the auxiliary memory in multiples of Nrottouts to improve performance through (i) stabilising the auxiliary updates and (ii) optimising on wider distribution of data. For agent using B using parallel environments and a rollout length R, the on-policy RL. memory has a shape [B, R, ...]. Our method simply involves storing data for the auxiliary tasks in a larger, separate buffer of size [Nroloous:: B, R, .....
5

---

4 Experimental setup
RL. We use customised implementation of Proximal Policy Optimisation (PPO) 500 from SKRL [2 to incorporate observation stacking, self- self-supervision. and separated environments for continuous evaluation. We using 4096 parallelised environments for training and 100 for evaluation.
Hyperparameters. To account for the fundamental changes introduced by the self-supervision on the state representation, we conducted an individual hyperparameter sweep for every environment and method combination, aligning with best practices for RL research [14] Fach sweep comprised 20 trials using the TPE sampler with 5 startup trials. The swept hyperparameters included PPO hyperparameters (learning rate ir, rollout length, number of minibatches, number of learning epochs, entropy loss scale Cent), self-supervision hyperparameters (learning rate Iraus, loss weight Cauz), and for forward dynamics objectives, the sequence length n.. All sweep information is in Appendix G
Compute. Experiments were executed on a GPU cluster (8x NVIDIA RTX A4500s). The simulation environment (Isaac Lab) required 16GB VRAM, 32GB RAM, and 8 CPU cores. The approximate time for a single experiment, including the hyperparameter sweep (~> 50 hours) and the final five seeds (~ 5 x 2 hrs) totaled 60 hours. With seven experiments across three tasks, the cumulative paper compute time is 7 x 3 x 60 =
1,260 hours.
Environments. We evaluate our method on three custom robotic manipulation tasks implemented within Isaac Lab [42 (Figure[]) They were designed to collectively cover a wide range of tactile inter- action patterns (i.e., sparse, intermittent, and sustained), to comprehensively assess the generalisability of our proposed methods. We release our environments and baselines as an open-source benchmark, the Robot Tactile Olympiad (RoTO) to foster progress in tactile-based manipulation.
Find: The agent must locate a fixed sphere within 20 em x 20 cm (300 timesteps).
Bounce: The agent must bounce ball as many times as possible in 10 seconds (600 timesteps).
bounce defined as a contact event after period of at least timesteps (~ 83ms) without contact. The ball is modelled off typical office stress ball (70mm diameter, 30g). The theoretical maximum is 100 bounces (given the 5-timestep separation), while the human Guinness World Record corresponds to 59 bounces in 10 seconds (Appendix C).
The agent must rotate two balls (1.5 inch diameter, 55g) around each other in-hand as many times as possible within 10 seconds (600 timesteps). This task is inspired by the historical Chinese practice used for dexterity and mindfulness. For context, the fastest human demonstration we found online achieves 13 rotations in 10 seconds (Appendix
Baoding:
The physics simulation operates at 120 Hz, while the control policy executes at 60 Hz. All robots are joint position controlled. We implement Find using the Franka robot with a E R attaching Nsensors = 2 tactile sensors onto the fingers, with observation history length h = 16. We implement Bounce and Baoding using the Shadow Hand with a, E R specifying each link as a contact sensor N Neensors 17 (Figure D with an observation history length k = 4. See Appendix E for full POMDP details including reward functions.
Experimental results
Figures 5
depict the mean evaluation return across 5 seeds, with 1 standard deviation shaded.
RL-only. To establish a baseline, we evaluated three RL-only agents: proprioceptive-tat tactiee agent, a proprioceetive-only agent, and a proprioceptive-only variant which excludes the last action at-1 (Figure 3). In Find, the proprioceptive--actis gains, achieving equivalent final performance to the proprioceptive agent. The success of the proprioceptive agent critically relied on the last action, which enabled implicit object contact inference via control error, confirmed by policy inspection revealing a strategy to maximise collision probability. Conversely, in Bounce, adding tactile information yielded higher sample efficiency and superior returns, although the proprioceptive agent still attained high returns by developing a degenerate, state-agnostic bouncing motion with an outstretched hand. The Baoding task demonstrated the greatest utility for tactile information, driving the agent from complete failure to functional success, albeit with high variance. These -eskddepnddnnt outcomes collectively suggest that tactile sensing only confers utility in specific scenarios, a theory we further develop in Section 6
agent offered only marginal sample efficiency

---

Bourcce
Baoding
270
820
2.0
420
300
220
50
1:00
02
Figure 3: RL-only. Mean evaluation returns of proprioceptive-taciie (o ① o tact) and proprio- ceptive (oProp) agents. We run one seed with the last action a removed from the oroop agent.
Bounce
Gaoding
400
230
730
Noan wonnoelltteeeeee
2.00
520
900
250
2.60
420
144
PrOlsrup locaid
320
+Fill Receremactio
luskin Rasnnraaaiionn (11)
100
Forward Dynerice
Ferword Bynamics
25
50
100
250
1/5
25
110
3.23
100
1/2
30
100
142
150
15
Inessopo
Figure 4: RLASSL. Mean evaluation returns of the self-supervised agents.
RL+SSL. We evaluate our four proposed SSL objectives: Tactile Reconstruction (TR). Full Recon- struction (FR), Forward Dynamics (FD), and Tactile Forward Dynamics (TFD) (Section 3.2) As evident in Figure agents trained with TR and FD consistently outperform RL-only baselines across all environments. Between these two leading methods, the FD agent yielded higher mean returns in Find and Bounce. Conversely, in Baoding, the TR agent achieved higher mean return due to tighter performance distribution, despite the FD agent reaching a higher performance upper bound. The performance the remaining methods (FR and TFD) was sensitive the specific environment, showing no consistent trend. For instance, TFD was the best objective in Find and the worst objective in Bounce. Further analysis provided in Appendices AD and H
RL+soy+M We investigated the impact of a separated auxiliary memory by applying it to the FD agent, selected for its superior general performance. To account for the widening data distribution, we conducted a targeted re-sweep over just the SSL. hyperparameters and Nrollouts E 2, 3, (limited due computational resources). As illustrated in Figure the addition of the separated memory had only minimal effect on returns Find and Bounce, but led to a substantial improvement in Baoding. We hypothesise that this result stems from the inherent complexity of the Baoding task, which requires the agent to observe and act upon a longer temporal horizon of dynamics. A two-ball rotation inherently spans more timesteps than a bounce contact event, potentially making the longer memory sequences more beneficial for learning.
Riid
Bourcce
Saoding
200
730
430
1.00
530
3.0
620
200
3.40
530
220
#Fomard Cynemica
125
1.0
175
150
Timeetrpe.
Figure 5: RL+SSL+Memory, Mean evaluation returns of the FD agent trained on the on-policy
rollout vs Nrottouts last rollouts. The effect is minimal in Find and Bounce but substantial in Baoding.
7

---

Discussion
Q1: Do binary contacts offer benefits beyond proprioceptive history?
Our results indicate that that binary contacts do offer benefits beyond proprioceptive histories, but the degree of utility task- desendennn.nt.. We hypothesise that explicit tactile information remains critically useful, but only in scenarios where proprioceptive joint control errors fail to reliably register all environment dynamics. For instance:
1. Decoupled object-robot dynamics: When object's motion has component orthogonal to the articulation direction, resulting in contact with negligible change in joint control error. For instance, in the Baoding task, the balls primarily move horizontally around the hand's plane, decoupled from the joint motion used for manipulation.
2. Low-inertia objects: Very light, deformable objects (e.g., the 30g ball in Bounce, sheet of paper, or a sponge) do not exert a sufficiently large reactive force upon contact. Consequently, the robot's proprioceptive sensors fail to register significant signal, necessitating reliance explicit tactile sensing for reliable detection.
3. Contact spatial ambiguity: When the policy requires knowing the specific location the contact along a single rigid link. The joint control eгror only provides the net force contribution to the motor, making it difficult ascertain the distance of the force from the joint.
4. Multi-contact resolution: When the policy needs to differentiate the source of the total force contribution (i.e., whether it resulted from one strong contact multiple simultaneous, weaker contacts). The control error only provides the total magnitude of the net force, blurring the distinction between multiple, -raigraiedd contact events.
Q2: Why exactly the self-supervision helping?
We hypothesise that self-supervised learning helps by enforcing the compression of task-critical information into the learned representation Z,. We test this hypothesis by measuring the mutual information T(2;; 8,) between the reduced latent representation 2 and a vector of ground-truth state variables, St- We utilise the Kraskov-Stoee Sttotegaauer-rrrsseeeeeeeee approach for continuous variables as outlined in [1]. For all agents, we collect 5,000 samples of (Zt, St) pairs. The raw 2, is 256-dimensional vector, which we reduce to D 13 components via Principal Component Analysis (PCA) to counteract the bias of the KSG estimator in high dimensions. We use K- nearest neighbors for estimation. For Bounce, St the ball position, quaternion, linear velocity, angular velocity, and number of timesteps without contact. For Baoding, St is the balls' position, linear velocity, and relative distance. See Figure o results.
(KSG) estimator (27), following the
In Bounce, the base PPO agent achieves the highest I(z,; st), primarily because its successful policy results in a highly repetitive, low-entropy motion (trapping the ball), which artificially inflates the MI. The reconstruction agents had zero MI, while both dynamics agents registered non-zero MI. In Baoding, the distribution of MI across agents
closely mirrors the performance spread, with the FD agent capturing nearly triple the MI of the PPO agent. We also perform marginal MI analysis on individual state elements [(ze; St.j) understand the specific encoding strategy used by each objective. For the Baoding environment, the FD agent was the only model to achieve non-zero marginals, successfully encoding the ball positions order of priority: x (parallel the hand), (perpendicular to the hand), and z. In the Bounce environment, all agents recovered the number of timesteps without contact. While the tactile agents (TR and TFD) further recovered the ball vertical velocity, the FD agent proved the most effective, uniquely encoding both the ball vertical velocity and positional information (ar and coordinates).
Bounce
Baoding
1.0
PROj
-Fi
(2::5)
0.6
-TR
1.0
-FD
-TFD
0.5
0.2
Figure 6: Mutual information estimation. We estimate T(z;; 8 between the reduced observation representation z₁ and the ground-truth state to evaluate the representation quality.
8

---

Q3: Is self-supervision strengthening the proprioceptive, tactile, or combined representation?
The influence of self-supervision sensory representations differs significantly between the recon- struction and dynamics objectives, with its effect being highly dependent on the environment.
Reconstruction. The relative performance between FR and TR is different in all environments (Figure In Find, FR outperforms TR, suggesting more benefit from compressing proprioceptive history since the agent Bounce, FR and TR performed almost identically, implying the primary gain came from compressing the tactile signal. Most compellingly, in Baoding, TR far outperformed FR, demonstrating that combining proprioceptive reconstruction with tactile reconstruction resulted negative interference. Given that only TR achieved zero failure runs, this suggests that agent failures directly linked to an inability to robustly represent critical tactile information when its objective is combined with proprioception.
more reliant the control errors for object detection. Conversely, in
Dynamics. The effect of the dynamies prediction is less conclusive, as the comparison between learning forward dynamics the combined representation (FD), summed with tactile reconstruction loss on the predicted latent state (TFD). TFD showed a minor benefit Find, hinting at a useful combination of losses. However, in the more dynamic tasks of Bounce and Baoding, TFD performed worse than FD. This result suggests that explicitly forcing separate tactile reconstruction loss is either redundant or counterproductive. Two primary explanations exist: 1) the FD objective the combined representation is robust enough to implicitly capture the necessary tactile information within its learned state representation; or adding explicit reconstruction loss introduces undesirable training conflict that compromises the overall predictive quality. Isolating the precise mechanism by which the tactile signal influences dynamics prediction will require further investigation, ideally utilising distinct tactile-only forward model.
Q4: How do the results correspond with physical metries?
The performance using physical metries shown in Figure 7 with the bold bars depicting the mean across seeds, and the shaded bars depicting the maximum across seeds. Overall, the performance improvements do correspond with meaningful changes in physical behaviours. For example, our best self-superriied agents on average find an object 36% faster (FD, 1.4 vs 1.9 seconds), bounce a ball 10 more times 10 seconds (FD, 79 vs 69), and complete 17 Baoding rotations compared seconds (FD with auxiliary memory). We note that the best Bounce agent achieves 88 bounces in 10 seconds, beating the (human) Guinness World Record of ~ 59. Similarly, the best RL-only Baoding agent achieves 13 rotations in 10 seconds, on par with skilled humans, while the best self-supervised agent achieves 25, We fully acknowledge these results are not likely to transfer directly as-is into the realdworld, but still include point of interest. See Appendix C for more dicussion of human performance.
in 10
Remaining time found 1s]
of bounces
Number Baoding rotations
3.8
85
3.6
20
PPO
3.4
75
15
3.2
66
+DD
+TFD
3.0
60
+FD++N
50
Figure 7: Physical metrics. Maximum (shaded) and mean (bold) across seeds.
Q5: How well can a forward model learn the dynamics of tactile interactions?
Very well. See Appendix D for plots of true positive rate, false negative rate, precision, and recall for up to 10 timesteps into the future for Bounce and Baoding. There we also also provide spatio- temporal visualisation of predicted vs actual next tactile states (Figures A6 A7). From analysing these figures, we observe there is a slight tendency to overpredict rather than underpredict contacts, likely due to the high positive weighting we apply in the BCE loss (pe 10). Despite having access to history of tactile states the current observation, some of these states are not always perfectly "copied" over. Interestingly, from the no-contact observation 07 in Bounce, the decoder correctly

---

anticipates contact in the next state (albeit in the wrong locations). This result aligns with the findings in Q2, where we show the representation encoding the ball 2 position and velocity component, which is sufficient to predict future contacts from. Also in Baoding, the some of the false positive predictions in states 010, 011 are just one-step too early.
Q6: Can on-policy agents benefit from learning representations on off- off-policy data?
The results in Figure Saareepproooo provide evidence for yes, with the effect ranging from minimal (Find, Bounce) to substantial (Baoding). We think the impact was pronounced on Baoding because the rotation task requiring reasoning over a longer temporal horizon. These findings highlight a promising research direction for incorporating off- off-policy data into on-policy training pipelines, potentially bridging the gap between the strengths of both paradigms.
Q7: How do our findings translate to practical recommendations?
We compress our findings into two recommendations. Our work has demonstrated that "blind" robotic agents trained jointly with self-supervision outperform RL-only agents across diverse range of control tasks. Thus, our first recommendation is to train tactile-based RL agents jointly with tactile reconstruction forward dynamics with a separated auxiliary memory, if you are working in similar setting and would like to get higher (and potentially more reliable) distribution of returns. Second, while we acknowledge that increased sensory information is theoretically advantageous, incurs substantial computational costs as well as being statistically harder i.c., the space of functions over 2 Mansns is much larger than {0, 1) Naauues In addition, the bandwidth of pixel-based signals directly limits the number of parallel environments that can be executed in Isaac Lab and other simulators. Since our work has revealed unexpected efficacy using binary tactile observations, we recommend initially implementing simpler tactile information formats (e.g., binary, continuous, or contact pose), and switching to pixel-based tactile representations only if needed.
Limitations
The primary limitation of this work is the absence of validation on physical robot hardware. However, we have minimised the sim-to-real gap by focusing on sparse binary contact signals, which avoids major complexities associated with continuous sensor noise. Crucially, our contributions optimisation strategies designed to yield superior and distinct policies. The benefits of these strategies should therefore be transferable, provided the final sim-to-real gap for the RL-only agent can be bridged. We also note that training self-supervised agents increases computation time compared to RL-only. The effect less noticeable for reconstruction. but becomes more dramatic the higher the value of sequence length in forward dynamics. In addition, a limitation of using separated auxiliary memory is that more memory is required. Regarding generalisation, we would expect see similar results if our approach is applied to other environment domains (e.g., locomotion).
Broader impacts. Our work advances the capabilities of autonomous sensory driven robotic agents in simulated environments. While real-world deployment down the line could positively impact healthcare and assisted living, we acknowledge potential negative consequences including workforce displacement and unintended applications, necessitating responsible development and appropriate regulatory oversight.
10

---

Acknowledgments. EM is supported by Ph.D. studentship from the UK Engineering and Physical Sciences Research Council (EPSRC) via the Robotics and Autonomous Systems CDT. We thank Dr. Stefano V. Albrecht for providing formative guidance and initial conceptualisation of this research.
References
[1] Rishabh Agarwal, Marlos C. Machado, Pablo Samuel Castro, and Mare G. Bellemare. Con- trastive behavioral similarity embeddings for generalization in reinforcement learning, In ICLR, 2021.
[2] Bo Ai, Stephen Tian, Haochen Shi, Yixuan Wang, Cheston Tan, Yunzhu Li, and Jiajun Wu. Robopack: Learning lactile-informed dynamics models for dense packing. Robotics: Science and Systems (RSS), 2024.
[3] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
[4] Ankesh Anand, Evan Racah, Sherjil Ozair. Yoshua Bengio, Alexanle Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. In NeurlPS, 2019.
[5] OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in- in-hand manipulation. The International Journal of Robotics Research, 39(1), January 2020.
[6] Philipp Becker, Sebastian Mossburger, Fabian Otto, and Gerhard Neumann. Combining reconstruction and contrastive methods for multimodal representations in rl. In Reinforcement Learning Conference (RLC), 2024.
[7] Kaiqi Chen, Yong Lee, and Harold Soh. Multi-modal mutual information (mummi) training for robust self-supervised deep reinforcement learning. In 202/ IEEE International Conference Robotics and Automation (ICRA), pages
2021.
[8] Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84):eadc9244, 2023.
[9] Yizhou Chen, Andrea Sipos, Mark Van der Merwe, and Nima Fazeli. Visuo-Tactile Transformers for Manipulation. In Conference on Robot Learning ( (CoRL), 2022.
[10] Vedant Dave, Fotios Lygerakis, and Elmar Rueckert. Multimodal visual-tactile representa- tion learning through self-supervised contrastive pre-training. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages
2024.
[11] Mhairi Dunion and Stefano V. Albrecht. Multi-view disentanglement for reinforcement learning with multiple cameras. In Ist Reinforcement Learning Conference, 2024.
[12] Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah Hanna, and Stefano Albrecht. Conditional mutual information for disentangled representations in reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[13] Mhairi Dunion, Trevor Mclnroe, Kevin Sebastian Luck, Josiah Hanna, and Stefano V. Albrecht. Temporal disentanglement of representations for improved generalisation in reinforcement learning. In International Conference Learning Representations (ICLR), 2023.
[14] Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how to tune them. In ICML, 2023.
[15] Samuel Garcin, Trevor McInroe, Pablo Samuel Castro, Prakash Panangaden, Christopher G. Lucas, David Abel, and Stefano V. Albrecht. Studying the interplay between the actor and critic representations in reinforcement learning. Internation Conference on Learning Representation (ICLR), 2025.
11

---

[16] Carles Gelada. Saurabh Kumar, Jacob Buckman, Ofrr Nachum, and Mare G. Bellemare. Deep- mdp: Learning continuous latent space models for representation learning. In ICML, 2019.
[17]
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukeuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020.
[18] Irmak Guzey, Yinlong Dai, Ben Evans, Soumith Chintala, and Lerrel Pinto. See to touch: Learning tactile dexterity through visual incentives. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 13825 -3832, 2024.
[19] Irmak Guzey, Ben Evans, Soumith Chintala, and Lerrel Pinto. Dexterity from touch: Self- supervised pre-training of tactile representations with robotic play. In Conference on Robot Learning (CoRL), 2023.
[20] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2020.
[21] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
[22] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, 2025.
[23] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu, Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Jean-France Francois Lafleche, Dieter Fox, and Gavriel State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In ICRA, 2022.
[24] Johanna Hansen, Francois Hogan, Dmitriy Rivkin, David Meger, Michael Jenkin, and Gregory Dudek. Visuotactile- Learning Multimodal Manipulation Policies with Deep Reinforcement Learning. In 2022 International Conference on Robotics and Automation (ICRA), pages 8298- 8304, Philadelphia, PA, USA, May 2022. IEEE
[25] Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In ICLR, 2017.
[26] Leslie Pack Kaelbling, Michael I. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
[27] Alexander Kraskov, Harald Stöghauer, and Peter Grassherger. Estimating mutual information. Phys. Rev. F, 69:066138, Jun 2004.
[28] Sascha Lange and Martin Riedmiller. Deep auto-encoder neural networks in reinforcement learning. In International Joint Conference on Neural Networks (IJCNN), 2010.
[29] Sascha Lange, Martin Riedmiller, and Arne Voigtländer. Autonomous reinforcement learning raw visual input data in a real world application. In International Joint Conference Neural Networks (IJCNN) 2012.
[30] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In NeurIPS, 2020.
[31] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages (639-5650 PMLR, 2020.
[32] Alex X. Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. Advances in Neural Information Processing Systems, 2020.
12

---

[33] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. Science Robotics, r(b7))abb October 2020. arXiv:2010. 11251 [cs, eess].
[34] Kuang-Huei Lee, Ian Fischer, Anthony Liu, Yijie Guo, Honglak Lee, John Canny, and Sergio Guadarrama. Predictive information accelerates learning rl. In NeurIPS, 2020.
[35] Michelle A. Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rioe Tasks. IEEE Transactions on Robotics, 36(3):582- 596, June 2020.
[36] Timothée Lesort, Natalia Diaz-Rodríguez, Jean-François Goudou, and David Filliat. State representation learning for control: An overview. Neural Networks, 108:379-392 -92,, December 2018.
[37] Matthew T. Mason. Toward Robotic Manipulation. Annual Review of Control, Robotics, and Autonomous Systems, (1), 2018.
[38] Trevor McInroe, Adam Jelley, Stefano V. Albrecht, and Amos Storkey. Planning to go out-of- distribution in offline-to-online reinforcement learning. In RLC, 2024.
[39] Trevor McInroe, Lukas Schäfer, and Stefano Albrecht. Multi-horizon representations with hierarchical forward models for reinforcement learning. Transactions Machine Learning Research (TMLR), 2024.
[40] Andrew Melnik, Luca Lach, Matthias Plappert, Timo Korthals, Robert Haschke, and Helge Ritter. Using Tactile Sensing to Improve the Sample Efficiency and Performance of Deep Deterministic Policy Gradients for Simulated In-Hand Manipulation Tasks. Frontiers Robotics and AI, 8, 2021.
[41] Hamza Merzić, Miroslav Bogdanović, Daniel Kappler, Ludovic Righetti, and Jeannette Bohg. Leveraging Contact Forces for Learning to Grasp. In 2019 International Conference on Robotics and Automation (ICRA), pages 3615-3621, May 2019. ISSN: 2577-087X.
[42] Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8(6):3740- -747,,7777077 2023.
[43] Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep Dynamics Models for Learning Dexterous Manipulation. In Proceedings of the Conference on Robot Learning. pages 1111-112. PMLR, May 2020. ISSN: 2640-3498.
[44] Pegah Ojaghi, Romina Mir, Ali Marjaninejad, Andrew Erwin, Michael Wehner, and Francisco J. Valcro-Cuevas. Curriculum is more influential than haptic feedback when learning object manipulation. Science Advances, 11(14), 2025.
[45] Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-Hand Object Rotation via Rapid Motor Adaptation. In Conference on Robot Learning (CoRL), 2022.
[46] Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Ma, Roberto Calandra, and Jitendra Malik. General In-hand Object Rotation with Vision and Touch. In Proceedings of The 7th Conference on Robot Learning, pages 2549-2564. PMLR, December 2023. ISSN: 2640-3498.
[47] Evan Racah and Christopher Pal. Supervise thyself: Examining self-supervised representations in interactive environments. arXiv preprint: arXiv:1906.1 1957, 2019.
[48] Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforce- ment learning. In ICML, 2021.
[49] Nikita Rudin, Junzhe He, Joshua Aurand, and Marco Hutter. Parkour in the wild: Learning general and extensible agile locomotion policy using multi-expert distillation and fine-tuning, 2025.
13

---

[50] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:107.00647, 2017.
[51] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- self-predictive representations. In ICLR, 2021.
[52] Antonio Serrano-Muñoz, Dimitrios Chrysostomou, Simon Bøgh, and Nestor Arana- Arexolaleiba. skrl: Modular and flexible library for reinforcement learning. Journal of Machine Leurning Research, 24(254):1-9, 2023.
[53] Carmelo Sferrazza, Younggyo Seo, Hao Liu, Youngwoon Lee, and Pieter Abbeel. The power of the senses: Generalizable manipulation from vision and touch through masked multimodal learning. In 2024 IEEE/RS/ International Conference on Intelligent Robols and Systems (IROS), pages 9698-9705, 2024.
[54] Leon Sievers, Johannes Pitz, and Berthold Bäuml. Learning Purely Tactile In-Hand Manip- ulation with a Torque-Controlled Hand. In 2022 International Conference on Robotics and Automation (ICRA), pages 2745-2751, May 2022.
[55] Austin Stone, Oscar Ramirez, Kurt Konolige, and Rico Jonschkowski. The distracting con- trol suite a challenging benchmark for reinforcement learning from pixels. arXiv preprint arXiv:2101.02722, 2021.
[56] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning, In ICML, 2021.
[57] Sudharshan Suresh, Haozhi Qi, Tingfan Wu, Taosha Fan, Luis Pineda, Mike Lambeta, Jitendra Malik, Mrinal Kalakrishnan, Roberto Calandra, Michael Kaess, Joseph Ortiz, and Mustafa Mukadam. Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation. Science Robotics, page ad10628, 2024.
[58] Stephen Tian, Frederik Ebert, Dinesh Jayaraman, Mayur Mudigonda, Chelsea Finn, Roberto Calandra, and Sergey Levine. Manipulation by Feel: Touch-Based Control with Deep Predictive Models. In 2019 International Conference Robotics and Automation (ICRA), pages 818 824, May 2019. ISSN: 2577-087X.
[59] Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method, April 2000. arXivvphysies0000055.
[60] Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick Van Der Smagt, and Jan Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 3928-3934, Daejeon, South Korea, October 2016. IEEE.
[61] Herke Van Hoof, Tucker Hermans, Gerhard Neumann, and Jan Peters. Learning robot in- hand manipulation with tactile features. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 121-127, Seoul, South Korea, November 2015. IEEE.
[62] Nikola Vulin, Sammy Christen, Stefan Stevšić, and Otmar Hilliges. Improved I Learning of Robot Manipulation Tasks Via Tactile Intrinsic Motivation. IEEE Roboties and Automation Letters, 6(2):2194-2001 April 2021. Conference Name: IEEE Robotics and Automation Letters.
[63] Niklas Wahlström, Thomas B. Schön, and Mare Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint: /0Xiv:/502.0225// 2015.
[64] Chen Wang, Shaoxiong Wang, Branden Romero, Filipe Veiga, and Edward Adelson. Swinghot: Learning physical features from in-hand tactile exploration for dynamic swing-up manipulation. In IROS, 2020.
[65] Han Wang, Erfan Miahi, Martha White, Marlos C Machado, Zaheer Abbas, Raksha Ku- maraswamy, Vincent Liu, and Adam White. Investigating the properties of neural network representations in reinforcement learning. Artificial Intelligence, 330: 104100, 2024.
14

---

[66] Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: locally linear latent dynamics model for control from raw images. In NewrIPS, 2015.
[67] Linhan Yang, Bidan Huang, Qingbiao Li, Ya-Yen Tsai, Wang Wei Lee, Chaoyang Song, and Jia Pan. TacGNN: Learning Tactile-Based In-Hand Manipulation With a Blind Robot Using Hierarchical Graph Neural Network. IEEE Robotics and Automation Letters, 8(6):3605-3611
June 2023.
[68] Max Yang, Yijiong Lin, Alex Church, John Lloyd, Dandan Zhang, David A.W. Barton, and Nathan F. Lepora. Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing. IEEE Robotics and Automation Letters, 8(9):5480-548) September 2023.
[69] Max Yang, Chenghua Lu, Alex Church, Yijiong Lin, Chris Ford, Haoran Li, Efi Psomopoulou, David A. W. Barton, and Nathan F. Lepora. Anyrotate: with sim-to-real touch. In Conference on Robot Learning (CoRL), 2024.
invariant in-hand object rotation
[70] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In AAAI, 2021.
[71] Zhao-Heng Yin, Binghao Huang. Yuzhe Qin, Qifeng Chen, and Xiaolong Wang. Rotating without seeing: Towards in-hand dexterity through touch. Robotics: Science and Systems, 2023.
[72] Bang You and Huaping Liu. Multimodal information bottleneck for deep reinforcement learning with multiple sensors. Neural Networks, 176: 106347, August 2024.
[73] Ying Yuan, Haichuan Che, Yuzhe Qin, Binghao Huang, Zhao Zhao-Heng Yin, Kang Won Lee, Wu, Soo-Chul Lim, and Xiaolong Wang. Robot synesthesia: In hand manipulation with visuotactile sensing. In ICRA, 2024.
[74] Wancong Zhang, Anthony GX-Chen, Vlad Sobal, Yann LeCun, and Nicolas Carion. Light- weight probing of unsupervised representations for reinforcement learning. In RLC, 2024.
15

---

Appendix
A Results as physical metrics
Figure A1 shows the mean number of seconds it takes the agent to locate the object within different tolerances. The distance d measured hetween the end-effector (imaginary fixed frame in the center of the parallel-jaw gripper) and the object center. While the performance between a proprioceptive and proprioceptive-tattin agent is similar for a = em threshold, the benefits tactile data become more pronounced with smaller tolerances. The relative performance between SSL objectives consistent except for d = 0.05 em, where FR and TFD are the fastest. Figure A2 shows the number of bounces the agent achieves in 10 seconds throughout learning, and Figure A3 shows the number of complete Baoding rotations achieved. Due to the overlapping performance distributions in Baoding, provide alternative figure versions with only subset of runs and/or bad seeds. Across all seeds, from Figure A3 (bottom left) we can see applying tactile reconstruction or full dynamics self-supervision approximately doubles the number of complete rotations achieved.
Within 3cm
within 1cm
within 0.05cm
PrOppaat
4.5
PrOppeap mell
4.5
4.5
sPul Recomatruc3 (F8)
+oriile Recoremettin (T))
Dynamia IP3)
2.5
Foreurd Dynamica (1+0
3.5
35-
150
125
150
175
125
150
175
130
315
Imosaag (91
Figure A1: Number of seconds to locate the object in Find. We show results for within 3, 1, and 0.05 cm distance tolerances.
Number of bounces
80
PPO(prop)
PPO(prop-tactile
+full Reconstruction (FR)
Tactille Reconstruction (TR)
Dynamics
--- +Tactile Forward Dynamics (TFD)
40
50
75
100
125
150
175
Timestops
Figure A2: Number of bounces seconds Bounce. A bounce is defined as contact after timesteps of contact.
B Impact of auxiliary memory on self-supervised learning
Figure A4 shows the average minibatch (full) dynamics loss of Baoding agent with varying auxiliary memory size. There are 19 rollouts between each peak, and with a rollout length of 32 steps this corresponds to 608 timesteps 1 episode. We interpret the loss peaks coming from episode resets when the Baoding balls are set above the palm and potentially spend large portion of one rollout falling into the palm, disrupting SSL the loss of tactile data is highly out-of-distribution Increasing the memory size flattens and widens out the peak, potentially leading to smoother gradient updates and the improved agent performance in this task.
16

---

Number of Baoding rotations (all seeds)
Number of Baoding rotations (na bad seeds)
PPO(srep)
PPolprop-tacttle)
14
+Fll Reconstruction (FR)
12
Tactile Reconstruction (TR)
Forward Dynamics (FD)
10
lactile Forward Dynamics (TFD)
8-
8
4-
2-
25
50
75
100
125
150
175
25
75
100
125
150
175
Timesteps
Timesteps
Number of Baoding rotations (minimal, all seeds)
Number of Baoding rotations (minimal, bad seeds)
16
14
12
12
10
8-
6-
6
2-
2
150
Timesteps (M)
Timesteps (M)
Figure A3: Number of complete rotations in Baoding in 10 seconds. rotation is when each ball returns to its initial position. Left: The distribution across all seeds. Right: The distribution across seeds that learned least rotation. Top: All experiments. Bottom: A subset of experiments.
Baoding (+ (+otea))) average SSL loss
0.050
Masuu
0.045
Masuus 2
Nassues =d
Average SSL loss
0.040
0.025
0.020
100
150
200
Number of RL rollouts
Figure A4: Auxiliary loss memory size. Increasing the auxiliary memory size reduced the spike amplitude of the mean minibatch self-supervised dynamics loss of a Baoding agent.
C Human capabilities
Find. Replicating the environment in real life and testing with one human subject, the average time to find and grasp the object across 10 trials was 2.1 seconds.
17

---

Bounce. The most tennis ball touches using the hand in one minute is 353, and was achieved by Manikandan Thirumaniselvam India on 4 February 202:[ This translates 353/6 59 bounces in 10 seconds. We note the properties of our ball (30g. 70mm diameter) are different to tennis ball (~ 58g, m7 diameter), but would expect to see similar results.
Baoding, The fastest demonstration we could find online achieves 13 rotations in 10 seconds We believe the properties of our Baoding balls and the ones in the video are identical (we acquired the same ones and took measurements: 55g, 1.5 inch diameter).
D Future tactile state prediction analysis
To understand how well MLPs can model the forward dynamics of tactile interactions, we provide var- ious classification metrics throughout training (Figure A5 and spatio-temporal rollout visualisations of trained Bounce and Baoding agents (Figures A and A7). We can compute metrics such as recall and precision since tactile reconstruction was formulated as classification (rather than regression), but unlike typical supervised learning the performance does not increase monotonically because of the nonstationary data distribution.
Overall, we were surprised how robust the performance remained n timesteps into the future (evidenced by how difficult it is to distinguish between the timesteps in Figure This suggests that the multi-step dynamies objective was very effective at encoding information relevant for future state predictions. Moreover, the rate of missed contacts was <1% throughout training for both Bounce and Baoding, which we attribute to the high positive weighting applied. Interestingly, despite the agent having access to the 3 last tactile readings that would form the first 3 tactile readings of the prediction, some of these states were not always perfectly "copied" over (e.g., O5, 011, and 012 in Bounce). This highlights that the readings are not merely being 'memorised', but being represented in some (imperfect) way. For future work, we believe dynamically updating the positive weighting as the inverse mean of the minibatch would be beneficial to reflect the nonstationary training distribution. Additionally, our implementation applied the same weighting to each activation region, but some regions are much more active than others (e.g., compare palm pinky in Figure A7). and this discrepancy should accounted for.
'Bounce record: https://www.guii wtg.iinnwweeeeeeeoooooooeeeerroe callii--alllttonee ball-touchhs-
using-the-hane Bounce video:
T4A
Baoding video: http://www..uutbbeecom/tttttttss
18

---

1.00
0.35
Bounce Baoding
0.98
0.96
0.25
Bounce Baeding
0.15
0.10
500
600
100
200
rollouts
(a) TPR TP-EN (also known as recall or probability of detection, proportion of actual con- tacts that were classified as contacts)
TP
(b) FPR PP
FT47X (also known as probubility
false alarm, proportion of null contacts that were misclassified contacts)
FN rate
TN rate
0.010
1.00
Bounce Baoding
0.008
0.38
0.002
0.92
Baoding
0.000
0.90
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Number rollouts
rallouts
FNR tacts that were missed)
FN TP-FN (proportion of actual con-
(d) TNR FP4TN (proportion of null contacts were classified as null contacts)
Accuracy
Precision
1.00
0.325
Bwoding
0.99
0.900
0.825
0.800
Bounce
0.775
Baoding
0.94
0.750
600
200
Number rollouts
Number rollouts
(e) Accuracy A TP+FP-IN+EN
TP+TN
(i) Precision
T.P TFFFF
Figure AS: Classification metries for TFD agents. We compare the predicted vs true future tactile states of Bounce and Baoding agents trained with TFD for 1, 3, and 1, 2 timesteps into the future respectively. The metrics for future timesteps are shown with decreasing opacity but strongly overlap with t 1, making it difficult to distinguish. The metries were computed on one minibatch after the learning update.
19

---

For Bounce, tactile interactions are increasingly sparse (e.g., compare Figures A6 and A7 Thus the metries we were most interested in were true positive rate (proportion of contacts caught) and false negative rate (proportion of contacts missed). True positive rate (TPR) decreased from ~999 to 90% throughout training, which we attribute to increased difficulty predicting the landing sites of bouncing ball. The proportion of contacts that were missed (FNR) was surprisingly low throughout training, converging to ~ 0.2%. Fascinatingly, from the no-contact observation o7, the decoder correctly anticipates contact in the next state (albeit in the wrong locations, however two predictions are just timestep early). This result suggests that object height information being encoded.
S4
Predicted activation
Overlap
55
S6
57
S8
S9
$10
S11
512
Figure A6: Spatio-temporal visualisation of 1-step predicted future tactile states of TFD Bounce agent. The tactile observation ofact displayed here as is comprised of k = 4 past readings, and each column corresponds the same reading. The sigmoid activation of the tactile prediction is displayed with varying opacity (e.g.. less confident is lighter blue).
20

---

For Baoding, the frequency of tactile interactions remains high throughout learning, thus all metrics are of relevance. The proportion of contacts that were correctly detected was ~99%, which is very high. Similarly to Bounce, very few true contacts were missed (FNR), and the false positive rate was relatively high at ~15%. The accuracy remained at >966 for majority of training. Also similarly to Bounce, some of the false positive predictions (e.g., in observations 010- 011) are just one-step too early. Finally, across all metrics in Figure A5 we can see negative performance spikes at fixed intervals throughout learning that are not present for Bounce. This is discussed in Appendix B
$3
S4
Real activation
Predicted activation
Overlap
S6
S7
S9
S10
S11
S12
Figure A7: Spatio-temporal visualisation of 1-step predicted future tactile states of TFD Baoding agent. The tactile observation sfact displayed here as st, is comprised of & = 4 past readings, and each column corresponds the same reading. The sigmoid activation of the tactile prediction is displayed with varying opacity (e.g.. less confident is lighter blue).
21

---

E POMDP
E.1
Observations
summary of the observations is shown in Table A1 Each environment uses k-length history of sensor readings to form the observation On (k 16 for Find and & = 4 for Bounce and Baoding. found empirically). We apply input preprocessing as follows: joint angles 0 are normalised between [-1,1]. Joint velocities á are scaled down by scalar (0.33 for Franka, 0.2 for Shadow). We retrieve the net contact forces through Isaac Lab's ContactSensor class and binarise it. To mimic real-world tactile sensing and make the task more challenging, we registered two pplate-ikee' bodies to atop the Franka fingers to act contact sensors (Figure A8). With this setup, the sensors would only register forces that resulted from collision with these bodies, which is only possible from the object or other finger, and not the ground. Since the Shadow Hand was fixed in midair we let each link become sensor, resulting in 17 sensors (Figure
Table Al: Observation spaces across environments.
Type
Symbol
Description
Find
Bounce
Baoding
0 fact
b
binary contacts
2
17
17
Drop
at-1
last action
20
20
0
joint angles
9
24
24
joint velocities
9
24
24
XgE
EE position
qEE
EE quaternion
4
gripper width observation
Ot
37xk
85
85xk
2
Figure A8: Contact sensors for Find. We placed two bodies on Franka's 's fingers to act as contact sensors.
E.2
Actions
All robots are joint position controlled, with a E R for Franka and a, E 20 for the Shadow Hand. The Shadow Hand is underactuacted, with coupled distal and proximal joints (2 wrist + 5 thumb + index + middle + ring + pinky + metacarpal = 20).
E.3 Rewards
The rewards for each environment step are given by the sum of the different terms each multiplied by the given scale. For enhanced value function learning, track running mean and variance for normalising the returns and values in the PPO update.
For Find there is reward term Faist that grows distance between the object and end-effector decreases.
22

---

For Bounce, the reward is given by Tair + rbounce + fflll. To aid initial exploration, the agent is rewarded proportionally to number of time steps since last contact, Tair- A bounce event is defined as there being contact after 5 timesteps of no contact, for which there is bonus Tbounce- The fall penalty is applied if the object more than em from fixed center position.
For Baoding the reward is given by Tdist, + Tdist, + Trotation + falll Our original approach was maximise the zy angular velocity the vector connecting the two balls which worked well, but sometimes the agent came up with creative strategies that no longer resembled the original Baoding task. Thus, we reformulated the reward around two fixed target poses (Figure A9 When the centers of both balls were within 1.0 cm of the given target centers, the targets switched and the agent receives a bonus reward rooaaionn We also used two dense ball-center-te target distance rewards Takst, Idistz to aid exploration in the beginning. This approach better constrained the ball positions and stabilised policies for comparing methodologies. The fall penalty applied if the distance between the balls exceeds 15 cm.
Table A2: Reward components across environments.
Symbol
Description
Equation
Scale
Find Bounce
Baoding
Tdist Obounce
distance to target time without contact successful bounce
tanh dtarget/O.1 0.11
0.1.1
+1 +1
0.01
10
Trotation fflll
successful rotation fall penalty
+1 if dh& < 1.0cm if d dmaz
10
Figure A9: Virtual targets in Baoding. When both balls are within lcm of their virtual targets (shown smaller balls), the targets switch and the agent recieves bonus.
E.4 Reset
Episodes can be terminated by failure state, truncated by time limit. The Find environment episode length is T = 300 timesteps (5s), and Bounce and Baoding environments are T = 600 timesteps (10s). The Bounce and Baoding episodes can be terminated early if the balls fall out of the hand, measured by a distance. At the beginning of each episode, the Franka joint angles are randomised up to 7 and the Shadow joint angles randomised upto + 20%. The ball in Find randomised to any position on the 20cm 20cm plate. The Bounce ball and Baoding balls are randomised by +1 cm and +0.5cm respectively along the global ryz axis
Network architectures
Encoder:
The encoder e is a 3-layer MLP with dimensions o, - 1024 - 512 + 256 + Zt. Layer normalisation and ELU activations are applied after each layer.
Policy. The policy æ 3-layer MI.P with dimensions 2t - 128 - 64 Ractions - a4. ELU activations are applied after the first two layers. The output layer activation is tanh.
23

---

Value function. The value function v is 3-layer MLP with dimensions - 128 - 64 + + (o,)). ELU activations are applied after the first two layers. The output layer activation is identity,
Reconstruction. The decoder is a 3-layer ML.. The full reconstruction decoder has dimensions Zt 512 - 512 ( Ôt. The tactile reconstruction decoder has dimensions 2t - 512 + 512 + [ ftact) + o ELU activations are applied after the first two layers. The output layer activation is sigmoid for tactile predictions, and identity for proprioception predictions.
Forward dynamics. The forward model is a 3-layer MLP with dimensions (Z,, a.) + 512 - 256 + 256 - 2,+1, with ELU activations after the first two layers. The projector p is 2-layer MLP with dimensions 2041 + 256 + 256 + L, with an ELU activation after the first layer. The target encoder eT is identical to e, but updated according to Equation 7 with 0.01.
Oer - - -7) + T0p
G Hyperparameter tuning
We carefully tuned all experiments give each agent the best shot. For faimesss we followed the same hyperparameter tuning recipe for each individual experiment (3 environments x 7 experiments 21 sweeps). We use the Optuna library B with the TPE sampler (5 startup trials) and no pruner. We wait for each sweep to reach 20 complete trials (some hyperparameter combinations lead to policy/value NaNs which are terminated early). The hyperparameters and possible ranges we tested are provided in Table[A3] with the optimised values in Table[A4] We did not sweep over the following hyperparameters: discount factor y = 0.99, value loss scale Cu 0.1, gradient norm clip 1.0, value clip 0.2, ratio clip 0.2. We provide the mean evaluation returns of each run in the sweep of Bounce with full dynamics self- self-supervision in Figure AA0 to demonstrate the sensitivity of our agents to hyperparameters, and illustrate the importance of performing quality hyperparameter tuning for each individual experiment.
Table A3: Tunable hyperparameters and ranges for each experiment.
Hyperparameter
Symbol
Tunable values
Rollout Minibatches
R
116,32,64 64]
mb
(4,8,16, 32, 64)
Learning epochs
le
{4,8, 32]
Learning rate
Ir
110-5,10 5,100-1
Entropy loss scale
Cent
10,0.05.0.1)
Auxiliary learning rate
Irauz
10 1000,100
Auxiliary loss weight
Cauz
[10 10] C R
Dynamics sequence length
1,,3,4,10} (2.3.4) 4)
Auxiliary memory size
Nroiloats
Bounce (+ full dynamics) sweep
900
800
Mesn Evaluation retur
700
600
500
400
300
200
100
25
100
125
150
Timestep (M)
Figure A10: Example Bounce sweep. All experiments were highly sensitive to different hyperpa- rameter combinations and seeds.
24

---

Table A4: Tuned hyperparameters for each experiment.
Environment Find
Experiment
mb
Ic
Cent
Fraur
Cauz
n
Nrollouta
PPO(prop)
32
16
8
1.06 x
0
(ropproptt
32
1.06 x10=
+FR
64
x10
5.91
5
0.0023
+TR
64
16
1.36 x10
0.1
2.55
x10-5
0.004477
+FD
64
64
4
1.15 x10
0.1
1,55
x10
0.0062
2
+TFD
64
64
4
2.32 x10
0
1.57
x10
0,0024563
4
+ +FD+Neottecta
64
64
4
1,15
x10
0.1
3.81
x10
0.1364
3
Bounce
PPO(prop)
32
32
4
5.93
x10
0
PPO(prop-tactile)
16
8
4
3,21
x10
0
+FR
64
16
16
1.88 x10
2.77 x10
0.05669
+TR
64
x10
5.13 x10-5
0.00384
+FD
64
1.50 x10
4.53 x10-6
0.8462
10
+TFD
64
32
8
1.22 x10
0.05
1.64 x10-
0.23547
10
FFD+NNoottnct
32
64
1.50 x10
1.16
x10
0,19954
4
Baoding
PPO(prop)
32
9.96 x10
0.05
(Po(prop-tcctile)
32
4
8
2.02 x10
0.05
+FR
16
32
4
3.68 x10
0
5.18 x10
0.058866
+TR
64
32
8
3.61
x10
0
1.00
x10-
0.2707
+DD
32
16
5.47
x10
0
2.87
x10
3.686
2
+TFD
32
16
x10-
0.05
1,53 x10-4
0.04839
+DD+Noott
x10
1.6349
H Latent trajectory analysis
Figures[AlT] A12 and A13 show a two dimensional latent representation of a single episode across all environments. Trajectories for RL-only and a subset of self supf-super agents shown (TR, FD, and TFD). The 256-dim observation representation 24 at each timestep was reduced using 2-component Principal Component Analysis (PCA). Note that the tactile activations shown are only the sum of activations in the current reading o tact", and does not sum across the reading history.
Baoding. The ring-like trajectory of the RL-only agent illustrates the repeated motion the agent develops. There are tactile peaks opposite sides of the ring, indicating symmetry in contact activations between half-rotations. The trajectory the TR agent is quite different (heart-shaped, diffuse). This shows each rotation is slightly different, and there is now asymmetry between the contact activations of half-rotations. From rendering the policy, the gait is smooth like the FD agent but keeps the balls close together like the RL-only agent. The trajectory of the FD agent is again ring-like, but with tighter bounds than the RL-only agent. Like the TR agent, there is contact activation asymmetry between half-rotations. Finally, the TFD agent trajectory appears be blend of the FD and TR trajectories.
Bounce. The latent trajectories of the self self-supervised agents are highly different to the RL-only agent. From the trajectory with the time colourbar, we can see that the sequential latent states of the RL-only agent highly discontinuous and far apart (e.g., yellow), and the agent repeats the same motion with high precision. There are two regions with non-zero tactile observations of upto activations, which is understood by the 'safe' gait of raising the index and pinky finger stabilise the ball. The gait changes completely for the self-supervised agents, which predominately use or contacts. Sequential latent states are still spread out in various regions, but these regions are much more diffuse than in the RL-only.
Find. It is clear the self- self-supervised agents find the object faster by observing the trajectories colourised by time. Otherwise, the shape of the latent trajectories is not drastically different between RL-only and seff-supervised agents. A distinction between and tactile activations appears in the FD agent trajectory.
25

---

Baoding (RL-only)
Baoding (RL-only)
10
Number of tacticc activations
5
0
-5
-10
-10
-5
-10
-5
Baoding + tactile reconstruction
Baoding + tactile reconstruction
5 Number of bactle activations
5
-5
2
0.2
-10
0.0
Baoding full dynamics
Baoding full dynamics
1.0
Number of tactle acivvations
0.8
2-
-2
0.0
-6
Baoding tactile dynamics
Baoding tactile dynamics
10
1.0
d.
Number tactile acivations
0.8
0.6
0.4
-
10
10
10
-10
10
Figure A11: Latent episode trajectory (PCA) of RL-only, TR, FD, and TFD Baoding agents.
Left: Samples colourised by time. Right: Samples colourised by summed binary contacts of the last tactile reading o fact!
26

---

Bounce (RL-only)
Bounce (RL-only)
10.0
10.0
1.0
7.5
7.5
Number of bacille activations
0.8
2.5
0.6
0.0
0.4
-25
225
0.2
5.0
0.0
-7.5
-10
Bounce + tactile reconstruction
Bounce tactile reconstruction
10.0
10.0
7.5
7.5
Number trcctio act sntvottons
5.0
5.0
2.0
2.5
2,5
1.5
0.5
0.0
0.0
Bounce + dynamics
Bounce dynamics
2,00
1.75
toctlle activatins
2
1.50
6
0.6
-
0.4
0.75
0.50
0.2
0.25
Bounce tactile dynamics
Bounce tactile dynamics
8-
2,00
1.75
Mumber activatoos
4
2
1.00
0..4
0.75
0.50
0.2
-4-
-2
Figure A12: Latent episode trajectory (PCA) of RL-only, TR, FD, and TFD Bounce agents. Left:
Samples colourised by time. Right: Samples colourised by summed binary contacts of the last tactile reading o tact*
27

---

Rind (RL-only)
Find (RL-only)
5.0
2.00
2,5
1.75
Number cacile activattod
0.0
1.25
-25
2.5
1.00
7.5
0.50
10.0
-10.0
0.00
10
10
Find tactile reconstruction
Find + tactile reconstruction
10.0
10.0
2.00
7.5
1.75
toctt activatons
5.0
1.50
2.5
-1.25
0.0
1.00
-25
0.75
-5.0
-5.0
0.50
0.00
110.0
-10.0
-12.5
-10.0
-7.5
-5.0
-2.5
0.0
5.0
-12.5
-10.0
-7.5
0.0
2.5
5.0
Find dynamics
Find + dynamics
10
10
1.75
Number to0 actvvatinn
0.8
1.50
0.6
1.25
1.00
0-
0.25
0.00
Find tactile dynamics
Find tactile dynamics
2,00
10:
1.75
activatton
5
0
1.00
0.4
0.75
0.2
5
0.0
0.00
Figure A13: Latent episode trajectory (PCA) of RL-only, TR, FD, and TFD Find agents. Left: Samples colourised by time. Right: Samples colourised by summed binary contacts of the last tactile reading o fact*
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: claim that our proposed SSL objectives improve upon the performance of end-to-end PPO agents, which matches our experimental results. The introduction accurately reflects the scope of the paper, which a study into tactile-based RL for robot control.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We include a limitations section in our discussion which acknowledges our lack of real-world robot experiments, and the computational requirements/effciiency of our method compared end-to-end RL.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not have theoretical results, only empirical.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We disclose the algorithms, hyperparameters. architectures and environmental characteristics needed to reproduce our results. We will release all code and data upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
30

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide full access to code and data upon acceptance, with scripts for generating each figure from the data and rumning each experiment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wwtpcc/pccppcpcceeeooooooop hutee/ggguudeee/eeeuuuubiiiiiiii ieriiisssoooooii
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we specify our full hyperparameter optimisation strategy and final values in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, we run 5+ seeds per experiment and plot the mean and + standard deviation of the distribution.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the discussion we provide the compute resources necessary and an estimation of total compute time rerun the full paper experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htw.up/ httpp//wwwpppp.ppp......aaaaaaaa
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss possible positive and negative consequences of our work in the discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper does not pose this risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite Isaac Lab following instructions on their webpage. We do not explicitly mention the licenses but they are properly respected: NVIDIA Isaac Sim is available freely under individual license, and Isaac Lab is open-sourced under the BSD-3-Claus license. We do not use external datasets models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release our code as a benchmark for the community upon acceptance. The code will be well presented and documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method in this research does not involve L.Mss.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
35


Reviews:
[{"review_id": "ghVOB1afdm", "reviewer": "Reviewer_zxcc", "summary": "This paper proposes a general-purpose self-supervised learning methodology that effectively leverages tactile input for downstream robotic control tasks in RL. Firstly, the manuscript provides a novel application and analysis of tailored reconstruction and multi-step dynamics objectives that enable the agent to leverage its tactile observations. Secondly, the proposed method proposes to separate the auxiliary memory from the RL rollout memory to stabilise and strength the self-supervision updates. Experimental results show the superiority of the proposed self-supervised agents compared to RL-only agents on three kinds of tasks: Find, Bounce, Baoding in Issac Lab.", "questions": "Please see Weaknesses section. In my opinion, the poor organization of the manuscript, the lack of real-world experiments, and the insufficient method design are the major issues of the manuscript. Therefore, I lean to recommend rejection in the current version of submission.", "limitations": "Yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. The proposed method show the superiority over RL-only agents on multiple kinds of tactile-based complex manipulation tasks. \n\n2. The authors provide various aspects of discussion to analyze the advantage of the proposed dynamics objective and self-supervised learning strategy.\n\n3. Supplementary materials are provided with detailed implementation details and additional experimental results.\n\n### Weaknesses\n1. A major concern is the organization and presentation of the manuscript, which largely impact the readability. For example, the Method section contains a lot of symbols without providing a notation summary table to explain the meaning of each variable. Moreover, the captions of figures do not explain the figure (e.g., the pipeline of method and the meaning of the symbols in Figure 1) in detail. Thirdly, what is the total training objective and how the self-supervised learning cooperate with the RL objective are unclear from Figure 1. Fourthly, In the experimental results, the authors only provide the comparison results in the form of figures instead of tables, making the comparison indirect. I strongly suggest the authors to refine and reorganize their manuscript to improve the clarification and readability.\n\n2. The design of the proposed method seems to lack enough technical details and contents. The core contribution is the self-supervised training strategy, which has been widely adopted in previous works. The separate auxiliary memory is also an important contribution as described in Line 77-80. However, the authors only introduce it briefly in Section 3.3. \n\n3. In Line 70, the authors claim that the proposed method is supposed to have low sim2real gap, however, the manuscript does not contain the real-world experiments, as indicated in the Limitation section. These makes the adaptability of the proposed method to the real world doubtful.", "was_revised": true, "final_justification": "I appreciate the authors for providing the rebuttal, which addresses some unclear points, such as the training objectives and missing implementation details. However, my concerns about the lack of technical contribution and sim2real transfer experiments are not well-solved through the rebuttal. The lack of real-world experiments are also indicated by many other reviewers. Based on the above concerns, I maintain my original score.", "rebuttal": "We thank **zxcc** for their comments and constructive feedback. As a reminder, we introduce a new approach that improves sensory-driven robotic agents for complex manipulation tasks that does not make idealised assumptions about the quality of tactile state information available. In addition, we introduce three new environments for benchmarking tactile-based manipulation methods. \n\n**zxcc**’s concerns primarily relate to the structure of the paper, missing details, and the lack of experiments on real robots. With the exception of the last one, which we attempt to justify why its absence does not impact our contributions, the other concerns can be addressed by updating the final camera ready text. \n\n\n**[zxcc-1] Organization and presentation of the paper.**\n\nWe thank the reviewer for highlighting these clarity issues, and will make the following changes in light of zxcc’s comments:\n* *Notation:* While all symbols are sequentially introduced in the text, we recognise that a central reference would be helpful. We will add a summary of the most important notation at the beginning of the Method section to serve as a quick reference for the reader. \n* *Figure 1 caption:* We will update the caption for Fig. 1 (see response **[Xsn5-3]**) so that it is more self-contained and will expand it to provide a more detailed step-by-step description of our method's pipeline in the camera-ready version.\n\n\n**[zxcc-2] Training objective and role of self-supervised learning.**\n\nThe total training objective is given in L147: _“The self-supervised loss is added to the policy, value, and entropy loss and backpropagated in a single pass. The auxiliary loss optimises the encoder e and its own task-specific networks.”_ We did not include this in the original figure because it is the default optimisation strategy in self-supervised RL, and is not a novel contribution for our work. In the camera-ready version, we can explicitly add the loss summation for improved clarity.\n\n\n**[zxcc-3] Quantitative results are presented as figure instead of table.** \n\nIn the main body of the paper we chose to display results as a figure of the agent’s mean evaluation return throughout learning, because we are interested in understanding the impact of incorporating self-supervised objectives throughout the optimisation process, and figures provide more information on sample efficiency and stability. For a more direct comparison, we provided a bar chart in Fig. 6 to show how different methodologies correspond to improvements in physical quantities. In the camera-ready version, we can add a table of the highest obtained mean evaluation returns for each methodology. \n\n\n**[zxcc-4] Missing implementation details about the proposed method.** \n\nOur problem setting is outlined in Sec 3.1, the self-supervised objectives in Sec 3.2, and auxiliary memory is described in Sec 3.3. Additional implementation details are provided in Sec 4 and Sec E, F, and I of the supplementary PDF. We will revise the descriptions and more clearly point to the additional implementation details in the main paper. If there are specific aspects of the implementation details that are not clear, we are happy to discuss them during the author-reviewer discussion period - please do not hesitate to ask additional questions.  \n \n**[zxcc-5] Self-supervised training has been used in previous works**\n\nYes, we agree, and do not claim in the paper that the use of self-supervised learning in combination with reinforcement learning is novel. In the related work (Sec 2) we summarise how different self-supervised training strategies have been adopted over the past decade. Indeed, reconstruction and dynamics-based objectives have been widely adopted in previous works, because of their general-purpose nature. Please see the summary of our contributions in the introduction (Sec 1) and response [zxcc-7] for the novelties we claim. \n\n\n\n**[zxcc-6] The separate auxiliary memory is an important contribution but only briefly discussed.**\n\nWe thank **zxcc** for highlighting the separate auxiliary memory as an important contribution. The description is brief because the method itself is intentionally simple, and we have described the implementation in its entirety in Sec 3.3. The contribution is not an unnecessarily complex new algorithm, but rather the insight that decoupling the auxiliary memory from the on-policy buffer and significantly expanding its size can be a simple yet powerful technique (see results in Fig. 5).\n\nTo be concrete, for an agent using $B$ parallel environments and a rollout length of $R$, the on-policy data has a shape of $[B, R, ...]$. Our method simply involves storing data for the auxiliary tasks in a larger, separate buffer of size $[N_{rollouts}, B, R, ...]$, where $N$ is the number of past rollouts we keep in memory. \n\n**[zxcc-7] Sim2real gap and real world experiments.**\n\nWe wish to clarify that our paper does not claim the proposed method has a low sim2real gap; rather that we designed our methodology with this long-term goal in mind. As stated in our Introduction (L62), _“We desire the following characteristics… (d) low sim2real gap: inspired by the low-cost setups in [61, 63], we study learning with binary tactile activations… to avoid transfer difficulties that can come with continuous measurements [61]”_. We also acknowledge in our Limitations (L314) the absence of real-world experiments and express we _\"hope our choice to study binary tactile activations greatly minimises the potential sim2real gap”._\n\nWe believe the absence of real-world experiments does not detract from our work’s core contributions for the following reasons:  \n\n1. *Establishing a simulation baseline for novel tactile-based robotics.* We address an important but largely unexplored domain, i..e, tactile-based robotic agents operating without visual input. Given the scarcity of prior work (see L41-50), our paper provides a crucial foundational benchmark for future research in this novel setting.  \n2. *Solving a fundamental problem of neural network-based RL agents.* Our paper's central contribution is a new methodology that markedly improves how agents leverage binary tactile observations in simulation (see Fig. 6). As demonstrated in the supplementary videos and latent trajectory analysis in Sec J of the supplementary PDF, agents using our method discover unique, effective strategies for complex, partially observable control tasks that traditional RL agents do not.   \n3. *Robustness across diverse dynamics.* To counter concerns about simulation-specific gains, we validated our method across three tasks with distinct contact dynamics and different simulated robotics hardware (Find: sparse, Bounce: intermittent, Baoding: continuous). The consistent high performance across these varied settings and robot morphologies (see Figs. 4 and 5), strongly indicates a genuine advancement in learning capabilities, rather than a mere simulation artifact.  \n4. *Designed for sim2real transfer*. Our design choices, particularly the use of binary contact activations, are deliberate steps to facilitate future real-world transfer. Prior work [61, 63] has demonstrated robust sim2real transfer with binary tactile activations even in dexterous manipulation tasks. Furthermore, our method's focus solely on modifying network weights simplifies real-world deployment. We contend that a superior simulated policy, assuming the sim2real gap is otherwise bridged for a tactile-based RL agent, should indeed transfer, and our validation across diverse tasks suggests our policies are not unduly reliant on unrealistic simulation physics as asked by WBY5.  \n\nWe believe that our comprehensive empirical study in controlled simulated environments provides substantial findings and novel insights that will be of immediate interest and benefit to researchers working at the intersection of machine learning and robotics. Specifically, we demonstrate the proven efficacy of binary tactile activations for complex dexterous manipulation, addressing a key open question in the tactile sensing community. We posit that it is important to first establish the efficacy of our findings in simulation, via a repeatable benchmark that can be used by others, before moving on to real world robotics experiments in future work."}, {"review_id": "CwNVBWPSKH", "reviewer": "Reviewer_gp4F", "summary": "This paper posits that the current tactile manipulation policies entangle the proprocioptive and tactile representations making it unclear and also hard to attribute the benefit of \"tactile\" in tactile manipulation. To investigate this, the authors propose to disentangle and learn SSL methods on top of the tactile states as a form of auxiliary loss to guide the RL policy learning. Specifically, they consider tactile dynamics and tactile reconstruction as 2 SSL tasks.\nFurther, to evaluate the method, the authors propose a benchmark - RoTO: Robot Tactile Olympiad that includes 3 tasks, namely Find, Bounce, and Baoding involving parallel gripper (franka arm) and dextrous manipulation (shadow arm).", "questions": "3. Figure 1 (right) -- the architecture for multi-step forward dynamics is confusing since there are both L1 and L2 losses being taken. I presume that this was a typo and it's L2 loss everywhere (based on the multi-step forward dynamics subsection). But I wanted to confirm if my assumption is correct.\n\n4. For the forward dynamics, is there any prioritized sampling performed to reduce the imbalance present in the tactile state transitions?\n\n5. Can the authors show the plots for the ablation study on $n_f$ for at least one task? I'm curious how the values were chosen. Was the last $n_f$ value that didn't result in a significant increase from $n_f-1$ selected as the optimal value?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Well-written paper\n\n- The idea of specifically learning tactile dynamics is an interesting one and a good contribution to the field of tactile manipulation.\n\n- Thorough hyperparameter search including learning rate(!), which is often missing in a lot of RL papers.\n\n\n**Weaknesses**\n\n\n1. Related works miss several tactile-based dynamics models such as RoboPack [1] and Swing Bot[2]. Citing & discussing them would be helpful for the reader.\n\n2. Lack of Sim2Real experiments. This is acknowledged by the authors, however, I must note that the correlation between simulation experiments working versus the policy actually working on the real robot is quite large, especially for dexterous manipulation, hence I do consider this limitation as a major drawback of the work.\n\n---\n\n**References**\n\n[1] RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing, Bo Ai et al., RSS 2020.\n\n[2]SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation, Chen Wang et al., IROS 2020", "was_revised": true, "final_justification": "Thanks to the authors for their comprehensive answers during the rebuttal phase. I have looked over all the comments and believe that the authors have adequately addressed all my major concerns. \n\nThanks for clarifying the summary of the paper as well. My major concerns were lack of sim2real experiments and related works. While I still am of the belief that experiments in sim, especially in the RL-simulation community are not trivially transferrable to real world (sorry for the typo in my comment where I said \"correlation between simulation experiments working versus the policy actually working on the real robot is quite large\" when I actually meant that the correlation is quite low) -- I do think that the methodological contributions of this work are well suited for this conference.\n\nHence, I will increase my score to **Accept**.", "rebuttal": "We thank **gp4F** for taking the time to review our paper and for the great questions.\n\n\n**[gp4F-1] Paper summary and the entanglement of proprioceptive and tactile representations.**   \n\nWe appreciate **gp4F**’s summary of our work, but we would like to clarify a key potential misunderstanding. The summary states that we posit entanglement of proprioceptive and tactile representations as the problem. However, our premise is precisely the opposite, i.e., we hypothesise that *the tactile observation often fails to adequately entangle or contribute meaningfully to the overall multimodal representation* (see discussion on L31-40). Thus, the challenge we address is not the difficulty in attributing the benefit of tactile input, but rather the observed lack of expected performance improvement when tactile observations are concatenated to the observation.\n\nFurthermore, while our tactile dynamics and tactile reconstruction objectives are designed to encourage the tactile state to be more salient within the multimodal representation, the key finding was that the most successful approach was the multi-step forward dynamics, which we denoted as “full dynamics” (see results in Fig 4).  \n\n\n\n**[gp4F-2] Thorough hyperparameter search is performed.**   \n\nWe appreciate **gp4F**’s recognition of our thorough hyperparameter search. We fully agree that the common omission of such rigorous tuning in many RL for robotics works can undermine the reliability of reported results. To help address this challenge, we plan to release our integrated hyperparameter optimisation code with Isaac Lab (where it is not available) alongside our benchmark, which we hope will encourage and facilitate the adoption of best practices within the research community. \n\n\n\n**[gp4F-3] Additional related works.**   \n\nWe thank **gp4F** for highlighting the omission of these two tactile-based dynamics works (RoboPack and SwingBot). Our related work section primarily focuses on model-free RL with self-supervised learning. We had originally omitted SwingBot because the focus of that paper is about generalisation to unseen physical properties and it does not use RL, whereas we focus only on in-domain expertise with RL. Similarly, RoboPack uses trajectory optimisation after the dynamics model is learned, whereas we explore improving online policy optimisation with online dynamics learning. However, we appreciate that a discussion of achievements in learning tactile dynamics within other communities, particularly model-based policy learning (e.g., deep tactile MPC [A]), would indeed be valuable and provide a more comprehensive context for the reader. We will revise our related work section to include a discussion of these additional references.\n\n[A] Tian et al., Manipulation by Feel: Touch-Based Control with Deep Predictive Models, ICRA 2019 \n\n**[gp4F-4] Lack of sim2real experiments on real robots for dexterous manipulation.**   \n\nWe acknowledge this point regarding the absence of sim2real experiments, which was also a focus of some of our discussion with reviewer **WBY5** (please see response **[WBY5-[1-3]]** for further context). We believe the absence of real-world experiments does not detract from our work’s core contributions for the following reasons:  \n\n1. *Establishing a simulation baseline for novel tactile-based robotics.* We address an important but largely unexplored domain, i..e, tactile-based robotic agents operating without visual input. Given the scarcity of prior work (see L41-50), our paper provides a crucial foundational benchmark for future research in this novel setting.  \n\n2. *Solving a fundamental problem of neural network-based RL agents.* Our paper's central contribution is a new methodology that markedly improves how agents leverage binary tactile observations in simulation (see Fig. 6). As demonstrated in the supplementary videos and latent trajectory analysis in Sec J of the supplementary PDF, agents using our method discover unique, effective strategies for complex, partially observable control tasks that traditional RL agents do not. \n  \n3. *Robustness across diverse dynamics.* To counter concerns about simulation-specific gains, we validated our method across three tasks with distinct contact dynamics and different simulated robotics hardware (Find: sparse, Bounce: intermittent, Baoding: continuous). The consistent high performance across these varied settings and robot morphologies (see Figs. 4 and 5), strongly indicates a genuine advancement in learning capabilities, rather than a mere simulation artifact.  \n\n4. *Designed for sim2real transfer*. Our design choices, particularly the use of binary contact activations, are deliberate steps to facilitate future real-world transfer. Prior work [61, 63] has demonstrated robust sim2real transfer with binary tactile activations even in dexterous manipulation tasks. Furthermore, our method's focus solely on modifying network weights simplifies real-world deployment. We contend that a superior simulated policy, assuming the sim2real gap is otherwise bridged for a tactile-based RL agent, should indeed transfer, and our validation across diverse tasks suggests our policies are not unduly reliant on unrealistic simulation physics as asked by **WBY5**.\n  \nWe believe that our comprehensive empirical study in controlled simulated environments provides substantial findings and novel insights that will be of immediate interest and benefit to researchers working at the intersection of machine learning and robotics. Specifically, we demonstrate the proven efficacy of binary tactile activations for complex dexterous manipulation, addressing a key open question in the tactile sensing community. We posit that it is important to first establish the efficacy of our findings in simulation, via a repeatable benchmark that can be used by others, before moving on to real world robotics experiments in future work. \n\n\n**[gp4F-5] Losses depicted in Fig 1.**   \n\nWe apologise for the confusion caused by the notation in Figure 1 (right) regarding the multi-step forward dynamics. **gp4F**'s assumption about it being a typo is understandable. To clarify, the subscripts L1​, L2​,…​ do not refer to different types of loss (e.g., L1-norm vs. L2-norm), but rather denote the loss calculated for the prediction at the first step (i.e., L1​), second step (i.e., L2​), and subsequent steps into the future from the current state. The specific loss function used for all these steps is consistently the L2 loss, as correctly identified by the reviewer and detailed in the multi-step forward dynamics subsection. We recognise that this notation is ambiguous and will revise Figure 1 and its caption (see response **[Xsn5-3]**) in the camera-ready version to explicitly clarify this.  \n\n\n**[gp4F-6] Is any prioritized sampling used to reduce the imbalance present in the tactile state transitions for the forward dynamics model?**   \n\nThis is an interesting question. While we did employ positive weighting for tactile reconstruction to address sparsity (see L288), we confirm that *no* prioritised sampling mechanism was used for the dynamics learning. We will clarify this in the revised text.  \n\nDuring initial investigations, we experimented with various sampling approaches, such as rule-based strategies and exponential weighting. However, we did not observe significant performance gains in our specific experimental setup. We wish to emphasise that this outcome does not diminish the potential value of prioritised sampling in tactile dynamics learning. On the contrary, we think that it might be a critical component for handling tactile sparsity and imbalance. The exploration of prioritised sampling, particularly in conjunction with a separated auxiliary memory for transitions, represents a potentially interesting direction for future research.  \n\n\n**[gp4F-7] Can the authors perform an ablation study on $n_f$?**   \n\nWe agree with **gp4F** that the impact of the number of forward prediction steps, $n_f​$, is highly interesting and critical for performance. To ensure robustness and account for potential interactions with other hyperparameters (e.g., learning rate), we treated $n_f​$ as an optimized hyperparameter within our search space, sweeping over values of {2, 3, 4, 10}. This approach is detailed in Appendix I, Table A3, and we will ensure this optimisation strategy is clearly stated in the main paper body.\n\nOur findings revealed task-specific optimal values for $n_f​$. For the 'Find' and 'Baoding' tasks, optimal performance was achieved with $n_f$​ values between [2, 4]. Interestingly, for the 'Bounce' task, a longer sequence length of $n_f​=10$ proved most effective, likely due to its ability to capture a full bounce trajectory, as illustrated in Fig. A5."}, {"review_id": "dLqqrAqD7w", "reviewer": "Reviewer_WBY5", "summary": "This work focuses on enhancing tactile-based (RL) for robotic control by addressing the challenge of effectively exploiting sparse tactile sensory signals. The paper proposes training of RL policies with a binary reconstruction of tactile data streams via an encoder architecture, hypothesizing the higher likelihood of signal value and sim2real transfer over continuous force readings. They further propose introduction of auxiliary reconstruction and prediction objectives and separate memory for the RL task policy and auxiliary objective policy in order to encourage the policies not to ignore the haptic signals in favor of richer proprioceptive signals. To test the hypotheses and validate their claims, the robot tactile olympiad simulation benchmark is introduced and results demonstrate the super-human performance on bounce and baoding tasks in simulation.", "questions": "I am interested in the authors' opinions on the value of pursuing and optimizing for superhuman results in simulation. I think it is great to see such results and the videos help to put the quantitative analysis into context, but I know first-hand how simulation inaccuracies can enable non-physical behaviors which policies can and will latch onto. For example, oscillatory impulses applied by contact resolution between two frames at high frequency could masquerade as \"bouncing\" or \"vibration\". Concretely, imagine how different soft contacts between a human hand and ball would be from a rigid approximation in an impulse based engine. The question boils down to: \"how far should we trust impulse-based simulation models with regards to highly dynamic tasks?\".", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall interesting hypothesis, well conducted research, and strong results/discussion. The auxiliary objectives, novel architectures, and new benchmark appear to be positive contributions to the field and I appreciate the focus on scientific discussion and suggestions to others working on similar problems. Of course it would be great to see sim2real results and any result obtained in a single impulse-based simulation on dynamic interactive tasks must be taken with a grain of salt. That said, I see clear impact and novelty here and support acceptance of the work.\n\nPros:\n- novel formulation \n- auxiliary objectives to force the policy to encode the haptic data\n  - future state prediction is a good objective to encourage learning from sparse signals\n- separate memory for auxiliary haptic objectives allow for optimization on a wider distribution of data offline\n- significant improvements in capability on test tasks\n  - simulated agent is superhuman in bounce and baoding\n- robot tactile olympiad dataset\n- binary tactile data does appear to be more transferable between sim and real in the short term\n- Good hyper-paramter tuning to ensure validity of results\n- Nice focus on and interesting format of discussion section adds scientific value to the work\n  - I like the \"translation to practical recommendations\" as a focus on advancing the field\n\nCons:\n- binary contact signals are limiting for more practical tasks\n- no real world transfer experiments\n  - because physics simulation is impulse based approximation, it isn't clear if superhuman performance would be possible in the real world vs. simulation\n- large variances for complex task policies\n- No unified hyper-paramter setting for all tasks indicates challenges in generalization", "was_revised": true, "final_justification": "During the rebuttal phase the authors have answered my and fellow reviewers questions adequately to keep my continued support for the acceptance of this work.", "rebuttal": "We thank **WBY5** for their careful review and for the highly insightful comments! \n\n**[WBY5-1] Are the superhuman policies realistic?**\n\nWe agree that any results for a dynamic contact-rich task in simulation must be critically examined.  Based on our analysis, we do not believe our superhuman performance is due to exploiting simulation inaccuracies. We have several pieces of evidence for this:  \n\n1. *Oscillatory impulses masquerading as bouncing*: The maximum bounce count was 88 over a 600-timestep episode, which translates to approximately seven timesteps between each bounce. By slowing down the provided videos to 0.25x speed, it is evident that the SSL-optimised agent has learned a sophisticated strategy involving both distinct bounces (see timestamp 00:09) and a secondary, vibration-like behaviour (see timestamp 00:07), which together enable the superhuman performance. This strategy contrasts with the RL-only agent’s repetitive and less effective motions, suggesting our method has learned a superior behaviour rather than exploiting a simulation artifact.\n2. *Similar improvements in static tasks*: Our method also shows significant performance improvements in the “Find” task, which is a completely static, non-dynamic environment. In this task, our agents found the target object on average 1.5 seconds faster than the baselines (see Fig. A1 in the supplementary PDF). This demonstrates that our method's benefits are not exclusive to dynamic environments, reinforcing the assertion that our core contribution is a more effective learning strategy rather than the exploitation of dynamic physics.\n\n\n**[WBY5-2] Is there value in optimising for superhuman performance in simulation?**\n\nWe believe there is great value. See L275 to 278 for a discussion of our performance in the context of human baselines. A key tenet of modern reinforcement learning research, from Atari to AlphaGo, has been to push beyond human capabilities in simulation as a means of discovering novel and superior strategies. In our work, we were initially focused on the fundamental challenge of getting a \"blind\" agent to perform any meaningful action at all. The discovery that our methodology enables superhuman capabilities was an unexpected but powerful outcome, attesting to its ability to unlock creative solutions for challenging, partially observable control tasks.  \n\n\n\n**[WBY5-3] What is the likelihood of sim2real transfer?**\n\nWe believe that the benefits of our method should translate to the real world, assuming a successful sim2real gap can be bridged for the base tactile agent. This is because our core contribution is a method for improving the learning process and generating superior policies. If the physical assumptions are generally valid, these strategies should be transferable. Our deliberate choice of binary contact signals and a methodology that only modifies network weights were made to specifically facilitate this future transfer. Please see response **[gp4F-4]** for more discussion of sim2real.   \n\n\n\n**[WBY5-4] Binary contact signals are limiting for more practical tasks.**  \n\nThe choice of binary signals is a deliberate one, driven by practical constraints and the state of sim2real research. While richer sensory data is theoretically advantageous, it incurs substantial computational costs (e.g., see discussion on L305) and faces a significant sim2real gap, even for continuous signals due to the difficulty of accurately simulating contact physics. \n\nBinary tactile sensors, in contrast, are cheap, robust, and require minimal calibration (e.g. can employ a low pass filter and then threshold the voltage [A]). Our work demonstrates, for the first time, that it is possible to achieve superhuman performance using only these simple measurements. This finding not only challenges the perception that binary signals are limiting but also establishes a highly feasible and effective baseline for real-world tactile learning.  \n\n[A] Lee et al., DexTouch: Learning to Seek and Manipulate Objects with Tactile Dexterity, IEEE RAL\n\n**[WBY5-5] Large variances for complex task policies.** \n\nThere is indeed a large variance observed in the policies for the complex Baoding task (see Fig. 4). We attribute this variance to the difficult nature of this partially observable control task and the relatively sparse reward signal (Sec D3 in the supplementary PDF). A single successful rotation provides a large, sparse bonus (+10 compared to distance reward within [0, 0.1]), making agents vulnerable to this high-variance behavior.\nThe ability to mitigate this challenge is a key contribution of our work. As shown in Fig. A3 (bottom left) of the supplementary PDF, the RL-only agents and the full dynamics agents both exhibit significant performance variance, with multiple seeds failing to achieve a successful rotation. However, our tactile reconstruction agent (i.e., “+tactile recon”) is the only method that achieves consistent, low-variance returns, with all seeds successfully learning the task. The ability of our approach to significantly reduce variance in this difficult setting is a notable and primary result. \n\n**[WBY5-6] No unified hyper-parameter setting for all tasks indicates challenges in generalization.**  \n\nThis is an interesting question. However, we believe that diverse hyperparameter settings across tasks are a necessary and a sound methodological choice. The tasks in our study (see Fig. 2) are intentionally diverse, encompassing distinct contact dynamics, devices,  and control objectives. As is standard practice in reinforcement learning research, it is highly unlikely that a single set of hyperparameters would be optimal across such a varied set of problems. Thus, we followed the best practice (e.g., as in [B]) of performing a hyperparameter search for each method on each task.\n\nThis approach is crucial for validating the true performance of our method. The fact that our methods consistently outperform the baselines after both are optimally tuned for each specific task provides stronger evidence for its efficacy. We will release our training code, including the hyperparameter search configurations and final values to facilitate reproducibility and future research. We will also clarify the hyperparameter settings in the revised text. \n\n[B] Eimer et al., “Hyperparameters in Reinforcement Learning and How To Tune Them”, ICML 2023"}, {"review_id": "EaLSY9wD1Z", "reviewer": "Reviewer_Xsn5", "summary": "This work presents a new method for in-hand robot manipulation tasks, which introduces two dynamics objectives: tailored reconstruction and multi-step forward dynamics with auxiliary memory to address the issue of 1) the policy prematurely converging to features, 2) the difficulty of learning from sharp transitions and sparse nature data. The experiments show that the new methods outperform the previous work in multiple in-hand manipulation tasks.", "questions": "1. Are the two objectives mainly to help the training of the encoder? The decoder (d),  the forward model (f), and the predictor (p) are not used during inference ?\n2. Are the two objectives designed for the encoder only? Not the value network and the policy?  If so, how can we prevent these two networks from falling into the problem in the hypotheses?\n3. How does the tactile reconstruction prevent the model from \"prematurely converging to features\"? How does \"Multi-step forward dynamics\" help learning from sharp transitions and the sparse nature of tactile data more easily? For me, this seems to be hard to prove, but it is claimed in the paper that the new methods address the problems somehow. It is important to prove it.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Tailored reconstruction and multi-step forward dynamics incorporate extracted learning objectives for the encoder, enabling the model to effectively capture and extract features from the observations, thereby facilitating the performance of downstream tasks.\n2. The incorporation of the novel loss does not significantly enhance the complexity of the pipeline. These straightforward objectives maintain clarity and simplicity within the pipeline.\n\n## Weakness\n\n1. The experiments show that the overall performance increased, and the ablations show that the new objectives benefit the performance. But the proposed solutions for the hypotheses are not validated in the work. \n    1. Tactile reconstruction softens the issue that learning \"prematurely converges to features\" is not discussed, as figures of value/policy loss show the gradients converging more smoothly.\n    2. Also, no experiments show that \"Multi-step forward dynamics\" helps the learning from \"sharp transitions and sparse nature\n\n2. Figure 1 is very confusing, making it hard to trace the details of the main contribution.A more detailed caption should be added to the paper.  I am assuming\n    1. the blocks/squares in the right two subfigures (ssl) from left to right are the feed-forward directions (even though they do not have arrows)\n    2. SSL: tactile reconstruction: the upper sequence is a feed-forward network, and the lower arrow is the ground truth. L_BCE is computed by MSE from the upper and the lower\n    3. SSL ulti-step forward dynamics: similar confusion to the above.", "was_revised": true, "final_justification": "The author addressed all my concerns and limitations during the discussion. I would like to accept the paper.", "rebuttal": "We thank **Xsn5** for taking the time to review our paper and we appreciate the helpful feedback.  \n\n\n\n**[Xsn5-1] This work “introduces two dynamics objectives: tailored reconstruction and multi-step forward dynamics with auxiliary memory.“**\n\nTo clarify, we introduce three objectives for training tactile-based agents (Sec 3.2): \n1. Tactile reconstruction: decode the tactile state from the multimodal representation through binary classification.  \n2. Full dynamics: multi-step forward dynamics on the multimodal representation.   \n3. Tactile dynamics: multi-step forward dynamics on the multimodal representation and decode tactile state from each predicted future representation.  \n\nTactile reconstruction (1) and tactile dynamics (3) are novel objectives, and the application of full dynamics (2) to tactile-based agents in model-free RL is novel.   \n\n\n\n**[Xsn5-2] The proposed solutions for the hypotheses are not validated in the work… how does tactile reconstruction prevent premature convergence… how does dynamics help learn from sharp transitions?”**\n\nOur results (Figs 4, 5, 6) strongly demonstrate the efficacy of our methods, but we refrained from making definitive causal claims about the precise mechanisms underlying their success. For example, in Fig. 6 we observe that the vanilla PPO model significantly underperforms on the real-world metrics, perhaps lending some support to the theory that it has “converged prematurely”. In the same figure, we also observe that the addition of tactile signal and dynamics further improves performance on our benchmark tasks which feature “sharp transitions” between interaction states. We also attempted to provide additional insight into the learned representations through visualising future tactile states (Figs A5, A6) and performing latent trajectory analysis (Sec J) in the supplementary PDF.  \n\nTo further validate these hypotheses we can provide the following analysis in the revised paper. Note, the inclusion or omission of these results do not take away from our central contributions, but instead serve to provide additional insight into the unique challenges in this underexplored problem setting. Thanks for the suggestion!  \n\n*H1: Preventing premature convergence through tactile reconstruction*   \n* Hypothesis: The tactile reconstruction objective forces the network to maintain a meaningful representation of tactile input throughout training, preventing it from being “down-weighted” prematurely in favour of other modalities.  \n* Experiment: We will track the L2-norm of the first layer weight vectors corresponding to proprioceptive and tactile inputs throughout training for each task and methodology. After normalising for input size, a larger contribution from the tactile input portion in agents trained with tactile reconstruction compared to RL-only agents will indicate that the model is extracting tactile features more effectively and from an earlier stage, preventing premature reliance on proprioception.\n\n*H2: Improving learning from discontinuous data through dynamics*   \n* Hypothesis: By forcing the encoder to predict future state representations, the multi-step dynamics objective creates a more continuous learning signal, regularising learning from sparse and discontinuous tactile inputs.  \n* Experiment: We will track the gradient norms of the input data with respect to the policy and value function. Firstly, we will establish if training with tactile data does indeed introduce greater gradient instabilities compared to proprioceptive agents. Secondly, we will evaluate if our dynamics-based objectives lead to enhanced optimisation (evidenced by reduced magnitude or frequency in gradient spikes). In addition, we will provide visualisations of the loss landscape of each agent to support our quantitative results on optimisation stability.\n\n\n**[Xsn5-3] Clarity of Fig. 1 and its caption.**  \n\nWe thank the Xsn5 for highlighting these clarity issues, and apologise if our figure was difficult to follow. In light of the comments, we will update Fig. 1 to resolve any potential ambiguities (e.g., adding missing arrows between components). We will add a legend to clarify that rectangular blocks denote feed-forward networks or layers, squares denote vectors, text denotes scalars, variables with “hats” denote model predictions, and those without are observed. Finally, we will update the caption to the following:  \n\n*(Top left)*: Each timestep, proprioceptive (prop) and tactile (tact) readings are concatenated to form the timestep observation $o_t$. A stack of the $k$ last observations forms the current state $s_t$. The encoder $e$ learns the state representation $z_t=e(s_t)$ which the policy and value function are conditioned on. During learning, the self-supervised loss is added to the RL losses and backpropagated in a single pass. The  self-supervised loss optimises the encoder $e$ and its own task-specific networks ($d$, $f$, $p$).   \n*(Top right)*: The tactile reconstruction objective optimises the encoder $e$ (and decoder $d$) to preserve the tactile state $s_t^{tact}$ in the representation $z_t$ by implementing reconstruction as binary classification with a binary cross-entropy loss $L_{BCE}$.   \n*(Bottom right)*: The multi-step forward dynamics objective optimises the encoder $e$ (and forward model $f$ and projector $p$) to extract information from the state $s_t$ that will aid in predicting representations $n_f$ timesteps into the future.  \n*(Bottom left)*: To further enhance learning, we propose to separate the RL memory and the auxiliary memory, which stores the last $N_{rollouts}$ of the RL memory.   \n\nWe confirm **Xsn5**'s understanding that the feed-forward direction of the blocks/squares on the right side of Fig. 1 is from left-to-right. To clarify, in tactile reconstruction $L_{BCE}$ is a binary cross-entropy loss (see Eqn 1) between the observed/ground truth tactile state (lower) and decoded/reconstructed tactile state (upper), not a MSE loss. We confirm that the dynamics loss is MSE (see Eqn 2), and we add that it is cumulatively summed for each timestep into the future: $L_1 + L_2 + … + L_{n_f}$. The computation of this loss is somewhat complex due to the need for a target network and projector (see L159-172 for full details), but we will strive to communicate this more clearly in the text and updated figure.   \n\n\n\n**[Xsn5-4] Are the decoder, forward model, and the predictor used during inference?** \n\nThese additional components are just used to assist with learning the encoder. Every network except the encoder and policy can be discarded for deployment.   \n\n\n\n**[Xsn5-5] Potential fallibility of policy and value networks running into the same problem as encoder.**  \n\nWe confirm **Xsn5**’s understanding that yes, the self-supervised objectives are applied solely to the encoder (and relevant auxiliary networks). The policy and value networks are only optimised with the RL loss. The motivation of our approach is that through self-supervision, the encoder learns to generate a richer and more robust latent representation $z_t$ of the agent’s state. This “pre-processed” representation is then readily leverageable by the downstream policy and value function. \n\nEmpirical evidence for this enhanced representation is In Sec J of the supplementary PDF, where we show the PCA reduction of the learned 256-dim state representation $z_t$ across an episode rollout. Taking Fig. A13 as the example, the latent trajectory of the self-supervised agents is much richer and diffuse between timesteps compared to the RL-only agents. This distinctive latent space provides evidence that self-supervision has a strong influence on the learned representation, allowing the policy and value networks to operate in a space less susceptible to the challenges of sensory data."}]

Author rebuttal:

Meta-review: This paper studies tactile manipulation and self-supervised approaches for it, creating simulation environments in Isaac Lab to benchmark different approaches. 

For strengths, reviewers mentioned the finding of using self-supervised learning as representation learning as superior to "RL only", studying "tactile dynamics" is interesting, thorough hyperparameter and learning rate search, and that the peak performances are impressive.

For weaknesses, many reviewers mentioned the absence of sim2real experiments, a lack of technical contributions, that Fig 1. is very confusing, and missing discussion of some related work.

The method is not novel, but the paper claims that its application and analysis is (specializing model-based RL to tactile manipulation settings).

Overall, the claimed contributions are supported by evidence and the claimed contributions are interesting enough, despite the main limitation of absence of real robot evaluation. I remain on the fence about this paper due to the weaknesses.
Decision: Accept (poster)